AVRO-4296: [python] Bound allocation when decoding length-prefixed values and collections

[iemejia]

What is the purpose of the change

A bytes or string value is encoded as a length prefix followed by that many bytes of data, and an array or map block is encoded as an element count followed by that many items. A malicious or truncated input can declare a very large length or count while carrying little or no actual data, which causes a correspondingly large allocation before the shortfall is noticed.

This applies the equivalent of the Java SDK fix AVRO-4241 to the Python SDK and extends it to collections. It has two complementary parts.

1. Validate available bytes before allocating

When the source can report how many bytes remain, a declared length (or a collection block count) that exceeds the bytes actually available is rejected before allocating for it. The collection check uses the minimum on-wire size of the element schema, so a zero-byte element type (such as null) is never falsely rejected. Sources that cannot report their remaining size are unaffected.

BinaryDecoder.bytes_remaining() reports the bytes still readable for a seekable reader (else None). read() rejects an over-large declared length above a threshold, and DatumReader.read_array/read_map reject a block whose element count could not be backed by the bytes remaining, using min_bytes_per_element() from the element schema.

2. Cap collection allocation for zero-byte elements

Zero-byte elements (null, a zero-length fixed, or a record with only zero-byte fields) consume no input, so the available-bytes check cannot bound their count: a tiny payload such as {{"type":"array","items":"null"}} declaring a block count of 200,000,000 would otherwise drive an unbounded allocation. In addition to the available-bytes check, _ensure_collection_available caps the cumulative count of zero-byte elements (DEFAULT_MAX_COLLECTION_ITEMS = 10,000,000) and applies a structural cap to every collection (DEFAULT_MAX_COLLECTION_STRUCTURAL = Integer.MAX_VALUE - 8) covering non-seekable readers. It is applied to read_array, read_map, skip_array and skip_map (cumulative across blocks, and after normalizing a negative block count); maps are additionally bounded by their >=1-byte keys. Rejections raise the new AvroCollectionSizeException. When set, the AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits.

This folds in and supersedes the standalone collection-limit change (AVRO-4282, #3845), so this PR is the single complete fix for collection/length-prefixed allocation DoS in the Python SDK.

This is a sub-task of AVRO-4292 and resolves AVRO-4296.

Verifying this change

This change added tests and can be verified as follows:

• Added TestBinaryDecoderAvailableBytes, TestDatumReaderCollectionAvailableBytes and the zero-byte cap tests in lang/py/avro/test/test_io.py, including an array<null> with a huge block count that must be rejected and a small one that still decodes.
• Run: cd lang/py && python3 -m unittest avro.test.test_io

Documentation

• Does this pull request introduce a new feature? (no — hardening / robustness)
• If yes, how is the feature documented? (not applicable)

[Copilot]

Pull request overview

This PR hardens the Python Avro binary decoding path against malicious or truncated inputs that declare excessively large length/count prefixes, by validating declared sizes against bytes remaining for seekable inputs before allocating/iterating.

Changes:

• Add BinaryDecoder.bytes_remaining() and use it to pre-reject oversized read(n) requests (above a threshold) when the reader is seekable.
• Add minimum on-wire-size estimation for schemas and use it to validate array/map block counts in DatumReader against remaining bytes.
• Add unit tests covering oversized length prefixes, oversized collection block counts, and a non-false-positive case (array of null).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
lang/py/avro/io.py	Adds remaining-bytes introspection plus pre-checks for large length-prefixed reads and collection block count validation using per-element minimum sizes.
lang/py/avro/test/test_io.py	Adds targeted tests for the new available-bytes validation behavior in BinaryDecoder and DatumReader (including array-of-nulls).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[iemejia]

This PR now also includes the collection block-count cap for [python], so it is the single complete fix for collection allocation DoS in this SDK. In addition to validating available bytes before allocating length-prefixed values, it bounds the number of array/map items per block:

• a heap-aware cap for zero-byte-element collections (e.g. array<null>), which otherwise bypass the available-bytes check because each element reads 0 bytes;
• a structural cap for all collections;
• bounded skip paths so projection/skip cannot loop unboundedly.

With this, the standalone collection-limit change for [python] (AVRO-4282, #3845) is redundant and is being closed as superseded by this PR.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[iemejia]

Preallocation (AVRO-4292 follow-up) evaluated — no change needed here. read_array/read_map build read_items/the dict via .append(), growing on demand rather than preallocating to the declared block count, so there is no up-front over-allocation to bound (unlike the Java/C++/C# readers, which sized a buffer to the count). The existing structural and zero-byte caps already bound the growth.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

lang/py/avro/io.py:1032

• read_map() uses len(read_items) as the "existing" element count for _ensure_collection_available(). Because maps can legally contain duplicate keys on the wire (which overwrite in the dict), len(read_items) can significantly undercount the number of key/value pairs actually decoded. This weakens the cumulative/structural cap and can allow an attacker to bypass the intended element-count limit by repeating keys across blocks.

        read_items: Dict[str, object] = {}
        # Map keys are strings (>= 1 byte length prefix) plus the value.
        min_bytes = 1 + _min_bytes_per_element(writers_schema.values)
        zero_byte_limit, structural_limit = _collection_limits()
        block_count = decoder.read_long()
        while block_count != 0:
            if block_count < 0:
                block_count = -block_count
                decoder.skip_long()
            self._ensure_collection_available(decoder, len(read_items), block_count, min_bytes, zero_byte_limit, structural_limit)
            for i in range(block_count):
                key = decoder.read_utf8()
                read_items[key] = self.read_data(writers_schema.values, readers_schema.values, decoder)
            block_count = decoder.read_long()