Skip to content

Cython Avro decoder reads past the buffer end on malformed input #3952

Description

@sungwy

Two places in pyiceberg/avro/decoder_fast.pyx advance the read pointer using a value taken from the stream, without bounding it against self._end.

1. read_bytes does not validate the decoded length

cpdef inline bytes read_bytes(self):
    cdef uint64_t length;
    if self._current >= self._end:      # only confirms 1 byte is available
      raise EOFError(f"EOF: read 1 bytes")

    decode_zigzag_ints(&self._current, 1, &length)

    if length <= 0:
        return b""
    cdef const unsigned char *r = self._current
    self._current += length             # not checked against self._end
    return r[0:length]

The guard confirms one byte is available before decoding the length, but the decoded length is then used to slice and to advance _current with no check that _current + length <= _end. A length field larger than the remaining buffer reads beyond it.

2. decode_zigzag_ints has no end pointer to bound against

void decode_zigzag_ints(const unsigned char **buffer, const uint64_t count, uint64_t *result);

The signature takes a buffer and a count but no end, so the varint walk cannot stop at the buffer boundary — a run of bytes with the continuation bit set keeps advancing. It is called from five sites in the decoder (lines 93, 101, 111, 124, 176), including from read_bytes above.

Both are reachable from a malformed or hostile Avro manifest.


Issue investigation generated via claude, reviewed by Sung, Kevin, Fokko.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions