Apache Iceberg version
main (development), commit 308768d99
Please describe the bug 馃悶
Manifest records do not retain their schema, and their property getters use hard-coded field positions. When the in-memory record layout differs from the file schema, this can cause incorrect field access or silent data loss during serialization.
For example, referenced_data_file has field ID 143 in both v2 and v3, but its zero-based position differs:
| Field |
v2 position |
v3 position |
first_row_id |
Not present |
16 |
referenced_data_file |
16 |
17 |
This causes two problems:
- Writing a v3 record to a v2 file:
DataFile.from_args() defaults to the v3 layout. When writing it through AvroOutputFile with a v2 file schema, omitting record_schema makes the writer read position 16 instead of 17 for referenced_data_file. If first_row_id is null, a non-null reference is silently written as null. Explicitly supplying the v3 record schema enables the existing field-ID projection and preserves the value.
- Accessing a v2 record:
DataFile.from_args(_table_format_version=2, ...) stores the reference at position 16, but the getters still assume v3 positions. Consequently, first_row_id returns the reference path and referenced_data_file raises IndexError. Reading a v2 file into DataFile without projecting to v3 has the same mismatch.
The normal manifest read/write helpers already supply the v3 projection. However, correctness still depends on callers separately tracking the in-memory layout and supplying the matching schema.
Proposed behavior
Records should keep the schema that describes their in-memory layout. Getters should use field IDs, not assume a particular version鈥檚 field positions.
When reading, use the schema from the file. If the read projects into another schema, retain that schema on the resulting record.
When writing, callers should only need to specify the target format version, or file_schema when using AvroOutputFile. IO should determine the input layout from the record and handle the conversion before encoding. This needs to account for nested records and partition fields, not just the top-level record.
We shouldn鈥檛 need to pass record_schema just to prevent a field from being read at the wrong position. Keep it supported for callers that explicitly provide it.
For the example above, writing a default v3 record to a v2 file should preserve referenced_data_file without passing record_schema. If a conversion isn鈥檛 supported, raise an error rather than silently writing the wrong value.
Willingness to contribute
N/A
Apache Iceberg version
main (development), commit
308768d99Please describe the bug 馃悶
Manifest records do not retain their schema, and their property getters use hard-coded field positions. When the in-memory record layout differs from the file schema, this can cause incorrect field access or silent data loss during serialization.
For example,
referenced_data_filehas field ID143in both v2 and v3, but its zero-based position differs:first_row_idreferenced_data_fileThis causes two problems:
DataFile.from_args()defaults to the v3 layout. When writing it throughAvroOutputFilewith a v2 file schema, omittingrecord_schemamakes the writer read position 16 instead of 17 forreferenced_data_file. Iffirst_row_idis null, a non-null reference is silently written as null. Explicitly supplying the v3 record schema enables the existing field-ID projection and preserves the value.DataFile.from_args(_table_format_version=2, ...)stores the reference at position 16, but the getters still assume v3 positions. Consequently,first_row_idreturns the reference path andreferenced_data_fileraisesIndexError. Reading a v2 file intoDataFilewithout projecting to v3 has the same mismatch.The normal manifest read/write helpers already supply the v3 projection. However, correctness still depends on callers separately tracking the in-memory layout and supplying the matching schema.
Proposed behavior
Records should keep the schema that describes their in-memory layout. Getters should use field IDs, not assume a particular version鈥檚 field positions.
When reading, use the schema from the file. If the read projects into another schema, retain that schema on the resulting record.
When writing, callers should only need to specify the target format version, or
file_schemawhen usingAvroOutputFile. IO should determine the input layout from the record and handle the conversion before encoding. This needs to account for nested records and partition fields, not just the top-level record.We shouldn鈥檛 need to pass
record_schemajust to prevent a field from being read at the wrong position. Keep it supported for callers that explicitly provide it.For the example above, writing a default v3 record to a v2 file should preserve
referenced_data_filewithout passingrecord_schema. If a conversion isn鈥檛 supported, raise an error rather than silently writing the wrong value.Willingness to contribute
N/A