Problem
The async Parquet reader currently uses a fixed 512 KiB metadata prefetch hint. For HDFS through OpenDAL’s native service, this can read substantially more data than the footer metadata requires.
Internal HDFS testing on a representative workload showed better read performance when following parquet-hadoop’s seek-based sequence instead: read the 8-byte footer, then fetch the exact footer metadata and any requested page-index range.
Proposed change
- Keep the existing 512 KiB metadata prefetch for object stores and unknown or custom backends.
- Use exact metadata range reads only for backends where positioned reads are known to be cheap, including HDFS native, local filesystem, and memory.
- Keep page-index loading demand-driven.
- Avoid loading
OffsetIndex for an empty external row selection.
- Keep the backend classification and Parquet metadata-read policy crate-private instead of expanding the public
FileRead contract.
- Add range-tracking tests for exact footer, metadata, and
OffsetIndex reads, and for retained prefetch behavior.
Validation
- Internal HDFS comparison on the target workload showed an improvement with exact metadata reads.
- Automated tests verify the expected request ranges and preserve the conservative object-store behavior.
- A standalone reproducible benchmark report is not required for this issue; additional performance measurements can be documented separately if needed.
Implementation: #686
Problem
The async Parquet reader currently uses a fixed 512 KiB metadata prefetch hint. For HDFS through OpenDAL’s native service, this can read substantially more data than the footer metadata requires.
Internal HDFS testing on a representative workload showed better read performance when following parquet-hadoop’s seek-based sequence instead: read the 8-byte footer, then fetch the exact footer metadata and any requested page-index range.
Proposed change
OffsetIndexfor an empty external row selection.FileReadcontract.OffsetIndexreads, and for retained prefetch behavior.Validation
Implementation: #686