Skip to content

Improve Parquet metadata reads by avoiding fixed-size prefetch #687

Description

@JunRuiLee

Problem

The async Parquet reader currently uses a fixed 512 KiB metadata prefetch hint. For HDFS through OpenDAL’s native service, this can read substantially more data than the footer metadata requires.

Internal HDFS testing on a representative workload showed better read performance when following parquet-hadoop’s seek-based sequence instead: read the 8-byte footer, then fetch the exact footer metadata and any requested page-index range.

Proposed change

  • Keep the existing 512 KiB metadata prefetch for object stores and unknown or custom backends.
  • Use exact metadata range reads only for backends where positioned reads are known to be cheap, including HDFS native, local filesystem, and memory.
  • Keep page-index loading demand-driven.
  • Avoid loading OffsetIndex for an empty external row selection.
  • Keep the backend classification and Parquet metadata-read policy crate-private instead of expanding the public FileRead contract.
  • Add range-tracking tests for exact footer, metadata, and OffsetIndex reads, and for retained prefetch behavior.

Validation

  • Internal HDFS comparison on the target workload showed an improvement with exact metadata reads.
  • Automated tests verify the expected request ranges and preserve the conservative object-store behavior.
  • A standalone reproducible benchmark report is not required for this issue; additional performance measurements can be documented separately if needed.

Implementation: #686

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions