Skip to content

[format] Fix RLE skipping at sparse row-range boundaries - #9127

Merged
leaves12138 merged 2 commits into
apache:masterfrom
CrownChu:fix/parquet-sparse-row-rle-boundary
Aug 10, 2026
Merged

[format] Fix RLE skipping at sparse row-range boundaries#9127
leaves12138 merged 2 commits into
apache:masterfrom
CrownChu:fix/parquet-sparse-row-rle-boundary

Conversation

@CrownChu

@CrownChu CrownChu commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change?

An RLE run covers the half-open interval [rowId, rowId + n). When its end is exactly equal to rangeStart, it has no overlap with the requested sparse row range and must be skipped. The existing strict < boundary can retain that run and misalign sparse Parquet reads.

For example, consider an RLE run starting at row 0 with n = 10 and a sparse read requesting row range [10, 10]:

RLE run:     [0, 10)   // rows 0 through 9
Sparse range: [10, 10] // row 10

These ranges do not overlap. However, the old condition rowId + n < rangeStart evaluates 10 < 10 to false and enters the overlap branch. It then computes start = 10, end = 9, and an invalid zero-length intersection. Using <= skips the preceding run and lets decoding continue at row 10.

Brief change log

  • Change all three affected RLE boundary checks from < to <=.
  • Add a data-evolution regression test covering sparse row positions.
  • Verify both a flat column and a repeated/list column at the boundary positions.

The regression writes 10,000 Parquet rows with an INT column and an ARRAY<INT> column, then performs a sparse indexed read at row positions 10 and 4210. It verifies the exact output:

1000:[0,1]
5200:[4200,4201]

This exercises both the flat definition-level path and the repeated/list path across Parquet page and RLE boundaries.

Verifying this change

mvn -pl paimon-core -am -DskipITs -Dfast \
  -Dtest=DataEvolutionSplitReadTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

Result: 14 tests passed. Spotless and Checkstyle also passed.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@leaves12138
leaves12138 merged commit 437bcc3 into apache:master Aug 10, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants