introduce optional rle reads from parquet by Rich-T-kid · Pull Request #24227 · apache/datafusion

Rich-T-kid · 2026-08-10T15:21:19Z

Which issue does this PR close?

Rationale for this change

When DataFusion reads a parquet file with dictionary-encoded string or binary columns, it currently decodes the dictionary and returns plain Utf8/Binary arrays, discarding the encoding. For low-cardinality columns (status, country, category, etc.) this doesn't take full advantage of the compacted format parquet gives the engine Preserving the dictionary encoding reduces memory usage and can improve aggregation performance on these columns.

What changes are included in this PR?

Adds datafusion.execution.parquet.enable_rle_to_dictionary (default false). When enabled:

DFParquetMetadata::fetch_schema rewrites dictionary-encoded string/binary fields to Dictionary(Int32, Utf8/Binary) at schema inference time, so physical plans are built with the correct types
prepare_filters in the parquet opener detects the same columns at execution time and passes the rewritten schema to ArrowReaderOptions::with_schema, telling the arrow-rs reader to produce dictionary arrays directly

Are these changes tested?

Unit tests in opener/mod.rs verify string and binary columns are read as Dictionary(Int32, Utf8) / Dictionary(Int32, Binary) when the flag is enabled, and as plain types when disabled
Unit tests in source.rs verify flag propagation through config and end-to-end type correctness through a DataSourceExec -> FilterExec -> ProjectionExec plan
A sqllogictest file writes a real parquet file and uses arrow_typeof() to assert the physical Arrow type changes between the two flag states

Are there any user-facing changes?

New session config option: SET datafusion.execution.parquet.enable_rle_to_dictionary = true. Default is false so existing behavior is unchanged.

Rich-T-kid · 2026-08-10T15:28:14Z

+        // to Dictionary(Int32, ...) so the parquet reader produces dictionary arrays.
+        // All three schemas are updated together so the reader, expr-adapter, and
+        // downstream operators all agree on the output type.
+        if prepared.enable_rle_to_dictionary {


All three schemas are updated together so the reader, expr-adapter, and downstream operators all agree on the output type.

Rich-T-kid · 2026-08-10T15:33:05Z

        pub max_in_list_size: usize, default = 20

+        /// (reading) If true, string and binary columns that are dictionary-encoded in
+        /// the parquet file are read as `Dictionary<Int32, Utf8>` / `Dictionary<Int32, Binary>`


I choose to keep the key type hard coded to int32 instead of u64, would not mind changing

github-actions · 2026-08-10T15:34:06Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [  59.006s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.038s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [  59.339s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.038s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.940s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 121.965s] datafusion
    Building datafusion-common v54.1.0 (current)
       Built [  33.524s] (current)
     Parsing datafusion-common v54.1.0 (current)
      Parsed [   0.068s] (current)
    Building datafusion-common v54.1.0 (baseline)
       Built [  33.021s] (baseline)
     Parsing datafusion-common v54.1.0 (baseline)
      Parsed [   0.065s] (baseline)
    Checking datafusion-common v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.004s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1196

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  68.894s] datafusion-common
    Building datafusion-datasource-parquet v54.1.0 (current)
       Built [  48.207s] (current)
     Parsing datafusion-datasource-parquet v54.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-datasource-parquet v54.1.0 (baseline)
       Built [  48.263s] (baseline)
     Parsing datafusion-datasource-parquet v54.1.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion-datasource-parquet v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.213s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure function_parameter_count_changed: pub fn parameter count changed ---

Description:
A publicly-visible function now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/function_parameter_count_changed.ron

Failed in:
  datafusion_datasource_parquet::file_format::apply_file_schema_type_coercions now takes 3 parameters instead of 2, in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/schema_coercion.rs:50
  datafusion_datasource_parquet::apply_file_schema_type_coercions now takes 3 parameters instead of 2, in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/schema_coercion.rs:50

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  98.208s] datafusion-datasource-parquet
    Building datafusion-proto-common v54.1.0 (current)
       Built [  21.315s] (current)
     Parsing datafusion-proto-common v54.1.0 (current)
      Parsed [   0.050s] (current)
    Building datafusion-proto-common v54.1.0 (baseline)
       Built [  21.332s] (baseline)
     Parsing datafusion-proto-common v54.1.0 (baseline)
      Parsed [   0.051s] (baseline)
    Checking datafusion-proto-common v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.477s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  45.307s] datafusion-proto-common
    Building datafusion-proto-models v54.1.0 (current)
       Built [  24.383s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.135s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  24.464s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.139s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   2.576s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:868
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:868

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  53.002s] datafusion-proto-models
    Building datafusion-sqllogictest v54.1.0 (current)
       Built [  98.508s] (current)
     Parsing datafusion-sqllogictest v54.1.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v54.1.0 (baseline)
       Built [  98.688s] (baseline)
     Parsing datafusion-sqllogictest v54.1.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.127s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 199.957s] datafusion-sqllogictest

codecov-commenter · 2026-08-10T19:42:57Z

Codecov Report

❌ Patch coverage is 92.74194% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.19%. Comparing base (1164d60) to head (fc360e8).
⚠️ Report is 30 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-common/src/generated/pbjson.rs	0.00%	14 Missing ⚠️
datafusion/datasource-parquet/src/metadata.rs	92.98%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24227      +/-   ##
==========================================
+ Coverage   81.14%   81.19%   +0.05%     
==========================================
  Files        1112     1110       -2     
  Lines      386933   388987    +2054     
  Branches   386933   388987    +2054     
==========================================
+ Hits       313978   315854    +1876     
- Misses      54462    54544      +82     
- Partials    18493    18589      +96

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Rich-T-kid · 2026-08-12T23:58:21Z

@adriangb pinging you since you seem interested in parquet related speed ups 👍

adriangb · 2026-08-13T12:49:06Z

If I understand correctly the goal is to evaluate filters during filter pushdown against dictionary / RLE encoded columns? We can't propagate these dynamic type changes to the rest of the query plan / scan. Is that right?

Rich-T-kid · 2026-08-13T13:55:45Z

@adriangb

If I understand correctly the goal is to evaluate filters during filter pushdown against dictionary / RLE encoded columns?

no not exactly. The goal of this PR is to keep RLE parquet columns in their compacted form by materializing them as dictionary arrays instead of regular strings.

We can't propagate these dynamic type changes to the rest of the query plan / scan. Is that right?

exactly! This is why it needs to be done as early in the plan as possible. we inspect DFParquetMetadata::fetch_schema to see if any RLE columns exist, if so change the plan type from utf8/binary to dict<_,utf8/binary>. Thanks to some plumbing in arrow-rs it will handle the conversions correctly returning a dictionary array, this can cause a 60x perf boost

adriangb · 2026-08-13T15:47:56Z

no not exactly. The goal of this PR is to keep RLE parquet columns in their compacted form by materializing them as dictionary arrays instead of regular strings.

Why only RLE and not dictionaries as well? How does this compare to / relate to the schema_force_view_types option?

It also looks like this goes through infer_schema right? A lot of code paths never touch that (CREATE EXTERNAL TABLE (a VARCHAR), any custom table providers, etc.).

I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into Utf8View. That would be applicable to all DataFusion users.

Rich-T-kid · 2026-08-13T17:13:11Z

Why only RLE and not dictionaries as well? How does this compare to / relate to the schema_force_view_types option?

my bad when I say RLE i'm referring to RLE_DICTIONARY

It also looks like this goes through infer_schema right? A lot of code paths never touch that (CREATE EXTERNAL TABLE (a VARCHAR), any custom table providers, etc.).

I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into Utf8View. That would be applicable to all DataFusion users.

I agree, ill update the PR to target all parquet scans.

Rich-T-kid · 2026-08-13T18:59:51Z

ideally we surface columns that are physically RLE_DICTIONARY-encoded in the parquet file as Arrow Dictionary(Int32, Utf8) arrays rather than decoding them back to plain Utf8.

To know whether a specific column is RLE_DICTIONARY-encoded you need to read the parquet file footer. For the infer_schema path this happens at table registration, but for explicit schemas (CREATE EXTERNAL TABLE (col VARCHAR)) and direct ParquetSource construction no footer is ever read during planning, so per-column encoding information isn't available for all paths.

Downstream physical operators (FilterExec, AggregateExec) are compiled against the scan's declared output schema during physical planning, before any files are opened. If the scan declares Utf8 but produces Dictionary(Int32, Utf8) at execution time that's a type mismatch.

So when the flag is enabled we promote all string/binary columns to dict at planning time, not just the ones that are actually RLE-encoded, because that's the only way to guarantee schema consistency across all parquet scan paths without introducing file I/O into the planning stage.

I feel like i'm missing something here. if we could take a peak at the parquets metadata before physical planning and change the schema for all operators from the point forward that would be perfect. Im not sure this is currently possible

Rich-T-kid · 2026-08-14T17:35:56Z

@adriangb When the flag is on, DataFusion promotes string and binary columns that are physically RLE_DICTIONARY encoded in the parquet file to Dictionary(Int32, Utf8) or Dictionary(Int32, Binary) at schema inference time. This applies to all parquet scans regardless of how the table was registered.

the PR is ready for review

github-actions Bot added sqllogictest SQL Logic Tests (.slt) common Related to common crate datasource Changes to the datasource crate labels Aug 10, 2026

Rich-T-kid commented Aug 10, 2026

View reviewed changes

github-actions Bot added the auto detected api change Auto detected API change label Aug 10, 2026

Rich-T-kid marked this pull request as ready for review August 10, 2026 15:45

github-actions Bot added proto Related to proto crate documentation Improvements or additions to documentation labels Aug 10, 2026

Rich-T-kid added 6 commits August 13, 2026 10:01

introduce optional rle reads from parquet

10d3b56

fix lint

e152f63

thread flag to protos

c1d232c

update docs/ fix ci

c734b15

fix lint

3d4cba6

regen proto to fix CI formatting check

6e1cbc1

Rich-T-kid force-pushed the rich-T-kid/introduce-rle-parquet-flag branch from 0479f9c to 6e1cbc1 Compare August 13, 2026 15:10

trigger CI

d1273bf

Rich-T-kid force-pushed the rich-T-kid/introduce-rle-parquet-flag branch from 6a4e89d to d1273bf Compare August 13, 2026 15:48

address PR comments

8bc183b

github-actions Bot added the core Core DataFusion crate label Aug 13, 2026

avoid casting non rle-columns

99a9f11

Rich-T-kid marked this pull request as draft August 13, 2026 19:53

Rich-T-kid added 3 commits August 13, 2026 15:59

trim pr

46ee5ac

trim PR to focus on logical plan schema re-write

17480b8

trim test

fc360e8

Rich-T-kid force-pushed the rich-T-kid/introduce-rle-parquet-flag branch from 717d31a to fc360e8 Compare August 14, 2026 17:27

Rich-T-kid marked this pull request as ready for review August 14, 2026 17:36

Navigation Menu

Uh oh!

introduce optional rle reads from parquet - #24227

introduce optional rle reads from parquet#24227
Rich-T-kid wants to merge 12 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/introduce-rle-parquet-flag

Rich-T-kid commented Aug 10, 2026 •

edited

Loading

Uh oh!

Rich-T-kid Aug 10, 2026

Uh oh!

Rich-T-kid Aug 10, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 10, 2026 •

edited

Loading

Uh oh!

Rich-T-kid commented Aug 12, 2026

Uh oh!

adriangb commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026 •

edited

Loading

Uh oh!

adriangb commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Rich-T-kid commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Rich-T-kid Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Rich-T-kid Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Rich-T-kid commented Aug 12, 2026

Uh oh!

adriangb commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriangb commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Rich-T-kid commented Aug 10, 2026 •

edited

Loading

github-actions Bot commented Aug 10, 2026 •

edited

Loading

codecov-commenter commented Aug 10, 2026 •

edited

Loading

Rich-T-kid commented Aug 13, 2026 •

edited

Loading

Rich-T-kid commented Aug 14, 2026 •

edited

Loading