GH-50866: [Python][Parquet] Add use_threads to ParquetWriter to encode a row group's columns in parallel - #50867
Open
anniegracehu wants to merge 1 commit into
Open
Conversation
… encode a row group's columns in parallel Every Parquet write from Python is single-threaded: pyarrow only binds FileWriter::WriteTable, which encodes a row group's columns one after another. The C++ writer can encode them in parallel (ArrowWriterProperties::use_threads), but only on the buffered row group path (NewBufferedRowGroup + WriteRecordBatch). ParquetWriter and write_table get a use_threads argument (default False). With it, write_table opens one buffered row group per row_group_size rows and writes that range's batches into it. Row groups, statistics and the stored schema are the same as WriteTable produces.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Every Parquet write from Python is single-threaded. The C++
FileWritercan encode a row group's columns in parallel (ArrowWriterProperties::use_threads), but only on the buffered row group path (NewBufferedRowGroup+WriteRecordBatch), and pyarrow only bindsWriteTable, which encodes columns one after another. This exposes the C++ writer's parallel column encoding through the Python bindings.What changes are included in this PR?
ParquetWriter(..., use_threads=False)andpq.write_table(..., use_threads=False); the flag is passed toArrowWriterProperties::Builder::set_use_threads.write_tableopens one buffered row group perrow_group_sizerows and writes that range's batches into it withWriteRecordBatch. Row groups, statistics and the stored schema are the same asWriteTableproduces;row_group_sizedefaults and the 64Mi cap are applied the same way. Empty tables and builds without threading take the existing path.ds.write_datasetis not covered; it calls C++WriteTabledirectly.Numbers, 23.0.1, 37-column flat table (18 string, 8 double, 6 bool, 5 int64), zstd, 1M rows, 24-thread pool:
write_table2.5 s;use_threads=True1.1 s at 65,536-row groups (about 3 cores busy; small groups cap it), 0.4 s at 262,144-row groups (about 7 cores).Are these changes tested?
Yes: serial vs threaded write of a chunked table with string, list, struct and dictionary columns and nulls, at four row group sizes, comparing row group lengths, per-column statistics, stored schema and the read-back table; the 1Mi default and 64Mi cap; an empty table; the schema-mismatch error.
Are there any user-facing changes?
New optional
use_threadsargument onParquetWriterandwrite_table. No behavior change without it.This change was written with AI assistance and reviewed by the submitter.