perf: Reduce record batch memory accounting overhead by ryux1 · Pull Request #24319 · apache/datafusion

ryux1 · 2026-08-13T03:08:10Z

Which issue does this PR close?

Closes Significant overhead in datafusion_common::utils::memory::get_record_batch_memory_size #24310.

Rationale for this change

Record batch memory accounting runs on hot execution paths. The current implementation materializes ArrayData for every array and allocates a hash set even for small batches, adding measurable overhead to queries that frequently update memory reservations.

What changes are included in this PR?

Traverse Arrow arrays directly and recursively count their backing buffers without materializing ArrayData.
Track the first 16 buffer identities inline, then promote to a hash set for wider batches or counters spanning many batches.
Preserve shared-buffer deduplication and full buffer-capacity accounting semantics.
Add parity coverage against the previous ArrayData traversal for primitive, binary/view, list/view, fixed-size, struct, union, dictionary, map, and all legal run-end index layouts.
Add a focused Criterion benchmark across column counts, row counts, and primitive/list/struct layouts.

Criterion point estimates from cargo bench -p datafusion-common --bench record_batch_memory, measured sequentially on upstream main and this PR on the same machine:

Column count (8,192 rows, Int64)

Columns	main	this PR	Speedup
1	81.999 ns	10.645 ns	7.70x
4	308.11 ns	38.669 ns	7.97x
16	1.2543 us	173.55 ns	7.23x
64	4.7457 us	1.3914 us	3.41x

Row count (4 Int64 columns)

Rows	main	this PR	Speedup
1	299.20 ns	39.939 ns	7.49x
128	302.99 ns	38.914 ns	7.79x
8,192	307.64 ns	39.740 ns	7.74x
65,536	310.61 ns	38.994 ns	7.97x

Array layout (4 columns, 8,192 rows)

Layout	main	this PR	Speedup
Primitive Int64	311.01 ns	40.140 ns	7.75x
List of Int64	778.29 ns	82.865 ns	9.39x
Struct of two Int64 fields	1.0072 us	113.70 ns	8.86x

Each list row contains two Int64 values. Each struct column contains two Int64 child fields.

Are these changes tested?

Yes.

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test -p datafusion-common utils::memory --lib (10 passed)
RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
Focused Criterion comparison shown above

Are there any user-facing changes?

No API or behavior changes. This reduces CPU and allocation overhead in record batch memory accounting.

AI assistance: OpenAI Codex assisted with implementation and test execution. I reviewed the change and its behavior end to end.

codecov-commenter · 2026-08-13T06:07:13Z

Codecov Report

❌ Patch coverage is 98.94366% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.14%. Comparing base (8c099dc) to head (890be61).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/common/src/utils/memory.rs	98.94%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24319      +/-   ##
==========================================
+ Coverage   81.13%   81.14%   +0.01%     
==========================================
  Files        1112     1112              
  Lines      388220   388488     +268     
  Branches   388220   388488     +268     
==========================================
+ Hits       314985   315244     +259     
- Misses      54696    54701       +5     
- Partials    18539    18543       +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

comphead

Thanks @ryux1
Would be nice to benchmark also different batch sizes and also nested types

Extend the microbenchmark across row counts and List/Struct layouts. Exercise all legal run-end index types in the ArrayData parity test.

ryux1 · 2026-08-13T18:15:07Z

Thanks @comphead — added in 890be61:

row-count cases for 1, 128, 8,192, and 65,536 rows with four primitive columns
primitive, List<Int64>, and two-field Struct<Int64, Int64> layout cases at 8,192 rows
concrete run-end encoded parity cases for all three legal index widths

The updated A/B tables are in the PR description. The row-count cases stay essentially flat as expected for buffer traversal; measured speedups are 7.49–7.97x for four primitive columns, 9.39x for lists, and 8.86x for structs.

cargo fmt, workspace Clippy with all targets/features, the focused memory tests, the full extended workspace test command, and the expanded Criterion benchmark all pass locally. Ready for another look.

alamb · 2026-08-13T21:10:17Z

Thanks @ryux1 -- I'll check it out shortly

alamb · 2026-08-13T21:10:43Z

run benchmark window_query_sql

alamb · 2026-08-13T21:10:55Z

run benchmark window_query_sql

adriangbot · 2026-08-13T21:16:06Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5286431070-1589-8dnfx 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing perf/batch-memory-accounting (890be61) to 8c099dc (merge-base) diff

Run configuration

run benchmark window_query_sql

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-08-13T21:18:22Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5286433070-1590-l26gz 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing perf/batch-memory-accounting (890be61) to 8c099dc (merge-base) diff

Run configuration

run benchmark window_query_sql

Results will be posted here when complete

File an issue against this benchmark runner

Clarify that the 16-entry threshold avoids allocations for small buffer sets while bounding inline storage and linear lookup. The threshold is a performance heuristic rather than a semantic limit.

adriangbot · 2026-08-13T22:03:19Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing perf/batch-memory-accounting (890be61) to 8c099dc (merge-base) diff

Run configuration

run benchmark window_query_sql

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                             HEAD                                   perf_batch-memory-accounting
-----                                                             ----                                   ----------------------------
window empty over, aggregate functions                            1.00     13.5±0.64ms        ? ?/sec    1.10     14.9±0.28ms        ? ?/sec
window empty over, built-in functions                             1.00    167.3±2.42ms        ? ?/sec    1.02    171.4±2.43ms        ? ?/sec
window order by, aggregate functions                              1.01    913.6±6.32ms        ? ?/sec    1.00    907.0±3.46ms        ? ?/sec
window order by, built-in functions                               1.02    856.7±4.62ms        ? ?/sec    1.00    843.4±2.26ms        ? ?/sec
window partition and order by, u64_narrow, aggregate functions    1.00    282.7±1.07ms        ? ?/sec    1.00    282.3±1.19ms        ? ?/sec
window partition and order by, u64_narrow, built-in functions     1.00    203.9±1.06ms        ? ?/sec    1.00    203.0±0.98ms        ? ?/sec
window partition and order by, u64_wide, aggregate functions      1.01   971.1±91.24ms        ? ?/sec    1.00  960.0±114.07ms        ? ?/sec
window partition and order by, u64_wide, built-in functions       1.03  893.6±111.33ms        ? ?/sec    1.00  869.1±129.59ms        ? ?/sec
window partition by, u64_narrow, aggregate functions              1.05     11.4±0.08ms        ? ?/sec    1.00     10.9±0.08ms        ? ?/sec
window partition by, u64_narrow, built-in functions               1.03     42.5±0.48ms        ? ?/sec    1.00     41.1±0.49ms        ? ?/sec
window partition by, u64_wide, aggregate functions                1.06   584.7±83.86ms        ? ?/sec    1.00   550.3±82.97ms        ? ?/sec
window partition by, u64_wide, built-in functions                 1.01   561.4±48.41ms        ? ?/sec    1.00   553.4±52.97ms        ? ?/sec

Resource Usage

window_query_sql — base (merge-base)

Metric	Value
Wall time	1405.3s
Peak memory	1.4 GiB
Avg memory	399.4 MiB
CPU user	4246.8s
CPU sys	37.0s
Peak spill	0 B

window_query_sql — branch

Metric	Value
Wall time	1415.3s
Peak memory	1.4 GiB
Avg memory	404.9 MiB
CPU user	4147.2s
CPU sys	36.6s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-08-13T22:03:44Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing perf/batch-memory-accounting (890be61) to 8c099dc (merge-base) diff

Run configuration

run benchmark window_query_sql

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                             HEAD                                    perf_batch-memory-accounting
-----                                                             ----                                    ----------------------------
window empty over, aggregate functions                            1.10     15.5±0.52ms        ? ?/sec     1.00     14.2±0.65ms        ? ?/sec
window empty over, built-in functions                             1.01    171.7±1.62ms        ? ?/sec     1.00    170.0±1.58ms        ? ?/sec
window order by, aggregate functions                              1.00    914.6±4.17ms        ? ?/sec     1.00    911.5±4.68ms        ? ?/sec
window order by, built-in functions                               1.01    852.1±6.12ms        ? ?/sec     1.00    841.8±4.41ms        ? ?/sec
window partition and order by, u64_narrow, aggregate functions    1.00    281.2±1.44ms        ? ?/sec     1.01    283.5±2.64ms        ? ?/sec
window partition and order by, u64_narrow, built-in functions     1.00    202.3±0.84ms        ? ?/sec     1.01    203.4±0.91ms        ? ?/sec
window partition and order by, u64_wide, aggregate functions      1.01  1008.6±135.12ms        ? ?/sec    1.00  995.0±107.48ms        ? ?/sec
window partition and order by, u64_wide, built-in functions       1.04  950.2±137.09ms        ? ?/sec     1.00  914.3±132.92ms        ? ?/sec
window partition by, u64_narrow, aggregate functions              1.03     11.3±0.14ms        ? ?/sec     1.00     11.0±0.07ms        ? ?/sec
window partition by, u64_narrow, built-in functions               1.00     41.4±0.49ms        ? ?/sec     1.00     41.3±0.38ms        ? ?/sec
window partition by, u64_wide, aggregate functions                1.02  616.1±100.13ms        ? ?/sec     1.00   602.8±81.05ms        ? ?/sec
window partition by, u64_wide, built-in functions                 1.02   582.0±61.06ms        ? ?/sec     1.00   571.7±58.83ms        ? ?/sec

Resource Usage

window_query_sql — base (merge-base)

Metric	Value
Wall time	1325.3s
Peak memory	1.4 GiB
Avg memory	437.9 MiB
CPU user	4386.7s
CPU sys	38.8s
Peak spill	0 B

window_query_sql — branch

Metric	Value
Wall time	1385.3s
Peak memory	1.5 GiB
Avg memory	406.3 MiB
CPU user	4299.4s
CPU sys	36.4s
Peak spill	0 B

File an issue against this benchmark runner

github-actions Bot added the common Related to common crate label Aug 13, 2026

comphead reviewed Aug 13, 2026

View reviewed changes

ryux1 added 2 commits August 13, 2026 19:39

perf: reduce record batch memory accounting overhead

0c51f33

bench: add row-count and nested memory accounting cases

890be61

Extend the microbenchmark across row counts and List/Struct layouts. Exercise all legal run-end index types in the ArrayData parity test.

ryux1 force-pushed the perf/batch-memory-accounting branch from e584d8b to 890be61 Compare August 13, 2026 18:14

comphead reviewed Aug 13, 2026

View reviewed changes

Comment thread datafusion/common/src/utils/memory.rs

document(common): explain inline buffer capacity

db959c3

Clarify that the 16-entry threshold avoids allocations for small buffer sets while bounding inline storage and linear lookup. The threshold is a performance heuristic rather than a semantic limit.

Navigation Menu

Uh oh!

perf: Reduce record batch memory accounting overhead - #24319

perf: Reduce record batch memory accounting overhead#24319
ryux1 wants to merge 3 commits into
apache:mainfrom
ryux1:perf/batch-memory-accounting

ryux1 commented Aug 13, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 13, 2026 •

edited

Loading

Uh oh!

comphead left a comment

Uh oh!

ryux1 commented Aug 13, 2026

Uh oh!

alamb commented Aug 13, 2026

Uh oh!

alamb commented Aug 13, 2026

Uh oh!

alamb commented Aug 13, 2026

Uh oh!

adriangbot commented Aug 13, 2026

Uh oh!

adriangbot commented Aug 13, 2026

Uh oh!

Uh oh!

adriangbot commented Aug 13, 2026

Uh oh!

adriangbot commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

ryux1 commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Column count (8,192 rows, Int64)

Row count (4 Int64 columns)

Array layout (4 columns, 8,192 rows)

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

ryux1 commented Aug 13, 2026

Uh oh!

alamb commented Aug 13, 2026

Uh oh!

alamb commented Aug 13, 2026

Uh oh!

alamb commented Aug 13, 2026

Uh oh!

adriangbot commented Aug 13, 2026

Uh oh!

adriangbot commented Aug 13, 2026

Uh oh!

Uh oh!

adriangbot commented Aug 13, 2026

Uh oh!

adriangbot commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ryux1 commented Aug 13, 2026 •

edited

Loading

codecov-commenter commented Aug 13, 2026 •

edited

Loading