Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
b3471f9
design: buffer-managed dataflow state
DAlperin Jun 10, 2026
ec40649
design: integrate buffer-managed state with differential's Chunk abst…
DAlperin Jun 10, 2026
b0af742
design: incremental migration and swap coexistence
DAlperin Jun 10, 2026
0472cb5
design: swap-backed extent store for filesystem-less nodes
DAlperin Jun 10, 2026
f35e9da
prototype: swap-extent buffer pool, layers 2 and 3
DAlperin Jun 10, 2026
5bd34ef
column-pager: wire buffer pool behind dyncfgs for staging
DAlperin Jun 10, 2026
81b4402
column-pager: log spill mechanism resolution for debugging
DAlperin Jun 10, 2026
f483d57
column-pager: keep tiered singleton configured in pool mode; resolve …
DAlperin Jun 10, 2026
479dece
design: batched-seek cursors as the probe prefetch story
DAlperin Jun 10, 2026
7c829c0
pool: off-worker spill threads for eviction I/O
DAlperin Jun 11, 2026
bf0ce66
compute-types: default pool spill threads to 2
DAlperin Jun 11, 2026
9806605
pool: insert_with writes serialized columns directly into the slot
DAlperin Jun 11, 2026
4cb565c
pool: 64 GiB class reservations; degrade to heap on slot exhaustion
DAlperin Jun 11, 2026
6cbb931
pool: scope slot lifetime to residency, dropping the stable-address i…
DAlperin Jun 11, 2026
bff2b60
pool: single-flight budget enforcement over a resident-only queue
DAlperin Jun 11, 2026
f4d307c
pool: size extents to the compressed payload, not lz4 worst case
DAlperin Jun 11, 2026
a16b9bb
pool: add 4 and 8 MiB size classes for boundary-overshooting chunks
DAlperin Jun 11, 2026
cafb6d8
storage: drain upsert stash seal chunk-at-a-time
DAlperin Jun 11, 2026
2eb7bda
design: record staging prototype findings; add remote-extents direction
DAlperin Jun 11, 2026
a7a6dfc
pool: cleanup pass from complexity review
DAlperin Jun 11, 2026
425d738
storage: tidy upsert drain seal path
DAlperin Jun 11, 2026
673321a
timely-util: collapse column pager config globals
DAlperin Jun 11, 2026
f252b04
pool: rename elided frees to writes elided
DAlperin Jun 11, 2026
a989db9
timely-util: keep short merge-batcher chains resident
DAlperin Jun 11, 2026
dff1ee4
compute: rename pool spill-threads dyncfg to column_paged_batcher_spi…
DAlperin Jun 11, 2026
b16844f
pool: warm slot reuse and hugepage-backed large classes
DAlperin Jun 12, 2026
850ee2c
pool: eager backing to compressed-but-resident off the workers
DAlperin Jun 12, 2026
3626c85
pool: compressed-resident extent tier under an RSS target
DAlperin Jun 12, 2026
fe3ea9a
pool: park eager-backing threads when nothing is backable
DAlperin Jun 12, 2026
e51f8a0
ore/pool: comprehension pass from the simplicity review
DAlperin Jun 12, 2026
44d03ea
design: update buffer-managed-state doc to match the implementation
DAlperin Jun 12, 2026
90425a1
Patch differential-dataflow to the Chunk-API branch
DAlperin Jul 5, 2026
87583d2
timely-util: ColumnChunk, differential's Chunk over pooled columns
DAlperin Jul 5, 2026
3f480c6
storage/upsert: move the v2 stash to differential's ChunkBatcher
DAlperin Jul 5, 2026
bb49c7b
storage/upsert: arrange persist feedback as chunk batches; drain via …
DAlperin Jul 5, 2026
e0a793b
ore/pool: retire the pinned read surface and the pager opt-in seam
DAlperin Jul 5, 2026
5178bb6
compute: move the last paged-batcher consumers to the chunk world
DAlperin Jul 5, 2026
4d5bcd7
Delete the pager: mz_ore::pager, ColumnPager, ColumnMergeBatcher
DAlperin Jul 5, 2026
bf2c858
chunk: review fixes — disjoint-merge fast path, bounded advance cuts,…
DAlperin Jul 5, 2026
67464b6
design: update buffer-managed-state doc for the Chunk-API architecture
DAlperin Jul 6, 2026
b1e94a4
Branch CI: fmt, source-allowlist, and dyncfg-coverage lint fixes
DAlperin Jul 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 15 additions & 17 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -641,6 +641,11 @@ debug-assertions = true
# merged), after which point it becomes impossible to build that historical
# version of Materialize.
[patch.crates-io]
# Chunk-API migration: differential master-next plus the UnloadChunk bulk-read
# capability, from the `unload-chunk` branch.
differential-dataflow = { git = "https://github.com/DAlperin/differential-dataflow.git", rev = "b27ad576c5634afc345b446f81ef686d403b9514" }
differential-dogs3 = { git = "https://github.com/DAlperin/differential-dataflow.git", rev = "b27ad576c5634afc345b446f81ef686d403b9514" }

# Waiting on https://github.com/sfackler/rust-postgres/pull/752.
postgres = { git = "https://github.com/MaterializeInc/rust-postgres" }
tokio-postgres = { git = "https://github.com/MaterializeInc/rust-postgres" }
Expand Down
6 changes: 5 additions & 1 deletion deny.toml
Original file line number Diff line number Diff line change
Expand Up @@ -322,4 +322,8 @@ unknown-registry = "deny"
# remain buildable even if upstream Git repositories disappear. If you don't
# have permissions to create a fork in the MaterializeInc organization, ask in
# #eng-infra on Slack.
allow-org = { github = ["MaterializeInc"] }
#
# TODO: `DAlperin` carries the differential-dataflow Chunk-API prototype branch
# this development branch builds against; fork it into MaterializeInc (or land
# it upstream) before merging.
allow-org = { github = ["MaterializeInc", "DAlperin"] }
Loading
Loading