Skip to content

int_proxy join: draw the proxy collection lazily, a block at a time - #809

Merged
frankmcsherry merged 3 commits into
TimelyDataflow:masterfrom
frankmcsherry:int-proxy-lazy-join
Jul 29, 2026
Merged

int_proxy join: draw the proxy collection lazily, a block at a time#809
frankmcsherry merged 3 commits into
TimelyDataflow:masterfrom
frankmcsherry:int-proxy-lazy-join

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

The proxy join performed the whole of a unit's work inside prep, and returned an iterator that spilled the finished containers.
The driver's fuel budget was therefore spent after the fact.

The backend interface

present0/present1 are replaced by a single advance:

fn advance(
    &mut self,
    instance: &JoinInstance<B0, B1>,
    from: &mut Option<u64>,
    bridge0: &mut ProxyBridge<B0::Time, Self::R0>,
    bridge1: &mut ProxyBridge<B0::Time, Self::R1>,
);

It populates both bridges with the intersecting keys from from onward and updates from.
This removes the two-phase presentation, in which the whole fresh side was materialized to derive the key filter for the other side.

A unit's progress through the key space rides in from rather than in the backend, so the harness can interleave the calls of the many units it has in flight.
A key hash must be reported entirely by the call that first mentions it.

cross populates a Vec<Self::Output> rather than returning one container, and takes its times and diffs by &mut Vec<_> so the harness keeps the allocations.
Both granularities are the backend's: advance sizes the block it draws, cross the containers it cuts.
The operator's own JOIN_CHUNK policy is gone, along with the mid-key flushing it needed; the operator draws a block, merge-matches it whole, and hands the matches back.
A single key's matches are held whole, as under the cursor tactic.

JoinInstance owns its batches, which drops its lifetime parameter and lets the iterator hold it as a single field.

Checking

Debug assertions enforce the advance contract: bridges sorted and consolidated, from strictly increasing or exhausted, and each reported key hash confined to the block that first mentions it.
The last guards the case that would otherwise be silently wrong rather than a panic, and the from check also guards liveness.

Nothing implements ProxyJoinBackend, here or elsewhere, so this is compile-checked only, as the framework was before.
A reference backend and a test asserting that the block-wise result matches the single-block result would be the natural follow-on.

Known gaps

Two questions this leaves open, both about what a backend can rely on:

  • The trait says the harness "interleaves their calls freely", but the code guarantees more: advance and the cross consuming its block are adjacent within one next(), and no other unit can interpose. That adjacency is what makes ephemeral per-block value_id assignment workable, and the doc currently reads as forbidding it.
  • There is no per-unit identity and no completion signal, so a backend cannot cache a cursor across the blocks of one unit, and gets no hook when the driver drops an unfinished iterator. ProxyReduceBackend's begin/next_window/finish session provides both.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ErrnMtBsvrvncVeYL51TMv

frankmcsherry and others added 2 commits July 29, 2026 13:06
The proxy join performed the whole of a unit's work inside `prep`, and returned an
iterator that spilled the finished containers.
The driver's fuel budget was therefore spent after the fact.

`present0`/`present1` are replaced by a single `advance`, which populates both bridges
with the intersecting keys from `from` onward and updates `from`.
This removes the two-phase presentation, in which the whole fresh side was materialized
to derive the key filter for the other side.
A unit's progress through the key space rides in `from` rather than in the backend, so
the harness can interleave the calls of the many units it has in flight.

`cross` populates a `Vec<Self::Output>` rather than returning one container.
Both granularities are the backend's: `advance` sizes the block it draws, `cross` the
containers it cuts.
The operator's own `JOIN_CHUNK` policy is gone, along with the mid-key flushing it
needed; the operator draws a block, merge-matches it whole, and hands the matches back.
A single key's matches are held whole, as under the cursor tactic.

`JoinInstance` owns its batches, which drops its lifetime parameter and lets the
iterator hold it as a single field.

Debug assertions enforce the `advance` contract: bridges sorted and consolidated, `from`
strictly increasing or exhausted, and each reported key hash confined to the block that
first mentions it.
The last guards the case that would otherwise be silently wrong rather than a panic, and
the `from` check also guards liveness.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErrnMtBsvrvncVeYL51TMv
…ontract

`cross` takes a single `JoinMatches`, whose `ids` carry `(key, (val0, val1))` rather
than repeating the key hash in two parallel arrays of pairs.
`join_key` fills it directly, in place of four loose vectors threaded through every
call.

`JoinInstance::lower` is a time rather than a one-element antichain.
It was only ever built from the capability's time, and a backend consolidating loaded
times wants `join_assign`, not `advance_by` against a singleton frontier.

`work` walks the two bridges in lockstep rather than merging them.
`advance` reports the intersecting keys, entirely and from both inputs, so the runs
carry the same keys in the same order; the previous merge silently skipped a
mismatch, which is a protocol violation that produces a wrong answer rather than a
failure.
Debug assertions now report it, and the loop keeps both bounds so a violating backend
truncates instead of indexing out of bounds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@frankmcsherry
frankmcsherry force-pushed the int-proxy-lazy-join branch 2 times, most recently from c1852b6 to f9c9319 Compare July 29, 2026 21:37
@frankmcsherry
frankmcsherry merged commit aa8745f into TimelyDataflow:master Jul 29, 2026
6 checks passed
@frankmcsherry
frankmcsherry deleted the int-proxy-lazy-join branch July 29, 2026 21:46
frankmcsherry added a commit that referenced this pull request Jul 30, 2026
* Integer-proxy chunks: backend-agnostic join and reduce over (key_hash, value_id)

A boundary where only integers cross: a storage backend presents each
record as ((key_hash, value_id), time, diff) — integer proxies for data
it keeps in its own layout — the operators own all the lattice/time
logic over those integers, and the backend supplies value semantics via
callbacks. Any columnar (or otherwise opaque-to-DD) value store can
then reuse join and reduce without materializing values.

- trace/chunk/int_proxy: ProxyChunk, a cursor-less Chunk of proxy
  columns, with from_unsorted (integer sort+consolidate with
  representative provenance) as the presentation-building helper.
- operators/int_proxy: ProxyJoinTactic / ProxyReduceTactic for the
  join_with_tactic and reduce_with_tactic seams (made pub here), and
  the backend traits: present-as-proxies (read), value callback with
  hash-minted output ids (write), materialize (egress). Reduce output
  ids are content hashes, so an output arrangement re-presents with the
  same ids downstream with no registry; pending interesting times are
  keyed by the stable key_hash across retires. The module doc carries
  the boundary contract and design notes (why value_id is not
  order-preserving; collision risk); each tactic and the in-memory
  reference backend (VecChunk arrangements, fnv hashes) sit in their
  own file under the module.
- Tests: join and count/distinct/min reduces against the row operators
  over multi-round retracting inputs, and a scripted Product-time
  retire sequence exercising synthetic corrections and pending.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* int_proxy: state the minimal value_id contract — hashing is one scheme, not a requirement

value_id never outlives one computation (a join unit's presentations;
one reduce retire, whose materialize resolves ids to real data before
anything leaves). The actual contract is a per-computation bijection
with value equality, plus within-retire agreement between the output
presentation and minted ids. Content hashing discharges all of it
statelessly (the reference backend's choice); exact schemes — dense
ordinals from grouping, a per-retire value→id map — are equally valid
and collision-free. Only key_hash must be a stable pure function of the
key (cross-retire pending, changed-key filter), making the key side the
irreducible collision exposure. Persisting ids into an output
arrangement itself would force stable value ids; this design does not.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* int_proxy: delta-proportional presents, checked — not asymptotically worse than the cursor tactics

Two regressions found by asking exactly that question, both fixed:

- The reference reduce backend implemented the changed-key restriction
  by scanning every batch and filtering — O(trace) per retire where the
  cursor tactic seeks, O(delta·log). Presents now seek the changed keys
  (novel keys resolve from this retire's delta-sized input batches;
  pending keys from the retire that pended them, which is exactly what
  the persistent hash→key map retains — pruned to the changed set each
  retire, so it is bounded by the delta, and the per-retire value map
  is cleared).

- The join interface had no restriction at all, forcing ANY backend to
  present the entire accumulated side per fresh batch — O(trace·log)
  per unit where the cursor join seeks. present0/present1 now take an
  optional sorted key-hash filter; the tactic presents the fresh side
  first and passes its key set for the accumulated side, the join
  analogue of reduce's changed-key restriction.

The check is mechanical, not wall-clock: counting backend wrappers
measure presented records — with 20k arranged keys and five single-key
rounds, reduce presents 37 records and join 10, where the scanning
versions present Ω(rounds·N) (join: 100,005) and fail the gate. What
remains above the cursor tactics is a log-factor sort of delta-sized
presentations, and per-(key, wave) rescans of a changed key's
presented range where the row replayer consolidates progressively —
same worst-case order, different constants.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* int_proxy: the self-contained assessment — identity backend, grid-oracle fuzz

The minimal harness for the framework, closed over itself: the tactics
demand only BatchReader of the trace, and ProxyChunk is already a
Chunk, so a batch of proxy chunks is the minimal arrangement — the
proxy data IS the data, no separate chunk class needed. The identity
backend makes values u64s with the identity as the id function: no
hashing, no resolution machinery, no collision possibility, and
materialize emits the proxy records verbatim (incidentally exercising
ChunkBatch<ProxyChunk> as a real output batch).

What remains under test is exactly the framework's own contribution —
interesting-time discovery, desired-vs-current deltas, pending, held
routing — fuzzed over Product-time grids: 300 random inputs retired
through random diagonal frontiers (so synthetic joins arise inside and
across intervals and must pend), driven through an emulation of the
reduce driver protocol, and checked against a brute-force oracle at
every grid point: the accumulated output must equal the reduction of
the accumulated input, everywhere, for count/distinct/min-shaped
reducers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* int_proxy: wall-clock bench; fix a capacity leak the bench found

An ignored benchmark (cargo test --release --test int_proxy -- --ignored
--nocapture) compares full stacks — the stock row operators against
chunk arrangements plus the proxy tactics over the reference backend —
for a bulk load and a steady-state incremental phase (warmed past the
post-load merge-amortization transient).

The bench caught what the counting gates could not: the reference
backend's hash→key map was pruned by retain (and the per-retire value
map by clear), both of which keep the backing table — so after a
million-key load, every retire walked a million-bucket table to visit
one entry, ~130µs/round of pure capacity. shrink_to_fit after the
prune and a fresh map per retire fix it.

Steady-state single-key rounds after the fix, proxy vs row: reduce
~1.4x at every scale (flat from 10k to 1M keys — the delta-
proportionality gates hold in wall clock too), join at parity below 1M
(~1.0x). Bulk load carries the presentation layer's constant factor
(per-record hashing, clones, by-hash sort): reduce 3-5x, join ~2x —
the costs a columnar backend's bulk primitives are meant to attack.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* int_proxy: batch the reduce value callback per wave (reduce_many)

The longer-term goal for the boundary is fewer crossings into the
backend's value logic (interpreted or columnar execution pays per-call
overhead): each crossing should carry a list of keys and a longer
bracketed list of value entries.

ProxyReduceBackend gains reduce_many(keys, ends, input) — group_offsets
-shaped brackets, one per key, each non-empty — returning concatenated
per-key outputs with their own bracket ends. A default implementation
loops the per-key reduce, so simple backends implement only that;
backends with bulk value logic override reduce_many.

The tactic now calls only reduce_many: retire's key-major loop becomes
two passes — derive each changed key's active times, group the work
into waves by time, then play the waves in ascending order (Ord extends
the partial order, so a key's earlier deltas always precede a later
time's reads) with at most one callback per wave, batching every key
active at that time.

The identity backend overrides reduce_many (asserting the bracket
protocol from the backend's seat), so the grid-oracle fuzz exercises
the batched path; the reference backend uses the default, so the
row-comparison tests cover the loop. All gates and benches unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* int_proxy: one value-callback crossing per retire, not per wave

The desired output at a (key, time) moment is a function of the key's
input accumulation at that time alone — no output-side state — so no
time ordering constrains the batch: every moment of the retire can
share one reduce_many call, a key contributing one bracket per active
time. The order-sensitive part (subtracting the current output, which
includes deltas emitted at earlier moments) is pure proxy-space
arithmetic and moves to a separate pass that plays the moments in
ascending time order.

The bracket, not the key, is reduce_many's unit: keys may repeat.
Contract docs updated accordingly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* int_proxy: port the history-replay machinery — the scaling shapes go linear

The row suite's reduce_scaling/join_scaling shapes (one key, many
distinct times, one batch) exposed quadratic behavior in both proxy
tactics: reduce rescanned a key's full presented range per interesting
time (and its interesting-time closure joined all pairs), and join
cross-produced full matched histories pairwise. Measured: reduce 6.4s
at scale 10k, 4.7x per doubling; join 9.8s at 10k, >90s at 20k.

The robust versions are the cursor variants with integers in place of
keys and values, as intended:

- history.rs: IdHistory, the id-space ValueHistory — (value_id, time,
  diff) edits replayed in ascending time order into a buffer repeatedly
  advanced by the meet of the times still to come and consolidated.
  The advancement is the collapse that keeps a key with many distinct
  times linear: accumulations read the small buffer, never the raw
  history. The presentations serve as the fused per-key load.
- reduce: discover_and_accumulate ports history_replay::compute —
  lazy interesting-time discovery (novel and pending seed; synthetics
  from joins with the advanced batch buffer and times_current) replaces
  the eager join-closure, and per-moment accumulation reads the
  advanced buffers. Phase B replays the output side per key over the
  discovered moments with the same machinery (suffix meets; emitted
  deltas advanced and consolidated). Still one batched reduce_many
  crossing per retire.
- join: join_key ports JoinThinker::think — each side's edits replayed
  against the other side's advanced buffer (identical emitted times:
  t0 ∨ (t1 ∨ meet) = t0 ∨ t1), with the dead-simple cross product kept
  for small histories.

proxy_reduce_scaling / proxy_join_scaling (scale 100k, the row tests'
shapes) now pin this; scale 10k dropped from seconds to milliseconds
and growth is ~4x per 4x scale. The grid-oracle fuzz over partially
ordered times, the scripted pending test, the row comparisons, the
delta-proportionality gates, and the steady-state bench all pass
unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* int_proxy: materialize in TARGET-sized chunks — the last scaling term

The row-vs-proxy comparison on the scaling shapes showed proxy reduce
still superlinear (9x per 4x scale; timeout at 4M where row takes
0.5s). Profiling pinned it outside the tactic: the reference backend's
materialize built ONE giant VecChunk and let the builder settle it,
and settle's split path peels TARGET-sized pieces off the front with
split_off, copying the remaining tail each iteration — O(m²/TARGET)
in the batch size. Feeding the builder TARGET-sized chunks directly
fixes it: 4M drops from >120s to 3.2s.

With that, both operators are in the row implementations' complexity
class on the scaling shapes (growth ~5x per 4x scale ≈ n·log n; the
hot frames are the presentation sort and fnv hashing — constants, not
structure): at scale 4M, join row 0.96s / proxy 3.6s, reduce row 0.48s
/ proxy 3.2s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* interactive: CorgiChunk arrangement backend (cursor-less Route B on the #778 Chunk split)

A columnar differential-dataflow backend for DDIR whose native representation
is corgi Value columns and whose scalar logic is corgi `eval_graph`, parallel
to `backend::vec`.

Arrangement:
  - CorgiChunk : Chunk (NOT NavigableChunk) — corgi columns + Vec time/diff,
    ordered (key,val) by corgi structural order then time. merge/extract/advance/
    settle ported from the VecChunk reference; drives corgi's discrimination sort
    (sort_perm) + batched compare_idx, never per-pair. Rides #778's cursor-less
    Chunk path, so it gets the fueled/graded ChunkBatchMerger for free.
  - Tactics (Route B): cursor-less reduce (incremental key selection over the
    input delta); join via corgi `find` (find_ranges → equal-range merge-join,
    multi-record). as_collection reads columns directly.

differential-dataflow proper: widen the JoinTactic/ReduceTactic/Fresh/
join_with_tactic/reduce_with_tactic seam from pub(crate) to pub, so an
out-of-crate tactic can be implemented at all. (Candidate for upstreaming as
the public extension point the #773 tactics were designed to be.)

State: all 6 canonical programs match `vec`; 33 lib tests pass. reach ~1.9×
slower than vec, compute-bound linear ~3× FASTER (columnar eval avoids the
row backend's ~22% Value::cmp pointer-chasing). The remaining reach gap is
entirely the reduce, still row-wise — the only operator not yet columnar.

Notes / follow-ups: corgi_arrange.rs (old CorgiBatch impl) is superseded by
corgi_chunk.rs and retained only as reference; scratch examples alongside the
corgi_perf/corgi_progs/corgi_prof harnesses. Depends on frankmcsherry/wip
corgi branch dd-arrange-api.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* interactive: operator scorecard — per-operator corgi-vs-vec perf triage

A standing harness that isolates each backend operator in a minimal DDIR
program, asserts corgi == vec, and reports the corgi/vec ratio against a
target. `PROG=<name> BACKEND=<corgi|vec>` loops one case for samply.

Baseline (n=100k linear / 20k arrange-y):
  map8            0.30x  ✓ (columnar compute, no row boundary)
  filter          0.75x  ✓
  arrange         1.5x   ✗   \
  join            1.5x   ✗    >  every operator with a columns<->rows
  reduce_distinct 2.0x   ✗   /   transcode boundary loses
  reduce_count    2.0x   ✗
  reach           2.0x   ✗

Corrects an earlier whole-program-profile conclusion ("only the reduce is
slow"): per-operator, arrange and join are independently ~1.5x — the shared
cost is the transcode boundary, not one operator. Beating vec means removing
row boundaries (chunk-native ingest, columnar operator emit).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* interactive: remove arrange/join transcode boundaries — both now beat vec

The columns↔rows transcodes at operator edges were self-inflicted, not
inherent. Removed all three:

  - as_collection: read chunk columns straight into a CorgiContainer
    (chunks_to_columns), no untranscode→from_updates round-trip.
  - join emit: the projection already yields corgi columns; emit them via
    give_container into a CapacityContainerBuilder<CorgiContainer> and drop the
    JoinToCorgi unary — no untranscode + per-row give + re-transcode.
  - arrange ingest: CorgiChunker (a ContainerBuilder) sort-consolidates each
    input CorgiContainer's columns directly into CorgiChunks — replacing
    ContainerChunker's drain-to-rows + VecMerger + row-based builder. It
    ACCUMULATES to TARGET before consolidating, so it emits few large chunks,
    not one tiny chunk per input container (else the columnar per-chunk set-up
    dominates on many small batches).

Scorecard (corgi/vec): arrange 1.5x→0.9x (BEATS vec), join 1.5x→0.7x (BEATS
vec), reduce_distinct/count 2.0x→1.35x, reach 1.9x→1.45x. All 6 programs match
vec; 33 lib tests pass. Remaining reduce gap is the row-wise reducer (Rust over
Value rows) — next: columnar consolidate via corgi group→fold_add→filter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* interactive: columnar consolidate building block (proven, unit-tested)

columnar_sum_by_key: the monoid Count/consolidate fold as pure corgi ops
(group → map(fold_add) → filter via parse_ml + eval_graph), no Value rows.
Signed diffs passed as raw two's-complement u64 bits; wrapping fold_add gives
the correct i64 sum, `ne 0` the sign-agnostic zero-drop. Unit test:
[(1,+1),(1,-1),(2,5),(3,9),(3,1)] -> [(2,5),(3,10)] (net-zero key dropped).

Groundwork for the wave-based columnar reduce; not yet wired into the live
reduce (which stays row-wise). Depends on corgi dd-arrange-api wrapping-Reduce.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* interactive: Count reduce fast-path — skip the unnecessary per-value consolidation

Count = Σ diffs ≤ t, and summing the consolidated per-value diffs equals summing
the raw diffs — so for Count the consolidate_vals (a Value sort + clones) is pure
waste. Sum diffs directly; no Value churn. reduce_count 1.39x→1.30x, all 6
programs match vec.

Distinct/Min still consolidate per value (they reason about individual values) —
that's the corgi two-level group→fold→filter path next; Collect stays row-wise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* interactive: columnar Distinct consolidate building block (two-level, tested)

columnar_distinct_keys: the keys with a present value (net diff != 0, the
nonzero simplification) as pure corgi ops — group by the (key,val) composite,
fold_add raw diffs, drop net-zero, distinct keys of survivors. group-by-(key,val)
is just group-by a projected composite; the nonzero test keeps everything on the
raw-bit `ne 0` path (no signed encoding needed yet). Unit test:
(1,10):+1,-1 → absent; (2,20),(2,21),(3,30) → present → keys [2,3].

Both monoid consolidate blocks now proven (sum for Count, two-level for
Distinct). Remaining: the wave integration (feed many keys per wave, driven by
the Rust time logic; reuse the existing delta/emit loop).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* interactive: group_offsets — the integers-only Rust<->corgi boundary primitive

Group a key column by identity, returning only integers: `perm` (row indices in
group order) + `ends` (per-group exclusive end within perm). DD walks groups by
index — group g is perm[ends[g-1]..ends[g]] — and drives its time/diff logic
without ever seeing a key Value; keys and payloads stay columnar in corgi. This
is the boundary model (corgi presents abstract int ids tied to times/diffs DD
sees; DD hands back indices to include). Built on sort_perm + one batched
compare_idx. Tested.

Foundation for the path-2 columnar reduce (wave loop over group ranges). Note:
cross-retire `pending` needs a STABLE key id (a u64 hash), since group indices
are per-retire — converges with hashed-keys-as-u64.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* SPIKE.md: 2026-07-02 checkpoint — Chunk backend, transcode removal, boundary model

Records: arrange & join now BEAT vec (transcode boundaries removed); multi-record
primitives exposed; reduce consolidate blocks + group_offsets proven; and the
design capture for a data-blind reduce tactic (framework presents as int-id/
time/diff, owns time navigation; backend supplies id-mapping + value callback;
output ids are hashes — mint = hash the new value).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* SPIKE.md: corgi-backend hand-off — int_proxy framework (PR #781) consumption plan

Branch rebased onto int-proxy: the framework the SPIKE's boundary model
called for now exists upstream (renamed ids -> int_proxy in review);
the old in-branch copy is dropped. The hand-off section documents the
current interfaces, the one-crossing-per-retire reduce_many contract,
the assessment harness to replicate, the reduce-first plan, the traps
already hit once (scan-presents, capacity leaks, giant-chunk settle),
and the suitability questions the corgi agent should answer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* corgi ProxyReduceBackend: columnar reduce over int proxies

Implement ProxyReduceBackend for corgi and swap CorgiReduceTactic ->
ProxyReduceTactic. The DD tactic owns all time/lattice logic over
(key_hash, value_id, time, diff); the corgi backend supplies only:
 - hashing: key_hash/value_id via corgi::arrange::hash_rows over the corgi
   key/val COLUMNS (columnar, content-addressed; DD never hashes).
 - reduce_many: ONE crossing per retire over every (key,time) bracket.
   Count/Distinct are integer-only (no value untranscode); Min/Collect
   gather+untranscode the bracket values once, never per key. Output ids
   minted by hashing the produced value column (agrees with present_output).
 - present_input/present_output: read arrangement chunks, restrict to changed
   keys by columnar semijoin (novel whole, history filtered); delta-proportional.
 - materialize: resolve ids -> real (key,val) rows, seal a CorgiChunk batch.

corgi_progs: all 6 match vec (scc min/recursion + unnest collect included).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* corgi reduce: burn the transcode — fully columnar id resolution

Replace the DValue resolution maps (HashMap<u64,DValue>) with corgi COLUMN
pools + integer id->row-index maps. The real keys/values never leave corgi
columns:
 - present_input/output register representative key/val columns into per-retire
   pools (key_index/val_index -> offsets into concatenated blocks), no untranscode.
 - reduce_many builds each reducer's output value COLUMN directly: Count -> u64
   prim, Distinct -> Unit, Min -> gathered input rows (id reused from input),
   Collect -> List. Ids minted by hash_rows over the built column. No transcode.
 - materialize resolves proxy ids to columns by gather + columns_to_batch
   (new corgi_chunk helper), column-native egress (no from_rows transcode).

The only residual untranscode is Min/Collect's value ORDERING (the reduction
contract is DDIR Ord, which is not corgi's structural order); the chosen rows
are still taken columnar.

Scorecard corgi/vec: reduce_distinct 1.01x -> 0.92-0.96x (BEATS vec),
reduce_count 1.03x -> 0.90-0.93x (BEATS vec), reach 1.10-1.22x -> 1.06-1.13x.
corgi_progs: all 6 match vec; interactive lib tests green (36).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* corgi reduce: Min/Collect order via corgi sort_blocks — zero transcode

Replace the last untranscode (Min/Collect value ordering) with one columnar
corgi sort_blocks per retire:
 - Min: gather positive-diff candidates, segment by bracket, sort_blocks ->
   each bracket's argmin = perm[block_start]; output row taken columnar,
   reusing the input value_id.
 - Collect: segment entries by bracket, sort_blocks -> sorted run per bracket,
   expanded by diff into a List column.

Uses corgi STRUCTURAL order, which equals DDIR Ord for the non-negative
scalar/tuple values these reductions see (all 6 programs); diverges only for
negative ints (unsigned leaf compare) and list-valued compares (length-first),
neither of which arises here — documented as an order invariant.

Needs corgi::arrange::sort_blocks (exposed on fm/corgi dd-arrange-api 9b41cdc,
the segmented sort re-exported from ops::cmp::order). The reduce backend is now
entirely transcode-free. corgi_progs: all 6 match vec; lib tests green (36);
reduce still beats vec (distinct 0.92-0.95x, count 0.93-0.94x).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* corgi reduce: identity hasher for the u64 id-index maps

key_index/val_index keys are already well-distributed hash_rows u64s, so
siphash on every register/lookup was wasted (~7% of the reduce tactic in
profiling). Pass the id through unchanged. reduce_count 0.93x->0.88x;
distinct/reach within noise (their id-maps are small). Gate green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* int_proxy: MSD-bucket sort in ProxyChunk::from_unsorted

The proxy records sort by (key_hash, value_id, time); key_hash is a full 64-bit
content hash, so one MSD counting pass on its high byte splits into 256
near-uniform buckets, each finished by a comparison sort on the full key. One
linear pass replaces most of the n log n (falls back to the plain sort for
n<512 or clustered hashes). Profiling: from_unsorted 20% -> 8% of the reduce,
driftsort gone. Wall-clock ~flat (0.89->0.90x) — the reduce is allocation-bound
(malloc 33-38%), so this just uncovers that as the next lever. Framework
(int_proxy + reduce_reference) tests green; corgi_progs all 6 match vec.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* int_proxy reduce: reuse per-key scratch across the retire

retire allocated fresh Vecs + an IdHistory for EVERY changed key (58% of the
reduce's Vec-growth in profiling, IdHistory another 17%). Hoist the transient
buffers out of the per-key loops and clear/reload them (IdHistory::load already
clears+refills, retaining capacity): rep, raw_moments (drain), and phase-B
meets, out_replay, emitted. Pure capacity reuse, no semantic change.

Scorecard: reduce_distinct/count 0.90->0.85-0.86x, reach 1.06->1.00x (parity)
at n=4000. Allocation was the real lever (sort CPU wasn't). Framework tests
(int_proxy + reduce_reference) green; corgi_progs all 6 match vec.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* int_proxy reduce: reuse the per-moment delta map (BTreeMap -> reused IdMap)

Phase B allocated a fresh BTreeMap per (key, moment) for the desired-vs-current
delta. Replace with one HashMap keyed by an identity hasher (value_ids are
already u64 content hashes), hoisted above the loop and cleared per moment
(retains capacity, no per-moment alloc/free). Iteration order is irrelevant —
deltas are consolidated and the output batch is sorted by from_unsorted.

Scorecard: reduce_count 0.86->0.85x, reach 1.06->0.98x at n=4000 (below parity).
Framework tests (int_proxy + reduce_reference) green; corgi_progs all 6 == vec.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* corgi_perf: add end-to-end SCC case (corgi vs vec)

SCC is the sharp recursive case (nested fwd/bwd label propagation via min,
negation, 4 joins). Findings: corgi is ~1.9-2.5x slower than vec and the gap
GROWS with n (unlike reach, which reaches parity) — scc's nested-recursion /
negation / min structure scales worse for corgi. And corgi DIVERGES from vec on
larger random graphs (n>=1000: 1297 vs 1292) — a corgi-backend correctness bug
(the proxy reduce tactic + reference backend pass the scc oracles in
reduce_reference.rs, and it is NOT the min value-ordering — tested with a
DValue-order argmin). Harness flags the mismatch instead of panicking.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Isolate SCC bug to corgi (not the int_proxy tactic)

Two debugging tools:
 - reduce_reference.rs: add reduce_int_proxy (int_proxy ProxyReduceTactic + Vec
   value backend over VecChunk, NO corgi) and scc_int_proxy_* tests that run SCC
   with only the reduce swapped to int_proxy, vs the cursor reduce. They PASS up
   to 50 nodes/120 edges/20 seeds — so the int_proxy reduce TACTIC is correct at
   SCC's nested-product-time depth. (reduce_reference's own scc_* compare cursor
   vs the *stock* reference tactic, never int_proxy — this fills that gap.)
 - interactive corgi_scc_min.rs: sweep+delta-debug an SCC corgi-vs-vec divergence
   to a minimal graph. Finds nodes=4, 5 edges [(0,0),(0,2),(1,3),(2,1),(2,2)]:
   corgi=3 vs vec=2.

Conclusion: the divergence is in the CORGI backend, not the int_proxy work. The
int_proxy tactic + Vec values reproduces SCC exactly; swapping in corgi is what
breaks it. Next: narrow within corgi (reduce value backend vs join vs negate).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Narrow corgi SCC bug to the outer recursion (not scopes/reduce structure)

Four minimizer/trace harnesses (interactive examples):
 - corgi_scc_min: minimal SCC divergence = 4 nodes, [(0,0),(0,2),(1,3),(2,1),(2,2)].
 - corgi_scc_trace: on that graph, export each SCC intermediate. Result:
   fwd::labels OK, trim_fwd OK, bwd::labels DIFF (node 2 -> label 1 in corgi vs
   2 in vec), trim_bwd/scc DIFF. So the divergence enters at the backward min-
   reduce, structurally identical to the correct forward one.
 - corgi_cc_min: single-level fwd min-label propagation NEVER diverges (to 80
   nodes) -> not single recursion.
 - corgi_bwd_min: two-pass fwd->trim->bwd with NO outer loop NEVER diverges ->
   not the chained nested scopes either.

Conclusion: the bug REQUIRES SCC's outer recursion (the iterate wrapping fwd/bwd
-> 3-deep product times, plus negation  and feedback
). Symptom is a failed retraction (stale label survives).
Suspects: corgi 3-level dynamic-time handling (leave_dynamic/results_in/enter_at)
or negation under recursion. Not int_proxy (proven), not the reduce structure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Pin corgi SCC bug to the reduce via update-stream diff

corgi_scc_stream: emit the full (data,time,diff) stream of SCC intermediates via
inspect, per backend; diff consolidated by (data,time) to find the first moment
of divergence (DD is deterministic at all times).

Result on the minimal 4-node graph: fwd::labels and trim_fwd streams AGREE
exactly (the 16-vs-10 raw counts were +1/-1 that consolidate away). The bwd
min-reduce: its INPUT (proposals+nodes) diverges only at inner-times 258/513,
but its OUTPUT diverges at 257/512 — one tick earlier. So at time (0,[1,257])
the reduce input agrees while the output diverges (vec retracts node2->1, corgi
does not). By the inputs-agree-output-differs test, the bug is IN the corgi
min-reduce. The int_proxy tactic is proven on SCC (scc_int_proxy), so it's the
CorgiReduceBackend (present/materialize/changed-key), not the tactic. The
forward min-reduce (same code) is correct -> a data-specific edge case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Pin corgi SCC bug: reduce-input present/arrange LOSES a record

Instrumented CorgiReduceBackend::present_input (env CORGI_DBG) to dump node-2's
presented input on the minimal repro, and compared to vec's true input stream:
for (node2, label1), vec has [0,257]+1 [0,258]+1 then [1,257]-1 [1,258]-1
(added round 0, retracted round 1 -> gone); corgi's present has [0,257]+1
[0,258]+1 [1,258]+1 -- the round-1 retractions are MISSING (plus a spurious +1).
And corgi's input STREAM (bwdin) agrees with vec at [1,257] (both -1), so the
retraction enters the arrange but present_input never returns it. => corgi loses
a reduce-input update when arranging/presenting under nested times; the min then
reads a non-retracted label1 and stays stale = 1. Not the min logic, not the
int_proxy tactic (proven) -- the reduce-input arrangement (CorgiChunk) or
present_input's changed-key/novel-history read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Complete SCC diagnosis: compaction cancels a novel retraction, erasing an interesting time

Full per-retire dump (CORGI_DBG in present_input/present_output/materialize) on
the 4-node repro nails the mechanism:
 1. corgi's reduce-input TRACE compacts the round-0 add (node2,label1)[0,257]+1
    (present in the input stream) forward to [1,257]+1.
 2. the novel retraction (node2,label1)[1,257]-1 then EXACTLY CANCELS it inside
    present_input's consolidation (from_unsorted drops the net-0 record) -> the
    (node2,label1) record at time [1,257] DISAPPEARS from the presentation.
 3. the int_proxy tactic seeds interesting times from the presentation, so with
    [1,257] erased it never re-evaluates node2's min there and never emits the
    retraction of node2->1 (materialize shows label1 only ever emitted +1, never
    -1) -> node2->1 is stuck -> wrong SCC.
The reference (VecChunk) passes SCC because it doesn't compact that record onto
[1,257] at that retire, so the novel retraction stays a distinct interesting
time. Root: trace compaction advancing a history record onto the same time as an
incoming novel retraction cancels it in present, erasing an interesting time the
reduce needed. Not the min logic, not the int_proxy tactic. Debug taps are
CORGI_DBG-gated. gate green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Adopt int_proxy seed fix (fm/int-proxy 960a6ec4) + implement present_novel in corgi

Framework (cherry-pick of the model-prescribed fix): ProxyReduceBackend::key_hashes
-> present_novel(novel) -> ProxyChunk. The tactic now seeds interesting times and
the synthetic-join closure from the BATCH's own support (present_novel = the novel
batches presented ALONE), matching Model.lean's `seedSet = b.support ∪ pending`,
instead of time-filtering the merged present_input. This fixes the SCC divergence
(scenario1_cancels): a legally-compacted stored record could cancel a novel
retraction out of the merged view, silently dropping a seed while a retraction was
still owed. Kept my churn-reduce edits (buffer reuse, IdMap, radix from_unsorted).

Corgi backend: CorgiReduceBackend implements present_novel (read novel chunks alone,
unfiltered -> from_unsorted), drops key_hashes; present_input simplified (novel keys
are all in `keys`, so one binary_search filter — dropped the seen<novel_len hack).
Removed the CORGI_DBG diagnostic taps.

Verification: corgi_scc_min sweep NO mismatch (was 4-node divergence); corgi_progs
all 6 == vec; DD int_proxy (11, incl. new seed/cancellation + compaction-adversary
fuzz) + reduce_reference (21) green. Scorecard: reduce ~parity (present_novel adds
a per-retire present pass, same cost the reference backend accepts); reach ~parity;
scc now CORRECT end-to-end (still ~2x slower — the sharp recursive case).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Adopt seed_times refinement (fm/int-proxy 34cc815) + implement it for corgi

The other agent's slimmer fix: seeds need only the batch's (key_hash, time)
support, not a value-present. Interesting-time over-derivation is sound (a
non-changing seed yields a zero delta), and the tactic reads only TIMES from the
seed source — so no value hashing, value_ids, sort-by-value, or consolidation.
Trait: present_novel(&mut self) -> ProxyChunk  =>  seed_times(&self) -> Vec<(u64,T)>
(stateless — seeds never refer back to real values). history.rs gains TimeHistory
(times-only replay, same meet-advancement, keeps the scaling shapes linear). This
also sidesteps the value_id=0 trap (a ProxyChunk would re-consolidate distinct
values at one time into a dropped zero, reintroducing the cancellation bug).

Corgi: CorgiReduceBackend::seed_times = hash_rows(novel key columns) zipped with
the Rust-side times, sorted by key_hash — no from_unsorted, no value hashing on
the seed path. The expensive half of the per-retire double-present is gone; only
a delta-sized second key-hash remains (noise). This matters most on the common
SNAPSHOT-then-increment load, where present_novel had been ~doubling the present
work on the big first batch.

Verify: corgi_scc_min NO mismatch; corgi_progs all 6 == vec; DD int_proxy (11) +
reduce_reference (21) green. Scorecard reduce back to ~parity (distinct 0.99-1.01x,
count 0.98-1.00x — was 1.01-1.04x under present_novel), reach 0.97-1.08x.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* corgi_perf: add native DD SCC (three-way native vs DDIR-vec vs DDIR-corgi)

SCC is now correct in corgi, so the original question is answerable: how does the
hand-written native differential-dataflow strongly_connected compare to the same
SCC run as a DDIR program (vec interpreter vs corgi columnar backend), on the same
random graph, same from-scratch single-worker methodology. native_scc_once drives
algorithms::graphs::scc::strongly_connected. Result: vec-DDIR ~1.2-1.6x native
(the DDIR interpretation tax is small); corgi-DDIR ~2.9-3.8x native / ~1.9-2.7x
vec, gap GROWING with n (SCC is arrangement+recursion-bound — corgi's worst case,
the mirror of the compute-bound linear cases where corgi is ~3x FASTER than vec).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* corgi_scc_prof: loopable SCC profiling target + attribution

Profiled corgi SCC (n=500, samply+pollard): the slowness is CORGI's arrangement/
merge layer, not the framework. CorgiChunk::merge = 27% total; 57% of it is the
JOIN re-merging its accumulated trace per work-unit (flatten_batches, O(accumulated)
per round -> the superlinear term, gap grows with n), 26% arrange-ingest batcher,
17% spine maintenance. corgi::compare_idx (structural discrimination compare) drives
another 24%; malloc ~25-30% from corgi rebuilding columns per merge. Framework is
small: ProxyReduceTactic::retire is 24% total but only 3.7% self (rest is corgi
present/materialize); timely scheduling negligible; scalar eval a sliver (CmpOp::eval
3.1%). Levers: cursor/fueled join read instead of re-flatten; merge without whole-
column re-gather; cut per-merge allocation. Not the reduce value logic, not int_proxy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* corgi join: delta-proportional restriction of the accumulated side

The cursor-less corgi join re-flattened the whole accumulated trace per
work-unit (O(trace)/round, superlinear on recursive joins). Restrict the
accumulated side to the fresh side's keys via find_ranges per chunk and
gather only matches — delta-proportional, structural (no hashing), no
full-trace flatten. Gate on acc_len > 2*fresh_len so the non-recursive
one-shot join (comparable sides) keeps the cheaper plain flatten and does
not pay the probe+reconsolidate overhead.

SCC n=1000: 2.84x -> 2.24x vec; non-recursive join unchanged at 0.66x;
corgi_progs 6/6 match vec, SCC corgi==vec at all sizes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Make corgi-chunk shareable: git-dep corgi + field report + large-N SCC harness

Swap the corgi path-dep (../../wip/corgi) for a git-dep pinned to
frankmcsherry/wip@9b41cdc (dd-arrange-api), so the branch builds standalone
without a local wip checkout. Add int-proxy-columnar-findings.md (the field
report tying #781 + this branch + the corgi kernel crate, with the SCC numbers
and the columnar-Time gap) and the large-N SCC / vec-profiling harnesses the
report's repro section references.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Collect dead generations: drop pre-Chunk arrange + pre-int_proxy reduce

Remove the two superseded modules (corgi_arrange.rs pre-Chunk arrangement,
corgi_reduce.rs pre-int_proxy reduce) — nothing in the live path referenced
them — plus the ~150-line dead stratum inside corgi_chunk.rs that only they
used (from_rows/to_rows/batch_to_rows, rows_to_batch, CorgiChunkBuilder,
group_offsets, semijoin_history) and the orphaned imports.

Also delete the examples that only exercised the dead modules (corgi_arrange_smoke,
corgi_join_mechanism/dataflow, corgi_reduce_dataflow/trace) and the SCC
bug-isolation harnesses (corgi_scc_min/trace/stream, corgi_cc_min, corgi_bwd_min)
for the interesting-times bug now fixed and pinned by tests + Lean.

Live path unchanged: build clean (no warnings), corgi_chunk unit tests pass,
corgi_progs matches vec. Net -2157 lines. Git history preserves the journey.

Also correct int-proxy-columnar-findings.md: the DD subtree is NOT byte-identical
to #781 — ~148 lines (MSD-bucket sort, reduce buffer-reuse, tests) sit ahead of
the PR and should be pushed back to int-proxy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Re-port corgi onto the merged #781 seam (windowed reduce + iterator join)

Reduce backend → the windowed ProxyReduceBackend: seed_times(+ReduceInstance),
begin (open tiled session), next_window (single window = all changed keys; build
input+output ProxyBridges via collect_present + consolidate_updates), and the new
reduce_corrections (reduce_brackets for desired, then difference vs. presented
output per value_id — the backend now owns the desired−current subtraction),
emit/finish (tiled column materialization). Content-hash value_ids + in_index
resolution as before; ProxyChunk/from_unsorted gone.

Join tactic → the #790 iterator contract: defer/work collapse to prep returning
Box<dyn Iterator<Item = CorgiContainer>>; run_unit now returns the container
instead of a session give_container. meet ignored (correctness-first).

CorgiContainer already impls Accountable; fixed the join_with_tactic container
(not builder) generic at the call site.

Gates green on merged master-next DD: corgi_progs 6/6 match vec, incl. scc.
Structural order (correctness first); hash-order + join hash-probe = Phase 2b.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Value-as-id for primitive keys/values in the reduce (skip hash_rows)

For a primitive column — a bare Prim(64) or a 1-field Prod([Prim(64)]) — the
value itself is already a collision-free id (i64 as u64 is a bijection), so pass
it through directly instead of content-hashing. Compound shapes still hash. The
id is used only as an identity for netting/dedup — its numeric order is never
relied on — so the raw two's-complement u64 is correct even for negatives; no
order-preserving swizzle needed.

Applied CONSISTENTLY at every id site (both value presentations AND the
reduce_brackets output values), so desired-current still nets by matching ids.

Gate: corgi_progs 6/6 (all programs match vec). Scorecard: reduce_distinct
1.18x->1.07x, reduce_count 1.17x->1.07x, reach 1.26x->1.20x. join unchanged
(the Phase 2a join is already hash-free). SCC flat within single-run noise
(allocation-bound, not hashing-bound) — confirming columnar times is the SCC lever.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Add ColTimes<T>: SoA per-tuple time column (T: Columnar) — foundation

Introduces columnar times for CorgiChunk without a DD change. Our
Time = Product<u64, PointStamp<u64>> already derives columnar::Columnar at every
layer (Product, PointStamp, u64), with the container's Ref deriving Ord/PartialOrd
(#[columnar(derive(Ord, PartialOrd))] on both Product and PointStamp). So the same
per-tuple times that were Vec<T> (one PointStamp SmallVec per row — the dominant
SCC allocation) live in <T as Columnar>::Container as SoA (offsets + values), one
allocation pair per column instead of n.

ColTimes exposes exactly what the CorgiChunk consumers need: push(&T) / push_ref
(Ref straight across, no T), get(i)->T (materialize only for Lattice ops + emit),
len, and cmp/cmp_cross (in-place ordering via the derived Ref: Ord, HRTB-bounded).

Not yet wired — Inner.times stays Vec<T> this commit. Containment rationale: the
Chunk trait boundary is whole-chunk + frontier-antichain only (CorgiChunk is Chunk
but NOT NavigableChunk, so no cursor TimeContainer/TimeGat), and the batcher/spine
move whole Rc<ChunkBatch> — so every O(data) per-tuple T is Inner.times, ours.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Wire columnar times: Inner.times Vec<T> -> ColTimes<T>

CorgiChunk's resident per-tuple times are now SoA (ColTimes over
<T as Columnar>::Container) instead of Vec<T> — killing the per-row PointStamp
SmallVec allocation that dominated the SCC profile (~40%). The hot maintenance
paths (merge/emit/extract/advance/settle/concat) build and read times columnar:
compares go through the derived Ref: Ord in place (no T materialized), range
copies push Refs straight across, and an owned T is reconstructed only where a
Lattice op is unavoidable (advance_by in compaction) or at the egress boundary
(SortedRun for the join, CorgiContainer for as_collection — both want owned T).

Bounds: a ColTime alias (Timestamp + Lattice + Columnar) bundles the constraint;
the viral HRTB `for<'a> Ref<'a,T>: Ord` is discharged once in ColTime's blanket
impl behind ColTime::cmp_refs, so downstream code needs only T: ColTime. Ingress
paths (sort_consolidate, CorgiChunker accumulator, flatten_restricted) stay Vec<T>
(they already hold owned T), converting at from_columns.

No DD change. Gate: corgi_progs 6/6; 29 lib tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Drop unused Timestamp/Lattice imports (subsumed by ColTime bound)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Adopt corgi survey (merge) + group_bounds (advance) — bump to f556868

CorgiChunk::merge now drives off corgi `survey(kv1, kv2)`: one bidirectional
gallop over the (key,val) columns yields Runs — maximal exclusive A/B ranges
(bulk-copied, no per-row compare) + single Both matched pairs (time-ordered /
consolidated here, since corgi owns no time). Replaces the per-pair two-pointer
compare_at (the 22%-and-growing merge-compare tax). CorgiChunk::advance uses
`group_bounds(ckv)` for the group-boundary scan (one pass) instead of per-row
compare_at. Bumped corgi pin 9b41cdc -> f556868 (survey/group_bounds + hash_order
reverted).

Note on times: survey's Both is a positional (key,val) match, so it can
transiently mis-order/under-consolidate the out-of-band times; advance re-sorts +
re-consolidates each (key,val) group before any consumer reads the chunk, so it's
correct end-to-end (gate 6/6 + 29 lib tests incl. merge/advance/is_graded oracle).

SCC: 1.94x -> 1.62x vec (n=100k 6.55s -> 5.43s, ~17% off corgi). Branch
corgi-survey-merge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* collect_present: keep the scan (find_ranges seek regresses SCC)

Tried a delta-proportional find_ranges seek of the changed keys (value-as-id
makes them seekable in structural order). It REGRESSED SCC 1.62x->2.16x: SCC's
changed set is broad (label propagation touches most keys each retire), so the
scan already touches ~every row while the per-chunk gallop only adds overhead.
Reverted to the scan; simplified the signature to `changed: &[u64]` and recorded
the negative result in the doc so it isn't re-tried. Gate 6/6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Hash unification: compound ids via native corgi::hash, not arrange::hash_rows

The `ids()` compound fallback now hashes via the canonical native `corgi::hash`
(the designed boundary-id fold — width-blind, consistent-with-equality) instead
of the branch-local `arrange::hash_rows` (width-seeded). Safe for DDIR: every leaf
transcodes to u64 so width-blindness is a no-op, and there is no cross-path hash
comparison (value-as-id for primitives and native hash for compounds are never
used for the same value — shape is uniform per column). DDIR no longer references
hash_rows, so the corgi side can retire it freely. Gate 6/6 (incl. compound-keyed
adt/binders/unnest), 29 lib tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Revert the survey merge — it bloats the arrangement at multi-chunk scale

The survey merge (17ff06d0) was a single-chunk MIRAGE. survey aligns only
(key,val) (corgi owns no time), so its positional Both under-consolidates
cross-side times; combined with consuming both chunks and no suffix push-back, it
leaves canceling rows in the arrangement. Single-chunk (n<=100k) it looked like a
win (SCC 1.94->1.62x, compare_idx 21.9->8.9%), but past the chunk boundary
(edges e=2n cross TARGET=262144 at n~131k) the un-consolidated arrangement BLOATS
every iteration and the reduce re-presents ever-growing input -> super-linear
(n=150k: 104.9s / 19.5x vec vs 9.9s / 2.08x with two-pointer). Profile smoking
gun: retire 96%, consolidate_updates 24.8% + quicksort 22.7% + collect_present
17% — the REDUCE drowning in bloated input, not the merge itself.

Restored the two-pointer merge (full (key,val,time) consolidation + survivor
push-back). Kept group_bounds in advance (perf-neutral but a clean one-pass scan;
doesn't touch consolidation). Back to ~1.96x vec, LINEAR to n=250k. A survey merge
needs a group-RANGE Both to compose with out-of-band times — flagged to corgi.

Gate 6/6. corgi pin stays f556868 (group_bounds + native corgi::hash).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Fair compiled SCC baseline: strongly_connected_at + four-way corgi_scc_big

Adds scc::strongly_connected_at(graph, logic) — the prioritized variant the
code comment always suggested — routing trim_edges through propagate_at so
label introduction is log-bucketed 256*(64-lz(logic(label))), byte-identical
to DDIR's enter_at($1[0]) delay in vec.rs/corgi.rs. strongly_connected is
unchanged (= strongly_connected_at(_, |_| 0)).

Oracle-tested: tests/scc.rs gains scc_at_{10_20_1000,100_200_10,100_2000_1}
running the prioritized variant against the sequential SCC oracle (incl. 1000
incremental rounds); all 6 pass.

corgi_scc_big gains the fair(enter_at) column. Measured (arm64, e=2n):
  n=100k: native 4.16s, fair 0.62s (0.15x nat), vec 3.06s (4.95x fair), corgi 5.94s (9.61x fair, 1.94x vec)
  n=150k: native 7.42s, fair 0.95s (0.13x nat), vec 4.83s (5.08x fair), corgi 9.90s (10.41x fair, 2.05x vec)
The compiled target was ~6.7x closer than the plain-native column suggested.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* B3 ident-join benchmark: pointer-doubling reps + term-resolution joins

E-graph-flavored identifier-lookup workload: parent forest (chains of CHAIN,
roots self-looped) resolved to roots by an iterative min-join fixpoint
(pointer doubling), then a term table (id,a,b) resolved by two integer joins
against reps. Closed-form oracle (root = x - x%CHAIN) checked against vec,
corgi asserted == vec, hand-written native twin oracle-checked too.

Measured (arm64, chains=64, terms=2n):
  n=100k: native 0.59s, vec 2.34s (4.00x nat), corgi 3.49s (5.95x nat, 1.49x vec)
  n=1m:   native 6.55s, vec 26.99s (4.12x nat), corgi 46.28s (7.07x nat, 1.71x vec)
corgi-vs-vec gap GROWS with n (1.31x @1k -> 1.71x @1m) — profile candidate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* B2 scc-compound benchmark: SCC over (group,index) tuple nodes

Same control structure as B1 but every node is a nested compound value
(x -> (x>>10, x&1023), order-preserving): no value-as-id fast path — compound
hashing, nested compares, and nested-column gathers carry the load.
enter_at($1[0][0]) buckets by group; the fair native twin is
strongly_connected_at over (i64,i64) with logic |n| n.0 (identical delay).

Correctness: corgi == vec on the full SCC edge set; compound output mapped
back through the encoding == the plain-encoding program on the same graph.

Measured (arm64, e=2n): compound keys hurt everyone, corgi worst:
  n=100k: fair 2.76s, vec 21.56s (7.80x), corgi 61.42s (22.22x fair, 2.85x vec)
  n=150k: fair 5.22s, vec 41.48s (7.95x), corgi 132.77s (25.43x fair, 3.20x vec)
(B1 plain-int corgi is 1.94-2.05x vec — the compound-key path is corgi's
weakest spot so far, and the ratio grows with n.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* B4 ast-compute benchmark: list/flatmap/variant/fold compute bookend

Per row (a,b < 2^15): 8-element arithmetic list, flatmap explosion, Fwd/Bwd
variant + case + 4-element fold per element, min per 16 buckets (min so the
compute can't be dead-coded; reduce stays tiny). Closed-form Rust oracle
checked against vec, corgi == vec, native twin oracle-checked.

Values constrained non-negative + non-overflowing: the first draft (values to
1e6, a-b negative, e*e overflowing) exposed corgi min DIVERGING from vec on
negative ints — corgi's structural order is unsigned at the integer leaf.
Real abstraction gap, logged in the read-out; out of contract for B4.

Measured (arm64): corgi is NOT columnar on this path — flatmap/case/fold run
the row-wise fallback (ir::eval + transcode), so the expected compute win is
absent:
  n=1m: native 1.09s, vec 8.35s (7.69x nat), corgi 8.89s (8.19x nat, 1.07x vec)
  n=4m: native 4.55s, vec 34.61s (7.60x nat), corgi 37.19s (8.17x nat, 1.07x vec)
Columnarizing flatmap/case/fold is a top lever candidate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* B2 profiling harness: corgi_compound_prof (BACKEND=corgi|vec)

Loops one backend of scc-compound at a fixed size for samply, built identically
for apples-to-apples profiles.

Iteration-1 profile findings (n=50k, corgi 28.0s vs vec 11.4s):
- retire subtree = 77% of corgi; vec's Value::cmp/partial_cmp/eq = 26% self.
- consolidate_updates_slice_slow 17.6% total. Two tactic-side fixes (permutation
  sort; interned-time integer consolidation + stable run-merge) were both
  measured FLAT and reverted: the hot consolidation is NOT the tactic's delta
  emit — it is next_window's presentation build in corgi_reduce_backend
  (collect_present clones an owned PointStamp per presented row, then
  consolidate_updates comparison-sorts the AoS records; the caller frame was
  inlined away). That backend site is the next lever.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* B4 profiling harness: corgi_ast_prof (BACKEND=corgi|vec)

B4 corgi profile (n=1m): ir::eval 19% (row-wise List/Case fallback), Value
churn ~15%, arrangement merge of the 8m exploded rows ~29%, reduce 14%.
Both backends run B4 row-wise — the columnar win is unrealized here.
Iteration-4 lever scoped in the read-out: columnarize List/FlatMap/Case in
the backend via existing corgi kernels (Iota/Gather/Flatten/Fold/Inject/
MapSum + gather_lanes interleave).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* B3 profiling harness: corgi_ident_prof (BACKEND=corgi|vec)

B3 corgi profile at n=1m surfaces two concentrated corgi-kernel buckets the
SCC profiles never showed: sort_leaf_blocks 17% self (discrimination-sort
leaf pass, under ingress sort_consolidate + materialize from_columns) and
the join's find_ranges probe 17.8% total (interpreted CmpOp::eval compare
per binary-search step). Both kernel-general levers; details in the read-out.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* corgi_scc_big: SEED env for instance-variance measurement

5 timing runs at n=100k (fixed graph): corgi 3.76-3.89s, vec 2.97-3.03s,
ratio 1.26-1.30x; n=250k: ratio 1.28-1.29x. 3 graph instances (SEED=1,
12345, 999983): ratio 1.24-1.29x, vec/fair 4.78-4.94x. Both axes ~ +/-2%;
the reported ratios are solid to about two figures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* phase 3: framework-tax decomposition profile + mimalloc for the suite

New native_scc_prof (fair enter_at compiled twin). Triptych at n=100k, self-time
by module: allocator native 2.2% / vec 28.4% / corgi 27.2%; memmove native 0.7%
/ vec 2.8% / corgi 11.9%; timely progress tracking ~0.5% in BOTH backends. The
4.8x shared tax is BUFFER LIFECYCLE (allocation + copy churn), not progress
tracking (kills lever 1b as a priority), not operator count (rung B), not edge
time repr (rung A). vec's churn is per-row ir::Value boxes (cmp+clone+eq ~20%
self); corgi's is column rebuilds (fresh Arc<Vec> per gather/merge/present).

The suite ran on the SYSTEM allocator (only ddir_server/ddir_vec linked
mimalloc). Adding mimalloc to all suite + prof binaries (one binary per
benchmark: every column shares it; native columns move <5%, DDIR 12-36%):
  B1 250k: corgi 8.67 -> 7.07s (0.96x vec, 4.29x fair; was 5.2x fair)
  B2 150k: corgi 53.45 -> 42.94s (1.22x vec)   100k: 1.08x vec
  B3 1m:   corgi 27.24 -> 19.79s (0.82x vec, 3.17x fair)
  B4 4m:   corgi 20.09 -> 13.00s (0.45x vec, 2.96x fair)
Gate 6/6 incl. fusion oracle; [checked] rows green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* CI prep: bump corgi pin to dd-arrange-api 1301b28; drop unused import

The pin moves from f556868 to the dd-arrange-api tip, which carries the four
landed kernel additions (wip#7). Gate green; workspace passes -D warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* examples: one ddir_bench runner; benchmark programs to .ddp files

Ten purpose-built binaries (four benchmarks + six profiling twins) collapse
into one runner: interpreted programs live in examples/programs/*.ddp where
they belong; the irreducibly compiled parts — input generators and the native
twins, including the enter_at-fair SCC — register in the runner by bench name.
BACKEND=/ITERS= select profiling mode (loop one column, a samply target),
replacing the prof twins. Output formats, env protocols (N/SEED/CHECK/CHAIN),
and all correctness assertions are unchanged; all four benches verified
[checked] plus a prof-mode smoke; gate green; workspace passes -D warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* examples: prune the build-up scaffolding

Eleven milestone/experiment binaries (rungs 0-4, M1-M4, Q3 of the backend
build-up) whose questions are answered and whose coverage the survivors carry:
the gate (corgi_progs) runs reach/scc and richer programs continuously, the
scorecard owns per-operator triage, and ddir_bench owns end-to-end measurement
and profiling. SPIKE.md narrates the arc they documented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* restore the gate's scc.ddp; the benchmark program is scc_bench.ddp

The .ddp extraction overwrote the pre-existing gate program with the
benchmark's embedded SCC (different export shape, comments lost). The gate
kept passing because the benchmark program is also a valid corgi==vec check —
which is how it slipped through. scc.ddp is restored byte-identical;
ddir_bench loads scc_bench.ddp.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* interactive: gather the corgi backend internals under a corgi module

The five crate-root corgi_*.rs files become src/corgi/{container,chunk,join,
reduce,logic}.rs. No code change beyond the module paths: container is the
CorgiContainer on dataflow edges (was corgi_backend.rs), chunk the columnar
Chunk + arrangement plumbing, logic the DDIR-term compiler, join/reduce the
tactic bindings. backend/corgi.rs (the Backend impl, parallel to backend/vec.rs)
and col_times.rs (a generic columnar-time utility that imports no corgi) stay
where they are. Gate 6/6 both tactics; workspace passes -D warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* interactive: gate to a test; remove benchmark tooling and working notes

Addresses review of the subsystem PR:
- corgi_progs (the corgi==vec gate) moves from examples/ to a tests/ integration
  test (tests/corgi_backend.rs) — it is a gate, so its home is cargo test / CI,
  not examples/. Runs the six canonical .ddp programs, one #[test] each.
- Remove the benchmark/measurement tooling that isn't a DDIR example: ddir_bench,
  corgi_scorecard, and the five benchmark-only programs (ast/ident/scc_bench/
  scc_compound/scc_plain .ddp — the latter three the duplicate-SCC sprawl). These
  are dev tooling, not part of the library; they stay on the dev branch and can
  land separately if the later perf PRs want reproducible in-tree numbers.
- Drop the working notes (SPIKE.md + two findings files, ~970 lines of markdown).

examples/ is back to DDIR programs plus the pre-existing server/dump binaries.
strongly_connected_at stays (a general algorithms/ addition, tested by tests/scc.rs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* interactive: move col_times into the corgi module

col_times exists to serve CorgiChunk (its own doc says so) and is used only by
the corgi backend's modules — it belongs in src/corgi/, not at the crate root.
No code change beyond the module path (crate::col_times -> crate::corgi::col_times).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* interactive: trim the corgi dependency comment

Drop the local-dev path note and the reference to a findings file that no longer
exists; keep the git rev pin (explicit and reproducible) with a one-line purpose.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* backend/corgi.rs: refresh the stale module header

The header described an early milestone: Spine<Rc<CorgiBatch>> (the arrangement
is a ChunkSpine<CorgiChunk> now), reduce 'awaits' a retire design (it is
implemented via the tactics), and a 'this iteration ... = todo!()' note (nothing
is todo). Rewrite it to describe the current, complete substrate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* backend/corgi.rs: hoist the type aliases above their first use

CC/Row/Upd/CTrace were defined ~110 lines below apply_ops, whose signature reads
in terms of CC. Move them to just after the imports so the shorthands are in
scope where the reader first meets them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* backend/corgi.rs: document CorgiBackend as an uninhabited type-level tag

Explain the empty enum: it is never a value, only a type carrying the Backend
impl (selected via render_tree::<CorgiBackend>); the empty enum makes it
unconstructable. Mirrors VecBackend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Remove strongly_connected_at; revert scc to master-next

strongly_connected_at was added for the benchmarks' fair-compiled SCC baseline;
those benchmarks left this PR, orphaning it (tests/scc.rs was its only remaining
user). Its scc_at tests exposed a PRE-EXISTING nondeterminism in propagate_at/
propagate_core (the rotated reduce_abelian propagation): scc_at_10_20_1000 is
flaky under 3 workers (the plain scc test, which uses a local iterate+enter_at
reachability, is deterministic). Rather than carry an orphaned API whose tests
surface someone else's bug, revert scc.rs + tests/scc.rs to master-next. The API
can return with the benchmark tooling once propagate_core's determinism is fixed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* corgi join: port to the int_proxy proxy-join seam, delete the bespoke tactic

CorgiJoinBackend implements ProxyJoinBackend (#809): advance draws blocks of
the key intersection as ((key, coordinate), time, diff) bridges; cross redeems
matched coordinates directly against the instance's chunks (gather_lanes) and
cuts TARGET_OUT-sized containers. run_unit and its whole-unit staging (the
large-snapshot OOM) are gone; peak state is one block plus one container.

Value tokens are canonical coordinates: the per-key merge across chunks gives
equal values the coordinate of their least occurrence, so bridge consolidation
cancels cross-batch churn (tokens are the unit of cancellation), while values
themselves are never copied into the presentation — they are gathered once,
from chunk storage into the projection input, for matched rows only.

Group tokens are the key's own u64 when the key column flattens to a single
64-bit leaf lane (what DDIR Int keys and 1-tuples transcode to): chunk order
is u64 order, so blocks resume by seeking from. Two regimes, mirroring the
bespoke tactic's probe heuristic: a much-smaller side drives and the other is
probed at the driver's keys (batched find_ranges); comparable sides are pulled
and merged symmetrically (probing would cost n log n against the merge's n).
Multi-lane keys fall back to a structural single-block walk (ordinal tokens).

Gate green 6/6, debug (harness asserts live) and release. Bench vs the bespoke
tactic: scc 3.71s vs 3.77s; ident@1M 21.79s vs 22.76s (0.92x vec); ast 0.83-
0.92x vec; scc-compound (2-lane keys, the structural fallback) 50.4s vs 46.5s,
a known follow-up: the probe/merge split applies there too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Remove the code the join port orphaned

The bespoke tactic's presentation layer (SortedRun, flatten_batches,
flatten_restricted) lost its last caller when run_unit was deleted, and the
DrainContainer row path (containers drained to rows for a reused row-wise
MergeBatcher) was never live: the corgi pipeline's one arrangement path is
chunk-native (CorgiChunker/ChunkBatcher). sort_consolidate and concat_blocks
stay — from_columns and the chunker still use them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Review pass: per-chunk as_collection, dead types, narrative pruning, hot-path TODOs

as_collection emits one container per chunk — key/val columns clone by Arc
bump, times materialize (the owned-time egress), diffs memcpy — instead of
gather-concatenating every batch's chunks into one container; the concat
bought nothing downstream, and chunks_to_columns goes with it. The vestigial
row-update alias (Upd) leaves chunk.rs, taking the layer's last row-type
import with it: nothing under src/corgi/ mentions DValue except logic.rs's
transcode boundary.

Comment hygiene: history-of-the-work narration (superseded implementations,
reverted experiments, rung/phase references, stale type and document names)
gives way to present-tense constraints; the fallback comments now say
accurately that the gap is unwritten lowering in logic.rs, not expressiveness
in corgi (which models sums and lists).

Two TODOs mark the complete hot-path row-at-a-time residue: merge() awaits
survey_groups (group-range Both; newer corgi revs export it) for a batched
rewrite, and collect_present can memoize key hashes, batch its membership
test, and move kept ranges via push_range.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Mark the container/chunk split and the row-trip fallbacks as convergence debt

Cross-reference CorgiContainer and the chunk's Inner as one payload that
should become one type: the differences (row-mutable Vec<T> times on edges,
sorted/consolidated/shared in the trace) each name the artifact that removes
them, with a bulk-mutation time container dissolving the last. from_updates/
into_updates likewise declare their intended end-state — external ingest and
inspection edges only — and each in-dataflow fallback caller names what
retires it (the Cas…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant