kernel: dispatch runnable steps in parallel

[kjgbot]

Closes the issue #132 item 5 kernel slice.

Protocol change — a terminal run refuses event.emit and stream.append

Added by the rebase onto 512723c. A reviewer should meet this here rather than
discover it in a test diff. The call was made by the lane lead; the signoff is
asked to re-derive it rather than inherit it.

This is not a new rule. main already holds it, and simply fails to apply it
consistently. A worker holds an llm lease, the run is cancelled out from under
it, and the worker does what a live worker does. On main, today:

late step.complete  -> THROW [lease_conflict]   <- main already refuses this
late stream.append  -> OK    {"offset":0}       <- and corrupts the journal
late event.emit     -> OK    {"matched":0}
>>> entries AFTER terminal run.completed: ["stream.appended"]

The late completion is refused and the late append is accepted, from the same
worker in the same moment. main states the rule for step completions at
engine/remote.rs:44 (#142); this PR extends main's own rule to the remaining
mutating verbs.

It matters because the accepted append corrupts the run. stream.append journals
stream.appended after run.completed, producing a journal the daemon can no
longer fold on resume — same data directory, both binaries:

MAIN   resume -> exit=0  {"status":"completed","completion_reason":"success"}
BRANCH resume -> exit=1  Error: fold run journal
                         Caused by: journal entry 5 appears after terminal run.completed

So this does not remove a working capability; it removes a way to corrupt a
journal that main currently reports as success. The wire-conformance test was
asserting a bug.

kernel/DESIGN.md:384-385 agrees: event.emit "satisfies wait.event", which a
terminal run has none of — so main's matched: 0 is a no-op wearing a success
shape — and stream.append "journals stream.appended", which is the corruption
itself.

Callers that appended to a finished run now receive run_terminal. No in-repo
product code did so; the one live in-repo use appends against a parked run and
is unaffected.

Two test commits carry this, each isolated and each stating that the call was the
lead's:

• 269fcc6 — a late step.complete after run.cancel reports run_terminal
  rather than lease_conflict. Error-code taxonomy only; the refusal and the
  journal assertions are unchanged.
• 4dec6af — a terminal run refuses event.emit and stream.append. Behaviour,
  not taxonomy. Both verbs keep full wire coverage: their success paths move to a
  parked run, and the terminal case now asserts the refusal, agreeing with
  crash_resume/protocol_admission.rs — which is left untouched, because it
  pinned the invariant correctly.

Full diagnosis, the four-way check that no legitimate late append exists, and the
rebase audit: ops/reviews/20260903-pr137-repair-0903.md.

What changed

• next_actions now emits journal-first start/action pairs for every step already Runnable in one folded state snapshot.
• Emission order is the authored RunSpec.steps order, so the same journal state and simulated time produce the same batch.
• Dependency barriers remain fold-driven: a join is not considered until every dependency completion is journaled and folded.
• Crash/resume coverage pins partial-batch recovery and deterministic recovery of every durable in-flight lease.
• The implementation stays inside relayflowd-core; no provider, product, tenant, network, filesystem, or wall-clock logic enters the scheduler. v1 remains supported/default.

Red-first evidence

Command on the pre-change scheduler after adding the fan-out test:

$ cargo test -p relayflowd-core machine_starts_every_runnable_step_in_authored_order
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.21s
     Running unittests src/lib.rs (target/debug/deps/relayflowd_core-eb215c870b5b8c71)

running 1 test
test machine::tests::machine_starts_every_runnable_step_in_authored_order ... FAILED

failures:

---- machine::tests::machine_starts_every_runnable_step_in_authored_order stdout ----

thread 'machine::tests::machine_starts_every_runnable_step_in_authored_order' (2153048) panicked at relayflowd-core/src/machine/tests.rs:61:5:
assertion `left == right` failed: both independent lanes must start
  left: 2
 right: 4
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    machine::tests::machine_starts_every_runnable_step_in_authored_order

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 28 filtered out; finished in 0.00s

error: test failed, to rerun pass `-p relayflowd-core --lib`

Final verification

$ cargo test --workspace --quiet

running 22 tests
......................
test result: ok. 22 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.80s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

running 19 tests
...................
test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.80s

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s

running 3 tests
...
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.21s

running 29 tests
.............................
test result: ok. 29 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.32s

running 5 tests
.....
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

running 17 tests
.................
test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

$ cargo clippy --workspace --quiet -- -D warnings
(no output; exit 0)

$ rustfmt --edition 2024 --check relayflowd-core/src/machine.rs relayflowd-core/src/machine/tests.rs relayflowd-core/src/machine/parallel_tests.rs
(no output; exit 0)

The repository wrapper was also attempted literally and could not start in this local environment:

$ ../ops/cargo.sh clippy --workspace -- -D warnings
mise ERROR cargo is not a valid shim. This likely means you uninstalled a tool and the shim does not point to anything. Run `mise use <TOOL>` to reinstall the tool.
mise ERROR Run with --verbose or MISE_VERBOSE=1 for more information

No merge performed; human review and merge remain required.

[coderabbitai]

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Free

Run ID: 8ee20638-afc2-4ae6-9fae-d07c27987ab4

📥 Commits

Reviewing files that changed from the base of the PR and between 512723c and 7f6a30f.

📒 Files selected for processing (42)

• kernel/relayflowd-core/src/machine.rs
• kernel/relayflowd-core/src/machine/parallel.rs
• kernel/relayflowd-core/src/machine/parallel_tests.rs
• kernel/relayflowd-core/src/spec.rs
• kernel/relayflowd-core/src/spec/tests.rs
• kernel/relayflowd-core/src/state.rs
• kernel/relayflowd-core/src/state/pins.rs
• kernel/relayflowd-journal/src/append.rs
• kernel/relayflowd-journal/src/lib.rs
• kernel/relayflowd/src/engine.rs
• kernel/relayflowd/src/engine/drive.rs
• kernel/relayflowd/src/engine/effects.rs
• kernel/relayflowd/src/engine/remote.rs
• kernel/relayflowd/src/server.rs
• kernel/relayflowd/src/server/protocol.rs
• kernel/relayflowd/src/server/session.rs
• kernel/relayflowd/src/server/session/assignments.rs
• kernel/relayflowd/src/server/session/matching.rs
• kernel/relayflowd/src/server/tests.rs
• kernel/relayflowd/src/server/wire.rs
• kernel/relayflowd/src/worker.rs
• kernel/relayflowd/tests/crash_resume.rs
• kernel/relayflowd/tests/crash_resume/concurrency.rs
• kernel/relayflowd/tests/crash_resume/llm_support.rs
• kernel/relayflowd/tests/crash_resume/parallel_lifecycle.rs
• kernel/relayflowd/tests/crash_resume/pin_projection.rs
• kernel/relayflowd/tests/crash_resume/protocol_admission.rs
• kernel/relayflowd/tests/crash_resume/surface_identity.rs
• kernel/relayflowd/tests/crash_resume/worker_capacity.rs
• kernel/relayflowd/tests/crash_resume/workspace_identity.rs
• kernel/relayflowd/tests/parallel_driver.rs
• ops/reviews/20260903-pr137-repair-0903.md
• sdk/src/journal-client.ts
• sdk/src/protocol.ts
• sdk/src/validate.ts
• sdk/src/worker.ts
• sdk/tests/deterministic-llm.test.ts
• sdk/tests/live-kernel.test.ts
• sdk/tests/validate.test.ts
• testdata/hello-ladder.flow.yaml
• testdata/hello-ladder.spec.canonical.json
• testdata/hello-ladder.spec.sha256

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The scheduler now starts compatible steps in deterministic batches, coordinates worker reservations and leases, validates canonical surfaces, and prevents mutations after run completion. Journal, protocol, SDK, and integration tests cover parallel execution, recovery, capacity, pin handling, and terminal admission.

Changes

Parallel execution and scheduler state

Layer / File(s)	Summary
Canonical surface contracts kernel/relayflowd-core/src/spec.rs, sdk/src/validate.ts, kernel/relayflowd-core/src/state/pins.rs, kernel/relayflowd/src/engine/effects.rs	Workspace and external surfaces require canonical identities. Matching and containment use semantic path rules.
Deterministic parallel scheduling kernel/relayflowd-core/src/machine.rs, kernel/relayflowd-core/src/machine/parallel.rs, kernel/relayflowd-core/src/machine/parallel_tests.rs	Runnable steps start in authored-order batches. Retry timers wake together. Agent steps serialize on conflicting workspace, external, and stream surfaces.
Capacity-aware worker dispatch kernel/relayflowd/src/engine/drive.rs, kernel/relayflowd/src/server/session/*, kernel/relayflowd/src/worker.rs, sdk/src/worker.ts	Worker capacity is reserved before dispatch. Worker selection accounts for reservations and assignments. Lease and assignment state updates follow durable completion entries.
Terminal-run mutation enforcement kernel/relayflowd-journal/src/*, kernel/relayflowd/src/engine.rs, kernel/relayflowd/src/server.rs, kernel/relayflowd/src/server/protocol.rs, kernel/relayflowd/src/engine/remote.rs	Journal appends and mutating protocol operations reject completed runs. Rejected agent completions do not persist end pins.
Integration and verification coverage kernel/relayflowd/tests/*, sdk/tests/*, testdata/*, ops/reviews/*	Tests cover parallel recovery, worker capacity, surface identity, pin projection, terminal admission, SDK validation, and protocol behavior.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RunState
  participant Engine
  participant ProtocolHub
  participant Worker
  participant Journal
  RunState->>Engine: emit runnable step batch
  Engine->>ProtocolHub: reserve worker capacity
  ProtocolHub->>Worker: send step.dispatch
  Engine->>Journal: persist attempt and dispatch state
  Worker->>Engine: submit heartbeat or completion
  Engine->>Journal: persist durable completion
  Journal->>ProtocolHub: release assignment and advance pins

Loading

Poem

A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.

Warning

Some tools did not complete. Review the errors below.

🔧 Clippy (1.97.1)

Clippy execution failed

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

_{Comment @coderabbitai help to get the list of available commands.}

[kjgbot]

FINDINGS — structural signoff at bdd598c05d3172fd2fab6e3ec7009135ebdc2543

This is not a verdict on the current PR head (83db98bd1c6f39c187bb342fba23df2b022b53f9); I have not re-reviewed the commits pushed after bdd598c.

P1 — rejected agent completion can forge the next inspect attempt\x27s starting pins. step.complete advances the live worker pin projection from caller-supplied end_pins before the completion is validated or durably accepted (kernel/relayflowd/src/server.rs:269-270). A mismatched started_pins is then rejected by kernel/relayflowd/src/engine/remote.rs:275-289, but inspect recovery selects last_end_pins (kernel/relayflowd-core/src/machine.rs:232-237).

I reproduced this through the Unix socket: worker attached at repo@rev-0; attempt 1 supplied mismatched started_pins and end_pins: repo@forged-revision; the server journaled worker_error and dispatched attempt 2 at repo@forged-revision. This violates RFC-0001 Appendix A rules 2, 4 (inspect starting state), and 6, and fails closed incorrectly.

The focused capacity, terminal-admission, and canonical-surface suites passed; the full literal evidence and required regression are in the committed report that follows.

[kjgbot]

Durable report committed and pushed: f13a51a — docs(review): record PR 137 structure findings.

Correction to the prior review wording: this is the next inspect attempt's starting pins.

[kjgbot]

Merging — basis at 7f6a30fd, with one correction to the Protocol change section

Independent signoff REVIEW_PASSED, no P0, no P1, at this exact SHA. I wrote the Protocol change framing above and part of it overstates. Correcting it here rather than quietly editing.

Correction: only one of the two verbs corrupts a journal

The section says the change "removes a way to corrupt a journal", of both verbs. On main, event.emit on a terminal run writes no journal entry at all — only stream.append does. So the corruption argument is true of stream.append and not of event.emit.

Gating event.emit is still right, but on a different argument: a terminal run has no wait.event to satisfy, so main’s OK {"matched": 0} is a success shape for a guaranteed non-delivery, and uniformity across mutating verbs is worth more than a response that tells a caller nothing happened by returning success. That is the argument, and the body should have made it.

Second, smaller: "corrupt" overstates even for stream.append. The reviewer showed the message remains readable on main (stream.read → {"messages":[{"answer":4}]}). It is an ordering-invariant violation — an entry after run.completed that the daemon cannot fold on resume — not data loss.

The argument that carries the change is the one the final commit leads with, and it holds entirely inside main: from one worker in one moment, a late step.complete throws lease_conflict while a late stream.append returns OK {"offset":0}. engine/remote.rs:44 already holds this rule for step completions; this extends it to the rest.

Both lead calls were re-derived, not inherited

I made two calls the owner escalated. The reviewer agreed with both and sharpened each.

On 269fcc6 it did not judge on which word reads better — it asked whether ensure_mutable-first swallows lease_conflict where that is true, and proved it does not: worker-b completing worker-a’s step on a live run still throws lease_conflict: completion came from a worker that does not hold the lease, on both binaries. And my premise was weaker than the truth — on main’s terminal path the message is lease_conflict: attempt has no active worker lease, so the code and its own message already disagreed.

On 4dec6af, the out-of-band worry I raised is answered textually rather than by judgement: kernel/DESIGN.md:383 names step.complete as the out-of-band path, and main already refuses that when terminal — so RFC-0001’s async worker creates no legitimate late append. It also checked two shapes I had not named — wait.human and durable channels (stream.append takes a run_id, so streams are per-run) — and both hold.

The P0 — closed, and verified against the commit that introduced it

Reviewing against main shows nothing here, because main never had this defect. The reviewer built c21f039 outside the repo and fired one effect twice, changing only the spelling: /provider/item → deduped=false, /provider/item/ → deduped=false. Both fire. At this head the second is refused. Corpus 26/26 at the kernel boundary with the SDK bypassed and 26/26 in TS — Rust and TS agree on every row across both surface kinds. spec_parity 5/5 with canonical JSON recompiled through the SDK matching the pinned 57cac294…. workspace_surfaces_equal across its 18 call sites cannot reintroduce equivalence: repo/ is refused at worker.attach here and accepted at c21f039.

The load-bearing fixture is right for the right reason — the unknown-key test now fails on unknown field 'workspaces' rather than on the surface spelling.

Gates

3× tsc exit 0; cargo 130/0; vitest 411 passed / 3 skipped / 0 failed. Verified by test-name set-diff against a re-run of pure 512723c with its own pinned binary: zero names lost in either suite (102+28=130, 388+23=411, exact). compile.ts and step-fields.ts byte-identical to 512723c. Whitespace-insensitive attribution reduced 13 apparent deletions to 7 true ones, none authored by #138.

Carried forward, non-blocking

• The CI gap deserves its own issue. protocol_admission and the wire-conformance test encoded opposite contracts and both passed, because neither runs in CI — one workflow, cargo build but never cargo test, and exactly four named SDK files. The tree can hold a contradiction indefinitely and stay green.
• workspace_surfaces_equal’s doc comment at spec.rs:230-232 still describes the behaviour 93c1a3c reverted — code right, comment now inverted.
• A bounded reservation-leak window at engine/drive.rs:153 where ? returns before release; self-heals on detach.
• Caller errors surfaced as internal, including the P0’s own refusal message.
• The PR body undersells the PR. It describes only the first of eight kernel commits. 8edba06 (forged pins) — which stops the hub advancing worker pins from unvalidated params.end_pins before verification — is arguably the strongest commit here and goes unmentioned.

Squash-merging under the checklist: independent signoff PASSED at the exact head, CI green there, base clean, no unresolved feedback, no unpushed WIP.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR