fix(kernel): release an event claim when a panic unwinds past it (#173) - #182
Conversation
#173, deferred from #171 on purpose so it would get its own review. `claim_event` tells every concurrent delivery that an event is a duplicate on the strength of a claim row, so whoever takes that claim owes the event either a registered run or the claim back. The `Err` path handed it back; a PANIC did not. An unwind passes every match arm, and a claim leaked that way survived for the life of the boot -- each later delivery answered "duplicate" by the same-boot rule, with no run to carry the event. Only a restart cleared it, because the claim then belonged to a previous boot and was repaired. A `Drop` guard covers the exit that has no `?` to take. The ORDINARY `Err` path still releases explicitly and propagates: a release that fails there strands the claim while later deliveries are told "deduped" with no run behind them, and that is a runtime failure the caller must see rather than one to report and swallow. Release happens before disarming, so a failing release leaves the guard armed and the unwind still attempts a best-effort cleanup on the way out. Best-effort is the right trade only where the alternative is no cleanup at all. First `Drop` impl in the crate, so the trade is stated in the code: it cannot return an error or be `?`-ed, so a failed release there is reported via `eprintln!` (the convention already used in `server/liveness.rs`) and swallowed -- leaving the event exactly where a leak would have left it, no worse. It must not panic: panicking in `Drop` during an unwind aborts the process, turning a stranded event into a dead daemon. Evidence. Every command below is literally runnable from the repository root, and the output is complete: $ shasum -a 256 kernel/relayflowd/src/engine/wake.rs bdc6d17b93f1aa5c3f256deeb5377943a68dec1fad6f018dee4dab88eaeb1cd3 kernel/relayflowd/src/engine/wake.rs $ (cd kernel && PATH="$HOME/.cargo/bin:$PATH" RUSTUP_TOOLCHAIN=stable cargo test -p relayflowd --lib claim_guard) Finished `test` profile [unoptimized + debuginfo] target(s) in 0.33s Running unittests src/lib.rs (target/debug/deps/relayflowd-01b0ec98cb5c81e7) running 4 tests test engine::wake::claim_guard_tests::a_disarmed_guard_leaves_the_claim_alone ... ok test engine::wake::claim_guard_tests::a_guard_only_releases_its_own_run ... ok test engine::wake::claim_guard_tests::an_armed_guard_releases_the_claim_when_dropped ... ok test engine::wake::claim_guard_tests::a_panic_between_claim_and_register_still_releases ... ok test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 25 filtered out; finished in 0.02s # MUTATION: Drop made a no-op -- `if true || !self.armed` $ shasum -a 256 kernel/relayflowd/src/engine/wake.rs 548550ba2430c59fa1dd1760bb763b1d14e9f6a234dbe667b62cb73721bbb524 kernel/relayflowd/src/engine/wake.rs $ (cd kernel && PATH="$HOME/.cargo/bin:$PATH" RUSTUP_TOOLCHAIN=stable cargo test -p relayflowd --lib claim_guard) Compiling relayflowd v0.1.0 (/Users/khaliqgant/AgentWorkforce/flows-173/kernel/relayflowd) Finished `test` profile [unoptimized + debuginfo] target(s) in 2.24s Running unittests src/lib.rs (target/debug/deps/relayflowd-01b0ec98cb5c81e7) running 4 tests test engine::wake::claim_guard_tests::a_disarmed_guard_leaves_the_claim_alone ... ok test engine::wake::claim_guard_tests::an_armed_guard_releases_the_claim_when_dropped ... FAILED test engine::wake::claim_guard_tests::a_guard_only_releases_its_own_run ... ok test engine::wake::claim_guard_tests::a_panic_between_claim_and_register_still_releases ... FAILED failures: ---- engine::wake::claim_guard_tests::an_armed_guard_releases_the_claim_when_dropped stdout ---- thread 'engine::wake::claim_guard_tests::an_armed_guard_releases_the_claim_when_dropped' (18172736) panicked at relayflowd/src/engine/wake.rs:367:9: a dropped armed guard must hand the event back ---- engine::wake::claim_guard_tests::a_panic_between_claim_and_register_still_releases stdout ---- thread 'engine::wake::claim_guard_tests::a_panic_between_claim_and_register_still_releases' (18172735) panicked at relayflowd/src/engine/wake.rs:397:13: spawn_claimed_run blew up between the claim and register note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'engine::wake::claim_guard_tests::a_panic_between_claim_and_register_still_releases' (18172735) panicked at relayflowd/src/engine/wake.rs:401:9: a claim leaked by a panic strands the event until the process restarts failures: engine::wake::claim_guard_tests::a_panic_between_claim_and_register_still_releases engine::wake::claim_guard_tests::an_armed_guard_releases_the_claim_when_dropped test result: FAILED. 2 passed; 2 failed; 0 ignored; 0 measured; 25 filtered out; finished in 0.01s error: test failed, to rerun pass `-p relayflowd --lib` # RESTORED $ shasum -a 256 kernel/relayflowd/src/engine/wake.rs bdc6d17b93f1aa5c3f256deeb5377943a68dec1fad6f018dee4dab88eaeb1cd3 kernel/relayflowd/src/engine/wake.rs Exactly the two tests that should fail do, while the disarmed and wrong-run cases keep passing -- so they pin the release without also pinning its absence. The panic case runs through `catch_unwind` and asserts the panic actually happened, so it cannot pass by never unwinding. Kernel workspace at this head: 156 passed, 0 failed. Known coverage boundary, stated rather than glossed: the panic test constructs `ClaimGuard` directly instead of injecting a panic through `submit_event`, so it pins the guard's contract, not its placement. The placement is pinned by reading -- armed immediately after the claim, disarmed only after `spawn_claimed_run` returns Ok -- and injecting a panic into production code would need a test-only seam in the very span this change exists to protect. Four earlier drafts were rejected by review, each fairly, and only one was about the code: the first described the evidence instead of showing it; the second abbreviated cargo's test paths while calling the block literal; the third routed the ordinary `Err` path through `Drop`, which silently downgraded a propagating release to best-effort -- a real regression; the fourth printed commands without the `cd kernel` they were actually run in, so "runnable from the root" was false. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
|
Warning Review limit reachedNext included review available in 12 minutes. View limit detailsLimit details: You’ve used the included review currently available. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (1)
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing. Comment |
…eview rounds Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
…ssues surfaced Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
Closes #167. The row claimed gate 2 was missing "a test proving a duplicate event does not double-execute". That was already false when #167 was filed, and tonight's work went further, so the row understated progress and the gate at once. Done, each claim checked against main rather than remembered: sequential duplicates (#14); concurrent racing deliveries under the production topology of one `Engine` per protocol request (#171); claims surviving the process that made them (#171 boot id, #182 panic unwind); resume adopting only a journal it can actually use (#177, #186). Still missing, narrowly: the RFC-0001 Appendix A wake-time context contract — nothing specifies what `wake_context` guarantees, or that a resumed run observes the same context rather than a recomputed one. And the correction #167 cared about most: the row was understating the gate. RFC-0001 §3's bar is `hn-monitor` running as a relayflow in production on its real events with zero bespoke persistence, not a passing test suite. Evidence at the merged head 8e57b17: - signoff: local 3-lens preswarm, maintainability / history / structure all REVIEW_PASSED - CI: none applies. `cloud-runtime-artifact.yml` filters on `kernel/**`, `sdk/**`, `testdata/**` and `scripts/cloud-artifact*`; this touches only `ops/SCOREBOARD.md`, so no artifact run was triggered — verified by reading the workflow's `paths:` rather than waiting on a run that was never going to start. The `review` check is red for the reason common to every flows PR: the gate invokes `agent-relay` and no step installs it. No counts in the row, deliberately — counts drift with the base, PR numbers and test names do not.
Nothing verified main. `cloud-runtime-artifact.yml` triggered only on `pull_request` and `workflow_dispatch`, and `review-swarm.yml` only on `pull_request` — so every PR is checked at its own head and never as merged, and since we squash-merge onto a main that has moved since that CI ran, the composed result went unverified. Not theoretical: twelve PRs merged on 2026-09-05 across the exactly-once claim path (#171, #182), resume adoption (#177, #186), the authored-flow executor (#184, #187) and the CLI run loop (#180) — each green on its own branch, the composed tree never run until dispatched by hand: ``` run 33987924703 workflow_dispatch main completed/success ed917bf ``` Main is fine. But nobody knew that, and finding out required knowing to ask. Without this, a bad compose surfaces as an unrelated PR going red — the most expensive way to find it, since the author debugs their own change first. Tonight already produced three failures on PRs that belonged to something else (#179, #185, the #174 chain), and each cost a tick to attribute. Paths are deliberately not filtered on the push trigger: on a PR the question is "does this change affect the runtime", on main it is "is the tree good", and a docs-only merge can land on a tree someone else broke. Evidence at the merged head d1cb32a: - signoff: local 3-lens preswarm, maintainability / history / structure all REVIEW_PASSED - CI: run 33988314276 success on d1cb32a, verified by headSha and event=pull_request - YAML parsed and asserted: triggers are `pull_request`, `push`, `workflow_dispatch`; push is branch-scoped to main; the pull_request path filter is unchanged
Closes #173, deferred from #171 on purpose so it would get its own review rather than riding under an exactly-once fix.
The gap
claim_eventtells every concurrent delivery that an event is a duplicate on the strength of a claim row, so whoever takes that claim owes the event either a registered run or the claim back.The
Errpath handed it back. A panic did not — an unwind passes every match arm, and a claim leaked that way survived for the life of the boot: each later delivery answered "duplicate" by the same-boot rule, with no run to carry the event. Only a restart cleared it, because the claim then belonged to a previous boot and was repaired.The shape of the fix
A
Dropguard covers the exit that has no?to take. The ordinaryErrpath still releases explicitly and propagates — a release that fails there strands the claim while later deliveries are told "deduped" with nothing behind them, and that is a runtime failure the caller must see, not one to report and swallow. Release happens before disarming, so a failing release leaves the guard armed and the unwind still attempts best-effort cleanup on the way out.This is the crate's first
Dropimpl, so the trade is stated in the code: it cannot return an error or be?-ed, so a failed release there is reported viaeprintln!— the convention already used inserver/liveness.rs— and swallowed, leaving the event exactly where a leak would have left it, no worse. It must not panic: panicking inDropduring an unwind aborts the process, turning a stranded event into a dead daemon.Evidence
Four tests, and the mutation that gates them. Making
Dropa no-op (if true || !self.armed) fails exactly the two that should fail —an_armed_guard_releases_the_claim_when_droppedanda_panic_between_claim_and_register_still_releases— whilea_disarmed_guard_leaves_the_claim_aloneanda_guard_only_releases_its_own_runkeep passing. So they pin the release without also pinning its absence.The panic case runs through
catch_unwindand asserts the panic actually happened, so it cannot pass by never unwinding.Full transcript with runnable commands and sha256 before/mutated/restored is in the commit message. Kernel workspace: 156 passed, 0 failed, no warnings.
Known boundary, stated rather than glossed
The panic test constructs
ClaimGuarddirectly rather than injecting a panic throughsubmit_event, so it pins the guard's contract, not its placement. Placement is pinned by reading — armed immediately after the claim, disarmed only afterspawn_claimed_runreturnsOk— and injecting a panic into production code would need a test-only seam in the very span this change exists to protect.Review history
Four drafts were rejected, each fairly, and one was about the code: routing the ordinary
Errpath throughDropsilently downgraded a propagating release to best-effort. That was a real regression the review caught, not a wording problem. The other three were evidence claims — describing output instead of showing it, abbreviating cargo's test paths while calling the block literal, and printing commands without thecd kernelthey were actually run in.🤖 Generated with Claude Code
https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR