ci: run the kernel and full SDK suites - #153
Conversation
The only workflow in the repo built the kernel but never tested it, and ran four of the SDK suite's 25 test files. Every kernel-side defect found on 2026-09-03 was invisible to CI by construction, including an exactly-once double-fire and two tests in the tree that encoded opposite contracts and both passed because neither ever ran. Add `cargo test --workspace` through ops/cargo.sh, which is the wrapper `npm test`'s test:prep already uses, so both steps share one toolchain home rather than populating two registries. Replace the four named vitest files with `npm test`, the repo's own entry point and a strict superset of what the step did before -- it builds sdk/dist, without which several files fail at collection, and the relayflowd binary that live-kernel.test.ts execs. Both suites verified locally at this base: kernel 130 passed, SDK 464 passed with 30 skipped, 0 failed. Evidence in ops/reviews/20260904-ci-coverage-gap.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe CI workflow adds kernel workspace tests after the release build. It replaces selected SDK commands with ChangesCI coverage expansion
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing. Comment |
The kernel test step added in #153 fails on every PR: error: rustup could not choose a version of cargo to run, because one wasn't specified explicitly, and no default is configured ops/cargo.sh unconditionally redirects RUSTUP_HOME to $HOME/.relayflows-toolchain/rustup. On a runner that directory is empty, so the rustup shim has no toolchain to resolve. It passed locally only because every command in that verification had RUSTUP_TOOLCHAIN=stable in front of it, and the workflow was written without it -- so the evidence behind #153 was not the command #153 shipped. The wrapper exists for a cloud sandbox, where the propagated tree drops files over a per-file size cap and a toolchain must be obtainable per step. A GitHub runner has neither constraint, and dtolnay/rust-toolchain has already installed a default toolchain into the standard home. Use plain cargo, as the adjacent release build already does and as its passing runs demonstrate. The SDK step has the same problem through npm test -> test:prep, so expand npm test to its constituents and drop test:prep. Its only real output is the relayflowd binary live-kernel.test.ts execs, and this job has already built one: point RELAYFLOWD_BIN at the release binary instead of compiling a second debug copy through a wrapper that cannot run here. test:prep's chmod of the preflight CLI fixtures is kept inline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Co-authored-by: kjgbot <kjgbot@agentrelay.dev>
The kernel test step added in #153 fails on every PR: error: rustup could not choose a version of cargo to run, because one wasn't specified explicitly, and no default is configured ops/cargo.sh unconditionally redirects RUSTUP_HOME to $HOME/.relayflows-toolchain/rustup. On a runner that directory is empty, so the rustup shim has no toolchain to resolve. It passed locally only because every command in that verification had RUSTUP_TOOLCHAIN=stable in front of it, and the workflow was written without it -- so the evidence behind #153 was not the command #153 shipped. The wrapper exists for a cloud sandbox, where the propagated tree drops files over a per-file size cap and a toolchain must be obtainable per step. A GitHub runner has neither constraint, and dtolnay/rust-toolchain has already installed a default toolchain into the standard home. Use plain cargo, as the adjacent release build already does and as its passing runs demonstrate. The SDK step has the same problem through npm test -> test:prep, so expand npm test to its constituents and drop test:prep. Its only real output is the relayflowd binary live-kernel.test.ts execs, and this job has already built one: point RELAYFLOWD_BIN at the release binary instead of compiling a second debug copy through a wrapper that cannot run here. test:prep's chmod of the preflight CLI fixtures is kept inline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Co-authored-by: kjgbot <kjgbot@agentrelay.dev> Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
#153 enabled the full SDK suite without giving CI a way to run it. One case in live-kernel.test.ts executes the REAL Claude analyzer and fails by default when it cannot -- deliberately, so that a reader never gets a green that proves nothing about gate 2. A GitHub runner has no `claude` binary: LIVE_ANALYZER_UNAVAILABLE: "...analyze-story-claude-cli auth status" exited 1: cannot run "claude": spawnSync claude ENOENT so the step could never pass. That is my defect from #153, not the test's. RELAYFLOWS_ALLOW_ANALYZER_SKIP is the escape hatch the test itself documents, for "environments that knowingly cannot reach a model and are not counting this run as gate evidence". Setting it here is that statement made explicitly: this workflow is not gate-2 acceptance evidence. Gate-2 evidence must come from a machine that can reach a model, and the skipped case prints as much. Every other case in the suite still runs and still gates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Co-authored-by: kjgbot <kjgbot@agentrelay.dev>
#153 enabled the full SDK suite without giving CI a way to run it. One case in live-kernel.test.ts executes the REAL Claude analyzer and fails by default when it cannot -- deliberately, so that a reader never gets a green that proves nothing about gate 2. A GitHub runner has no `claude` binary: LIVE_ANALYZER_UNAVAILABLE: "...analyze-story-claude-cli auth status" exited 1: cannot run "claude": spawnSync claude ENOENT so the step could never pass. That is my defect from #153, not the test's. RELAYFLOWS_ALLOW_ANALYZER_SKIP is the escape hatch the test itself documents, for "environments that knowingly cannot reach a model and are not counting this run as gate evidence". Setting it here is that statement made explicitly: this workflow is not gate-2 acceptance evidence. Gate-2 evidence must come from a machine that can reach a model, and the skipped case prints as much. Every other case in the suite still runs and still gates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Co-authored-by: kjgbot <kjgbot@agentrelay.dev> Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
Split out of #165, which bundled this with a wholesale revert of the CI workflow. The workflow half restored `ops/cargo.sh` for the kernel test step (that wrapper redirects RUSTUP_HOME to an empty dir on a runner, so rustup cannot choose a toolchain) and dropped the analyzer-skip env, i.e. it reverted #153, #154 and #159 together. Only the sdk tooling is carried here; `.github/` is byte-identical to main. scripts/test.sh runs the same chain the inline `test` script did, in the same order, with `set -eu` for the fail-fast the `&&` chain gave. The new behaviour is the EXIT trap, which prunes `.map` and `.d.ts` from sdk/dist afterward. CI is unaffected either way: the workflow runs the expanded chain minus test:prep, not `npm test`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
The sdk half of #165, with that PR's CI revert dropped. `.github/` is byte-identical to main (verified: `git diff origin/main --stat -- .github/` returns 0 lines); #165's workflow hunk would have reverted #153, #154 and #159 together, restoring `ops/cargo.sh` for the kernel step, which redirects RUSTUP_HOME to an empty dir on a runner. `scripts/test.sh` runs the same chain the inline `test` script did, in the same order, with `set -eu` supplying the fail-fast the `&&` chain gave. Pruning is source maps only. Deleting `.d.ts` would have left `"types": "./dist/index.d.ts"` pointing at a file `npm test` had just removed. Every current in-repo consumer imports `.js` (`ops/probes/**`, `workflows/drive.yaml`, `workflows/drive-cloud.yaml`, `testdata/backlog-picker.flow.yaml`), so nothing breaks today — but the next TypeScript consumer would meet a failure caused by running the tests. Evidence at this head: - independent signoff: local 3-lens preswarm review, maintainability / history / structure all REVIEW_PASSED. The history lens caught a false scope claim in an earlier message ("only ops/probes consume the sdk"), which was corrected. - CI: linux-x64-artifact success, packed-consumer pass, CodeRabbit pass. The `review` check fails for a reason independent of this change and common to every flows PR: the gate invokes `agent-relay` and no step installs it (exit 127). Supersedes the sdk half of #165.
…190) Addresses #156 — but not as filed. #156 read an intermittent `ENOENT ... .relayflow/backlog-picker-entry.json` as a relative path racing under vitest parallelism. That cause does not hold: every caller of the flow's steps already runs in its own `mkdtempSync` directory with an explicit `cwd`. What actually happens is simpler and fails nothing. Several tests assert a step **fails** — `select-entry` refusing an unactionable backlog, `emit-package` with no entry to read — and `execFileSync` echoes a failing child's stderr into the parent's, so a green run prints a full ENOENT stack trace and looks broken. Measured over six consecutive full suites on main before changing anything: `662 passed, 3 skipped` every time, with the ENOENT text present every time. So the defect is the noise, because it is what got misdiagnosed. Child stderr is captured rather than echoed; the text remains on the thrown error for any test that wants to assert on it. - with the change, three consecutive runs: `662 passed | 3 skipped`, ENOENT lines **0** each time - with the `stdio` line reverted: `662 passed | 3 skipped`, ENOENT lines **2** Tests pass either way, which is exactly why this was read as a flake. **#156 is deliberately left open.** Its first run genuinely showed `Test Files 1 failed`, so something failed then; the suite has changed substantially since (#153 widened it from four files to all of them). I could not reproduce it in six runs, and would rather say so than declare a bug I never observed to be absent. Evidence at the merged head 7758814: - signoff: local 3-lens preswarm, maintainability / history / structure all REVIEW_PASSED - CI: run 33987493221 success on 7758814, verified by headSha
The gap
.github/workflows/cloud-runtime-artifact.ymlis this repo's only workflow. Before this PR:cargo build --locked --release -p relayflowd.cargo testappeared nowhere — the kernel suite (130 tests) never ran in CI;typed-output,validate,spec-parity,deterministic-llm) out of 25 collected.Every kernel-side defect found on 2026-09-03 was invisible to CI by construction: an exactly-once double-fire where one effect fired twice; a
$refcycle that aborted the daemon and re-ran the effect on each resume; and two tests in the tree encoding opposite contracts that both passed, because neither was ever executed.This is also why six of eight independent signoffs that day found P0s in PRs that CI called green.
The change
.github/only — two steps, one file. Nokernel/,sdk/, ortestdata/change.ops/cargo.shrather than barecargo:npm test'stest:prepalready shells out to that wrapper, which redirectsCARGO_HOME/RUSTUP_HOMEto$HOME/.relayflows-toolchain. Using it here means both steps share one toolchain home and target dir instead of populating two registries. Its bootstrap branch is not taken on a runner —dtolnay/rust-toolchainhas already put cargo onPATH, so it takes the first branch and execs.npm testrather than a longer file list: it's the repo's own entry point and a strict superset of what the step did before (test:prep && typecheck && build && typecheck:tests && vitest run).test:prepbuilds therelayflowdbinarytests/live-kernel.test.tsexecs, and that build is why a barevitest runwould be wrong here — several files fail at collection withoutsdk/dist.Local verification
Kernel, the exact command the workflow now runs — 130 passed, 0 failed:
SDK full suite — 464 passed, 30 skipped, 0 failed:
The one failed file was
tests/live-kernel.test.ts, and the cause was my harness, not the tree — that run hadRELAYFLOWD_BINexported to a path built from a failedcargo metadataparse (/debug/relayflowd). Re-run with the real binary, and again with the variable unset, which is the condition CI actually runs under:So
live-kernelself-locates its binary with no environment variable, viatest:prep+locateRelayflowd().--lockedconsidered and not usedThe test step omits
--lockedwhile the adjacent build step has it. Deliberate: lockfile drift is already caught by that adjacentcargo build --locked --release, andcargo test --workspaceleftCargo.lockuntouched (git status --porcelainshowed only the workflow file) — which is the evidence--lockedwould have passed. The step ships as the command with test counts behind it rather than one inferred to be equivalent.Expected effect
Both suites are green locally at this base (
27b7151), so this should not turn CI red. If it does, the failure is real and pre-existing — it was simply never executed — and the correct response is to report it, not to weaken the workflow or fix the test here.Marginal cost is the test run only; the toolchain install, kernel build, and
npm ciwere already paid for.Full literal evidence:
ops/reviews/20260904-ci-coverage-gap.md.Not self-merging — this changes the gate that judges everything else in the v2 lane, and turning CI red overnight on a repo with open PRs is a call for @khaliqgant, not for me.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR