Skip to content

ci: run the kernel and full SDK suites - #153

Merged
khaliqgant merged 1 commit into
mainfrom
ci/run-the-suites-0904
Sep 3, 2026
Merged

ci: run the kernel and full SDK suites#153
khaliqgant merged 1 commit into
mainfrom
ci/run-the-suites-0904

Conversation

@kjgbot

@kjgbot kjgbot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

The gap

.github/workflows/cloud-runtime-artifact.yml is this repo's only workflow. Before this PR:

  • the only cargo invocation was cargo build --locked --release -p relayflowd. cargo test appeared nowhere — the kernel suite (130 tests) never ran in CI;
  • vitest ran exactly four named files (typed-output, validate, spec-parity, deterministic-llm) out of 25 collected.

Every kernel-side defect found on 2026-09-03 was invisible to CI by construction: an exactly-once double-fire where one effect fired twice; a $ref cycle that aborted the daemon and re-ran the effect on each resume; and two tests in the tree encoding opposite contracts that both passed, because neither was ever executed.

This is also why six of eight independent signoffs that day found P0s in PRs that CI called green.

The change

.github/ only — two steps, one file. No kernel/, sdk/, or testdata/ change.

ops/cargo.sh rather than bare cargo: npm test's test:prep already shells out to that wrapper, which redirects CARGO_HOME/RUSTUP_HOME to $HOME/.relayflows-toolchain. Using it here means both steps share one toolchain home and target dir instead of populating two registries. Its bootstrap branch is not taken on a runner — dtolnay/rust-toolchain has already put cargo on PATH, so it takes the first branch and execs.

npm test rather than a longer file list: it's the repo's own entry point and a strict superset of what the step did before (test:prep && typecheck && build && typecheck:tests && vitest run). test:prep builds the relayflowd binary tests/live-kernel.test.ts execs, and that build is why a bare vitest run would be wrong here — several files fail at collection without sdk/dist.

Local verification

Kernel, the exact command the workflow now runs — 130 passed, 0 failed:

$ cd kernel && sh ../ops/cargo.sh test --workspace
test result: ok. 22 passed; 34 passed; 42 passed; 18 passed; 5 passed;
              4 passed; 3 passed; 1 passed; 1 passed  (0 failed throughout)

SDK full suite — 464 passed, 30 skipped, 0 failed:

$ cd sdk && ./node_modules/.bin/vitest run
Test Files  1 failed | 23 passed | 1 skipped (25)
     Tests  437 passed | 30 skipped (467)

The one failed file was tests/live-kernel.test.ts, and the cause was my harness, not the tree — that run had RELAYFLOWD_BIN exported to a path built from a failed cargo metadata parse (/debug/relayflowd). Re-run with the real binary, and again with the variable unset, which is the condition CI actually runs under:

$ RELAYFLOWD_BIN=<real path> vitest run tests/live-kernel.test.ts   → 27 passed
$ env -u RELAYFLOWD_BIN      vitest run tests/live-kernel.test.ts   → 27 passed

So live-kernel self-locates its binary with no environment variable, via test:prep + locateRelayflowd().

--locked considered and not used

The test step omits --locked while the adjacent build step has it. Deliberate: lockfile drift is already caught by that adjacent cargo build --locked --release, and cargo test --workspace left Cargo.lock untouched (git status --porcelain showed only the workflow file) — which is the evidence --locked would have passed. The step ships as the command with test counts behind it rather than one inferred to be equivalent.

Expected effect

Both suites are green locally at this base (27b7151), so this should not turn CI red. If it does, the failure is real and pre-existing — it was simply never executed — and the correct response is to report it, not to weaken the workflow or fix the test here.

Marginal cost is the test run only; the toolchain install, kernel build, and npm ci were already paid for.

Full literal evidence: ops/reviews/20260904-ci-coverage-gap.md.

Not self-merging — this changes the gate that judges everything else in the v2 lane, and turning CI red overnight on a repo with open PRs is a call for @khaliqgant, not for me.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

The only workflow in the repo built the kernel but never tested it, and ran
four of the SDK suite's 25 test files. Every kernel-side defect found on
2026-09-03 was invisible to CI by construction, including an exactly-once
double-fire and two tests in the tree that encoded opposite contracts and
both passed because neither ever ran.

Add `cargo test --workspace` through ops/cargo.sh, which is the wrapper
`npm test`'s test:prep already uses, so both steps share one toolchain home
rather than populating two registries. Replace the four named vitest files
with `npm test`, the repo's own entry point and a strict superset of what the
step did before -- it builds sdk/dist, without which several files fail at
collection, and the relayflowd binary that live-kernel.test.ts execs.

Both suites verified locally at this base: kernel 130 passed, SDK 464 passed
with 30 skipped, 0 failed. Evidence in ops/reviews/20260904-ci-coverage-gap.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Review Change Stack

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Free

Run ID: a2261dda-5173-4fb0-982a-bdff801c4486

📥 Commits

Reviewing files that changed from the base of the PR and between 27b7151 and ecfce54.

📒 Files selected for processing (2)
  • .github/workflows/cloud-runtime-artifact.yml
  • ops/reviews/20260904-ci-coverage-gap.md

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The CI workflow adds kernel workspace tests after the release build. It replaces selected SDK commands with npm test. A review document records the coverage changes, local verification, lockfile behavior, and expected CI execution.

Changes

CI coverage expansion

Layer / File(s) Summary
Workflow test entry points
.github/workflows/cloud-runtime-artifact.yml
The workflow runs the complete kernel workspace test suite through ops/cargo.sh. The SDK step uses the repository-wide npm test command.
Coverage review and verification
ops/reviews/20260904-ci-coverage-gap.md
The review records the previous coverage gaps, local test results, --locked handling, lockfile behavior, and expected CI execution.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.


Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

@khaliqgant
khaliqgant merged commit 3725025 into main Sep 3, 2026
1 of 2 checks passed
kjgbot added a commit that referenced this pull request Sep 4, 2026
The kernel test step added in #153 fails on every PR:

  error: rustup could not choose a version of cargo to run, because one
  wasn't specified explicitly, and no default is configured

ops/cargo.sh unconditionally redirects RUSTUP_HOME to
$HOME/.relayflows-toolchain/rustup. On a runner that directory is empty, so
the rustup shim has no toolchain to resolve. It passed locally only because
every command in that verification had RUSTUP_TOOLCHAIN=stable in front of
it, and the workflow was written without it -- so the evidence behind #153
was not the command #153 shipped.

The wrapper exists for a cloud sandbox, where the propagated tree drops files
over a per-file size cap and a toolchain must be obtainable per step. A
GitHub runner has neither constraint, and dtolnay/rust-toolchain has already
installed a default toolchain into the standard home. Use plain cargo, as the
adjacent release build already does and as its passing runs demonstrate.

The SDK step has the same problem through npm test -> test:prep, so expand
npm test to its constituents and drop test:prep. Its only real output is the
relayflowd binary live-kernel.test.ts execs, and this job has already built
one: point RELAYFLOWD_BIN at the release binary instead of compiling a second
debug copy through a wrapper that cannot run here. test:prep's chmod of the
preflight CLI fixtures is kept inline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Co-authored-by: kjgbot <kjgbot@agentrelay.dev>
kjgbot added a commit that referenced this pull request Sep 4, 2026
The kernel test step added in #153 fails on every PR:

  error: rustup could not choose a version of cargo to run, because one
  wasn't specified explicitly, and no default is configured

ops/cargo.sh unconditionally redirects RUSTUP_HOME to
$HOME/.relayflows-toolchain/rustup. On a runner that directory is empty, so
the rustup shim has no toolchain to resolve. It passed locally only because
every command in that verification had RUSTUP_TOOLCHAIN=stable in front of
it, and the workflow was written without it -- so the evidence behind #153
was not the command #153 shipped.

The wrapper exists for a cloud sandbox, where the propagated tree drops files
over a per-file size cap and a toolchain must be obtainable per step. A
GitHub runner has neither constraint, and dtolnay/rust-toolchain has already
installed a default toolchain into the standard home. Use plain cargo, as the
adjacent release build already does and as its passing runs demonstrate.

The SDK step has the same problem through npm test -> test:prep, so expand
npm test to its constituents and drop test:prep. Its only real output is the
relayflowd binary live-kernel.test.ts execs, and this job has already built
one: point RELAYFLOWD_BIN at the release binary instead of compiling a second
debug copy through a wrapper that cannot run here. test:prep's chmod of the
preflight CLI fixtures is kept inline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Co-authored-by: kjgbot <kjgbot@agentrelay.dev>
Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot added a commit that referenced this pull request Sep 4, 2026
#153 enabled the full SDK suite without giving CI a way to run it. One case in
live-kernel.test.ts executes the REAL Claude analyzer and fails by default when
it cannot -- deliberately, so that a reader never gets a green that proves
nothing about gate 2. A GitHub runner has no `claude` binary:

  LIVE_ANALYZER_UNAVAILABLE: "...analyze-story-claude-cli auth status" exited 1:
  cannot run "claude": spawnSync claude ENOENT

so the step could never pass. That is my defect from #153, not the test's.

RELAYFLOWS_ALLOW_ANALYZER_SKIP is the escape hatch the test itself documents,
for "environments that knowingly cannot reach a model and are not counting this
run as gate evidence". Setting it here is that statement made explicitly: this
workflow is not gate-2 acceptance evidence. Gate-2 evidence must come from a
machine that can reach a model, and the skipped case prints as much.

Every other case in the suite still runs and still gates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Co-authored-by: kjgbot <kjgbot@agentrelay.dev>
kjgbot added a commit that referenced this pull request Sep 4, 2026
#153 enabled the full SDK suite without giving CI a way to run it. One case in
live-kernel.test.ts executes the REAL Claude analyzer and fails by default when
it cannot -- deliberately, so that a reader never gets a green that proves
nothing about gate 2. A GitHub runner has no `claude` binary:

  LIVE_ANALYZER_UNAVAILABLE: "...analyze-story-claude-cli auth status" exited 1:
  cannot run "claude": spawnSync claude ENOENT

so the step could never pass. That is my defect from #153, not the test's.

RELAYFLOWS_ALLOW_ANALYZER_SKIP is the escape hatch the test itself documents,
for "environments that knowingly cannot reach a model and are not counting this
run as gate evidence". Setting it here is that statement made explicitly: this
workflow is not gate-2 acceptance evidence. Gate-2 evidence must come from a
machine that can reach a model, and the skipped case prints as much.

Every other case in the suite still runs and still gates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Co-authored-by: kjgbot <kjgbot@agentrelay.dev>
Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot pushed a commit that referenced this pull request Sep 5, 2026
Split out of #165, which bundled this with a wholesale revert of the CI
workflow. The workflow half restored `ops/cargo.sh` for the kernel test
step (that wrapper redirects RUSTUP_HOME to an empty dir on a runner, so
rustup cannot choose a toolchain) and dropped the analyzer-skip env, i.e.
it reverted #153, #154 and #159 together. Only the sdk tooling is carried
here; `.github/` is byte-identical to main.

scripts/test.sh runs the same chain the inline `test` script did, in the
same order, with `set -eu` for the fail-fast the `&&` chain gave. The new
behaviour is the EXIT trap, which prunes `.map` and `.d.ts` from
sdk/dist afterward.

CI is unaffected either way: the workflow runs the expanded chain minus
test:prep, not `npm test`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot added a commit that referenced this pull request Sep 5, 2026
The sdk half of #165, with that PR's CI revert dropped. `.github/` is
byte-identical to main (verified: `git diff origin/main --stat -- .github/`
returns 0 lines); #165's workflow hunk would have reverted #153, #154 and #159
together, restoring `ops/cargo.sh` for the kernel step, which redirects
RUSTUP_HOME to an empty dir on a runner.

`scripts/test.sh` runs the same chain the inline `test` script did, in the same
order, with `set -eu` supplying the fail-fast the `&&` chain gave.

Pruning is source maps only. Deleting `.d.ts` would have left
`"types": "./dist/index.d.ts"` pointing at a file `npm test` had just removed.
Every current in-repo consumer imports `.js` (`ops/probes/**`,
`workflows/drive.yaml`, `workflows/drive-cloud.yaml`,
`testdata/backlog-picker.flow.yaml`), so nothing breaks today — but the next
TypeScript consumer would meet a failure caused by running the tests.

Evidence at this head:
- independent signoff: local 3-lens preswarm review, maintainability / history /
  structure all REVIEW_PASSED. The history lens caught a false scope claim in an
  earlier message ("only ops/probes consume the sdk"), which was corrected.
- CI: linux-x64-artifact success, packed-consumer pass, CodeRabbit pass. The
  `review` check fails for a reason independent of this change and common to
  every flows PR: the gate invokes `agent-relay` and no step installs it
  (exit 127).

Supersedes the sdk half of #165.
kjgbot added a commit that referenced this pull request Sep 5, 2026
…190)

Addresses #156 — but not as filed.

#156 read an intermittent `ENOENT ... .relayflow/backlog-picker-entry.json` as a relative path racing under vitest parallelism. That cause does not hold: every caller of the flow's steps already runs in its own `mkdtempSync` directory with an explicit `cwd`.

What actually happens is simpler and fails nothing. Several tests assert a step **fails** — `select-entry` refusing an unactionable backlog, `emit-package` with no entry to read — and `execFileSync` echoes a failing child's stderr into the parent's, so a green run prints a full ENOENT stack trace and looks broken. Measured over six consecutive full suites on main before changing anything: `662 passed, 3 skipped` every time, with the ENOENT text present every time.

So the defect is the noise, because it is what got misdiagnosed. Child stderr is captured rather than echoed; the text remains on the thrown error for any test that wants to assert on it.

- with the change, three consecutive runs: `662 passed | 3 skipped`, ENOENT lines **0** each time
- with the `stdio` line reverted: `662 passed | 3 skipped`, ENOENT lines **2**

Tests pass either way, which is exactly why this was read as a flake.

**#156 is deliberately left open.** Its first run genuinely showed `Test Files 1 failed`, so something failed then; the suite has changed substantially since (#153 widened it from four files to all of them). I could not reproduce it in six runs, and would rather say so than declare a bug I never observed to be absent.

Evidence at the merged head 7758814:
- signoff: local 3-lens preswarm, maintainability / history / structure all REVIEW_PASSED
- CI: run 33987493221 success on 7758814, verified by headSha
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants