ci: let the SDK suite run where no model is reachable - #159
Conversation
#153 enabled the full SDK suite without giving CI a way to run it. One case in live-kernel.test.ts executes the REAL Claude analyzer and fails by default when it cannot -- deliberately, so that a reader never gets a green that proves nothing about gate 2. A GitHub runner has no `claude` binary: LIVE_ANALYZER_UNAVAILABLE: "...analyze-story-claude-cli auth status" exited 1: cannot run "claude": spawnSync claude ENOENT so the step could never pass. That is my defect from #153, not the test's. RELAYFLOWS_ALLOW_ANALYZER_SKIP is the escape hatch the test itself documents, for "environments that knowingly cannot reach a model and are not counting this run as gate evidence". Setting it here is that statement made explicitly: this workflow is not gate-2 acceptance evidence. Gate-2 evidence must come from a machine that can reach a model, and the skipped case prints as much. Every other case in the suite still runs and still gates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
|
Warning Review limit reachedNext included review available in 3 seconds. View limit detailsLimit details: You’ve used the included review currently available. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (1)
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing. Comment |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
#153 enabled the full SDK suite without giving CI a way to run it. One case in live-kernel.test.ts executes the REAL Claude analyzer and fails by default when it cannot -- deliberately, so that a reader never gets a green that proves nothing about gate 2. A GitHub runner has no `claude` binary: LIVE_ANALYZER_UNAVAILABLE: "...analyze-story-claude-cli auth status" exited 1: cannot run "claude": spawnSync claude ENOENT so the step could never pass. That is my defect from #153, not the test's. RELAYFLOWS_ALLOW_ANALYZER_SKIP is the escape hatch the test itself documents, for "environments that knowingly cannot reach a model and are not counting this run as gate evidence". Setting it here is that statement made explicitly: this workflow is not gate-2 acceptance evidence. Gate-2 evidence must come from a machine that can reach a model, and the skipped case prints as much. Every other case in the suite still runs and still gates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Co-authored-by: kjgbot <kjgbot@agentrelay.dev> Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
…ucible Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
Split out of #165, which bundled this with a wholesale revert of the CI workflow. The workflow half restored `ops/cargo.sh` for the kernel test step (that wrapper redirects RUSTUP_HOME to an empty dir on a runner, so rustup cannot choose a toolchain) and dropped the analyzer-skip env, i.e. it reverted #153, #154 and #159 together. Only the sdk tooling is carried here; `.github/` is byte-identical to main. scripts/test.sh runs the same chain the inline `test` script did, in the same order, with `set -eu` for the fail-fast the `&&` chain gave. The new behaviour is the EXIT trap, which prunes `.map` and `.d.ts` from sdk/dist afterward. CI is unaffected either way: the workflow runs the expanded chain minus test:prep, not `npm test`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1 Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
The sdk half of #165, with that PR's CI revert dropped. `.github/` is byte-identical to main (verified: `git diff origin/main --stat -- .github/` returns 0 lines); #165's workflow hunk would have reverted #153, #154 and #159 together, restoring `ops/cargo.sh` for the kernel step, which redirects RUSTUP_HOME to an empty dir on a runner. `scripts/test.sh` runs the same chain the inline `test` script did, in the same order, with `set -eu` supplying the fail-fast the `&&` chain gave. Pruning is source maps only. Deleting `.d.ts` would have left `"types": "./dist/index.d.ts"` pointing at a file `npm test` had just removed. Every current in-repo consumer imports `.js` (`ops/probes/**`, `workflows/drive.yaml`, `workflows/drive-cloud.yaml`, `testdata/backlog-picker.flow.yaml`), so nothing breaks today — but the next TypeScript consumer would meet a failure caused by running the tests. Evidence at this head: - independent signoff: local 3-lens preswarm review, maintainability / history / structure all REVIEW_PASSED. The history lens caught a false scope claim in an earlier message ("only ops/probes consume the sdk"), which was corrected. - CI: linux-x64-artifact success, packed-consumer pass, CodeRabbit pass. The `review` check fails for a reason independent of this change and common to every flows PR: the gate invokes `agent-relay` and no step installs it (exit 127). Supersedes the sdk half of #165.
Second defect in my #153, found once the suite actually ran far enough to hit it.
The problem
live-kernel.test.tsruns one case against the real Claude analyzer and fails by default when it cannot reach one. That default is deliberate and correct — from the test's own comment:A GitHub runner has no
claudebinary:So the step could never pass. #153 turned on the full suite without giving CI any way to run it — my defect, not the test's.
The fix
RELAYFLOWS_ALLOW_ANALYZER_SKIP=1, which is the escape hatch the test documents for "environments that knowingly cannot reach a model and are not counting this run as gate evidence."Setting it is an explicit claim, not a convenience: this workflow is not gate-2 acceptance evidence. Gate-2 evidence has to come from a machine that can actually reach a model, and the skipped case prints exactly that in its own output. Every other case in the suite still runs and still gates.
I put the reasoning in the workflow rather than the commit alone, because the next person to see a skipped gate-2 case in a CI log deserves to find out why without archaeology.
Where this leaves the CI chain
Three of my own defects, in order:
rustup could not choose a version of cargo. Fixed by ci: stop routing CI's cargo through the sandbox wrapper #154, merged.worker_capacity::default_capacity_one_... ok.Also seen in that run, and not addressed here:
ENOENT '.relayflow/backlog-picker-entry.json') fired again. Real flake, separate issue.agent::rung_c_sigkill_...hung for 30 min and was cancelled on two earlier runs, but passed on the rerun, so it looks flaky rather than deterministic. I had wrongly told feat(sdk): settle data and code gate contract #139 it was branch-specific; corrected there.🤖 Generated with Claude Code
https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR