Skip to content

ci: let the SDK suite run where no model is reachable - #159

Merged
kjgbot merged 1 commit into
mainfrom
ci/analyzer-skip-0904
Sep 4, 2026
Merged

ci: let the SDK suite run where no model is reachable#159
kjgbot merged 1 commit into
mainfrom
ci/analyzer-skip-0904

Conversation

@kjgbot

@kjgbot kjgbot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Second defect in my #153, found once the suite actually ran far enough to hit it.

The problem

live-kernel.test.ts runs one case against the real Claude analyzer and fails by default when it cannot reach one. That default is deliberate and correct — from the test's own comment:

An unavailable analyzer is diagnostics, never acceptance, so this FAILS by default. Skipping is the opt-in, not the default: a reader who runs the suite without special knowledge must not get a green that proves nothing about gate 2.

A GitHub runner has no claude binary:

LIVE_ANALYZER_UNAVAILABLE: "/home/runner/work/flows/flows/testdata/preflight/analyze-story-claude-cli auth status"
exited 1: cannot run "claude": spawnSync claude ENOENT
— failing because gate-2 acceptance requires the real analyzer to execute.

So the step could never pass. #153 turned on the full suite without giving CI any way to run it — my defect, not the test's.

The fix

RELAYFLOWS_ALLOW_ANALYZER_SKIP=1, which is the escape hatch the test documents for "environments that knowingly cannot reach a model and are not counting this run as gate evidence."

Setting it is an explicit claim, not a convenience: this workflow is not gate-2 acceptance evidence. Gate-2 evidence has to come from a machine that can actually reach a model, and the skipped case prints exactly that in its own output. Every other case in the suite still runs and still gates.

I put the reasoning in the workflow rather than the commit alone, because the next person to see a skipped gate-2 case in a CI log deserves to find out why without archaeology.

Where this leaves the CI chain

Three of my own defects, in order:

  1. ci: run the kernel and full SDK suites #153 enabled the suites but routed cargo through the sandbox wrapper → rustup could not choose a version of cargo. Fixed by ci: stop routing CI's cargo through the sandbox wrapper #154, merged.
  2. P1: #137 made dispatch order to a capacity-1 worker nondeterministic (60/60 → ~15% failure) #155, a real pre-existing dispatch race the suites then caught. Fixed by fix(kernel): a dead leased attempt serves no retry delay #158confirmed on the runner: worker_capacity::default_capacity_one_... ok.
  3. This one.

Also seen in that run, and not addressed here:

🤖 Generated with Claude Code

https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

#153 enabled the full SDK suite without giving CI a way to run it. One case in
live-kernel.test.ts executes the REAL Claude analyzer and fails by default when
it cannot -- deliberately, so that a reader never gets a green that proves
nothing about gate 2. A GitHub runner has no `claude` binary:

  LIVE_ANALYZER_UNAVAILABLE: "...analyze-story-claude-cli auth status" exited 1:
  cannot run "claude": spawnSync claude ENOENT

so the step could never pass. That is my defect from #153, not the test's.

RELAYFLOWS_ALLOW_ANALYZER_SKIP is the escape hatch the test itself documents,
for "environments that knowingly cannot reach a model and are not counting this
run as gate evidence". Setting it here is that statement made explicitly: this
workflow is not gate-2 acceptance evidence. Gate-2 evidence must come from a
machine that can reach a model, and the skipped case prints as much.

Every other case in the suite still runs and still gates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 3 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Free

Run ID: 092c8488-bde1-46a8-aef1-19b6fed0d1de

📥 Commits

Reviewing files that changed from the base of the PR and between ee28397 and 217318f.

📒 Files selected for processing (1)
  • .github/workflows/cloud-runtime-artifact.yml

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

kjgbot pushed a commit that referenced this pull request Sep 4, 2026
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
@kjgbot
kjgbot merged commit 4df6d94 into main Sep 4, 2026
1 of 2 checks passed
kjgbot added a commit that referenced this pull request Sep 4, 2026
#153 enabled the full SDK suite without giving CI a way to run it. One case in
live-kernel.test.ts executes the REAL Claude analyzer and fails by default when
it cannot -- deliberately, so that a reader never gets a green that proves
nothing about gate 2. A GitHub runner has no `claude` binary:

  LIVE_ANALYZER_UNAVAILABLE: "...analyze-story-claude-cli auth status" exited 1:
  cannot run "claude": spawnSync claude ENOENT

so the step could never pass. That is my defect from #153, not the test's.

RELAYFLOWS_ALLOW_ANALYZER_SKIP is the escape hatch the test itself documents,
for "environments that knowingly cannot reach a model and are not counting this
run as gate evidence". Setting it here is that statement made explicitly: this
workflow is not gate-2 acceptance evidence. Gate-2 evidence must come from a
machine that can reach a model, and the skipped case prints as much.

Every other case in the suite still runs and still gates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Co-authored-by: kjgbot <kjgbot@agentrelay.dev>
Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot pushed a commit that referenced this pull request Sep 4, 2026
…ucible

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot pushed a commit that referenced this pull request Sep 5, 2026
Split out of #165, which bundled this with a wholesale revert of the CI
workflow. The workflow half restored `ops/cargo.sh` for the kernel test
step (that wrapper redirects RUSTUP_HOME to an empty dir on a runner, so
rustup cannot choose a toolchain) and dropped the analyzer-skip env, i.e.
it reverted #153, #154 and #159 together. Only the sdk tooling is carried
here; `.github/` is byte-identical to main.

scripts/test.sh runs the same chain the inline `test` script did, in the
same order, with `set -eu` for the fail-fast the `&&` chain gave. The new
behaviour is the EXIT trap, which prunes `.map` and `.d.ts` from
sdk/dist afterward.

CI is unaffected either way: the workflow runs the expanded chain minus
test:prep, not `npm test`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot added a commit that referenced this pull request Sep 5, 2026
The sdk half of #165, with that PR's CI revert dropped. `.github/` is
byte-identical to main (verified: `git diff origin/main --stat -- .github/`
returns 0 lines); #165's workflow hunk would have reverted #153, #154 and #159
together, restoring `ops/cargo.sh` for the kernel step, which redirects
RUSTUP_HOME to an empty dir on a runner.

`scripts/test.sh` runs the same chain the inline `test` script did, in the same
order, with `set -eu` supplying the fail-fast the `&&` chain gave.

Pruning is source maps only. Deleting `.d.ts` would have left
`"types": "./dist/index.d.ts"` pointing at a file `npm test` had just removed.
Every current in-repo consumer imports `.js` (`ops/probes/**`,
`workflows/drive.yaml`, `workflows/drive-cloud.yaml`,
`testdata/backlog-picker.flow.yaml`), so nothing breaks today — but the next
TypeScript consumer would meet a failure caused by running the tests.

Evidence at this head:
- independent signoff: local 3-lens preswarm review, maintainability / history /
  structure all REVIEW_PASSED. The history lens caught a false scope claim in an
  earlier message ("only ops/probes consume the sdk"), which was corrected.
- CI: linux-x64-artifact success, packed-consumer pass, CodeRabbit pass. The
  `review` check fails for a reason independent of this change and common to
  every flows PR: the gate invokes `agent-relay` and no step installs it
  (exit 127).

Supersedes the sdk half of #165.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant