Skip to content

fix(cli): a running run is not a protocol error (#179) - #180

Merged
kjgbot merged 3 commits into
mainfrom
fix/179-running-not-a-protocol-error
Sep 5, 2026
Merged

fix(cli): a running run is not a protocol error (#179)#180
kjgbot merged 3 commits into
mainfrom
fix/179-running-not-a-protocol-error

Conversation

@kjgbot

@kjgbot kjgbot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Fixes the live-kernel > follows a live worker dispatch through flows run flake from #179, which failed CI three times tonight and forced a re-run each time.

Reproduced first

Built a worktree at main's tip to CI's own spec and ran the test in a loop: 1 failure in 12, then 1 in 4 on a second pass. Instrumenting the fall-through branch named the state exactly:

PROBE_D unclassifiable inspection={"status":"running","needsHuman":false}

The bug

classifyOutcome loops while the run is parked, asking inspectOutOfBandStep what to do next. That helper reports a step only when a non-deterministic step is needs_human, runnable or running.

When a worker has just completed the step the run parked on, there is a window where none of those hold while the snapshot's own status is still running — the daemon has not yet finished driving what follows. Every branch missed that window, so the loop broke with status === 'parked' and no parkedStep, and the tail reported:

FAILED [protocol_error] relayflowd could not complete the run request:
relayflowd returned status parked without a classifiable completion

The run was healthy and about to succeed. The CLI failed it for being observed mid-stride.

The fix

A run that reports running is progressing, so it is polled rather than abandoned — bounded at 40 × 50ms = 2s, far longer than the sub-second window observed and short enough that a genuinely stuck run still reports instead of hanging. The unchanged break below keeps it failing closed if it never resolves.

Evidence

result
before 1 fail / 12, then 1 fail / 4
after 0 fail / 40
full SDK suite 31 files passed, 649 tests passed, 3 skipped
typecheck clean

This also explains the cli-hn-monitor failure shape from #179's table: both are the CLI treating a transient, healthy state as terminal. Worth re-checking that one once this lands.

Note on the unprovable_effects warning

Both CI failures quoted WARNING [unprovable_effects] Step "greet" command "printf"... in their assertion message. That is a red herring — it appears in stderr on every run, passing or failing, and is simply the first line of the captured stderr the assertion prints. The real error was always the protocol_error beneath it.

Reproduction environment

Not obvious, and it cost me a tick to get right: surface must be installed and built before the sdk, with bun (bun install --frozen-lockfile --ignore-scripts && bun run build) — npm fails there because surface has no package-lock.json. Then npm ci --prefix sdk --ignore-scripts, a release relayflowd for RELAYFLOWD_BIN, and RELAYFLOWS_ALLOW_ANALYZER_SKIP=1.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

`flows run` intermittently failed a healthy run with:

  FAILED [protocol_error] relayflowd could not complete the run request:
  relayflowd returned status parked without a classifiable completion

Reproduced locally: 1 failure in 12 runs, and 1 in 4 on another pass.
Instrumenting the fall-through branch named the state exactly:

  PROBE_D unclassifiable inspection={"status":"running","needsHuman":false}

`classifyOutcome` loops while the run is parked, asking
`inspectOutOfBandStep` what to do. That helper reports a step only when a
non-deterministic step is `needs_human`, `runnable` or `running`. When a
worker has just completed the step the run parked on, there is a window
where none of those hold while the snapshot's own status is still
`running` -- the daemon has not yet finished driving what follows.

Every branch missed that window, so the loop broke with
`status === 'parked'` and no `parkedStep`, and the tail reported an
unclassifiable completion. The run was healthy and about to succeed.

A run that reports `running` is progressing, so it is now polled rather
than abandoned: bounded at 40 x 50ms = 2s, which is far longer than the
sub-second window observed and short enough that a genuinely stuck run
still reports instead of hanging. The unchanged `break` below keeps it
failing closed if it never resolves.

Evidence:
  * before: 1 fail / 12, then 1 fail / 4
  * after:  0 fail / 40
  * full SDK suite: 31 files passed, 649 tests passed, 3 skipped
  * typecheck clean

This is #179's `live-kernel > follows a live worker dispatch` flake, and
also explains the earlier `cli-hn-monitor` failure shape: both are the
CLI treating a transient, healthy state as terminal.

Reproduction environment, since it is not obvious: surface must be built
with `bun install --frozen-lockfile --ignore-scripts && bun run build`
BEFORE the sdk (npm there fails -- surface has no package-lock.json),
then `npm ci --prefix sdk --ignore-scripts`, a release `relayflowd` for
RELAYFLOWD_BIN, and RELAYFLOWS_ALLOW_ANALYZER_SKIP=1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Free

Run ID: 6abb343c-8db3-406b-9dec-56b0e8d2a167

📥 Commits

Reviewing files that changed from the base of the PR and between e0ecae3 and ae04d2a.

📒 Files selected for processing (2)
  • sdk/src/cli/run.ts
  • sdk/tests/classify-outcome.test.ts

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

The maintainability lens caught a regression I introduced: the poll used
a bare `setTimeout`, while every other wait in this file goes through the
cancel-aware `delay(ms, signal)`. A Ctrl-C or lifecycle abort during the
2-second window would have been ignored and the CLI would have kept
polling the daemon after being told to stop.

Now `throwIfCanceled(options.signal, ...)` then `delay(..., options.signal)`,
matching `waitForRunningStep` directly above it.

Also stated the contract the loop leans on -- `run.resume` is idempotent
on a run already progressing, returning current state rather than
re-dispatching -- because up to 41 resumes in 2 seconds is a load-bearing
assumption that was written nowhere.

Style: the constants used `///`, a Rust doc comment, in a file that uses
`//`.

Re-verified after the change: 20/20 on the previously-flaking test, and
the full SDK suite still 31 files / 649 tests passed, 3 skipped.

Still outstanding from that review, and NOT done here: the new branch has
no unit test. `classifyOutcome` is unexported with no test file, so
pinning it means making it injectable or exported first -- a design
change that deserves its own pass rather than a rushed one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot pushed a commit that referenced this pull request Sep 5, 2026
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
The maintainability lens wanted the new branch tested, and was right that
without one a refactor of `inspectOutOfBandStep` -- or a tightening of
the outer `while (current.status === 'parked')` guard -- would kill it
silently and regress #179.

`classifyOutcome` is now exported for tests. The branch only occurs in a
sub-second window against a live daemon, so pinning it needs a stubbed
client rather than a real run: the integration test that found the bug
reproduced it roughly 1 time in 12, which is not a gate.

Two cases, and they state the rule together:

  * a run reporting `running` with no identifiable step is waited out and
    then classified normally -- no `protocol_error`, exit 3, the parked
    step identified
  * a run that NEVER becomes classifiable still reports rather than
    polling forever, which is what keeping the original `break` buys

Mutation: disabling the branch (`if (false && ...)`) fails the first test
with the exact production string --

  expected [ 'protocol_error' ] to not include 'protocol_error'
  "relayflowd returned status parked without a classifiable completion"

-- while the bound test still passes. sha256 8124009e -> e80ee063 ->
restored 8124009e, no `if (false &&` left in the tree.

Full SDK suite at this head: 32 files passed, 651 tests passed, 3
skipped; typecheck clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot pushed a commit that referenced this pull request Sep 5, 2026
…lf is a distinct bug

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
@kjgbot
kjgbot merged commit cf82e91 into main Sep 5, 2026
3 of 5 checks passed
@kjgbot
kjgbot deleted the fix/179-running-not-a-protocol-error branch September 5, 2026 16:34
kjgbot pushed a commit that referenced this pull request Sep 5, 2026
…5 measurement

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtQSAcGDta5VH9xiZFT4sR

Session-Id: c228933d-4f94-4d83-9a9a-daf3c83b94f1
kjgbot added a commit that referenced this pull request Sep 5, 2026
Nothing verified main. `cloud-runtime-artifact.yml` triggered only on `pull_request` and `workflow_dispatch`, and `review-swarm.yml` only on `pull_request` — so every PR is checked at its own head and never as merged, and since we squash-merge onto a main that has moved since that CI ran, the composed result went unverified.

Not theoretical: twelve PRs merged on 2026-09-05 across the exactly-once claim path (#171, #182), resume adoption (#177, #186), the authored-flow executor (#184, #187) and the CLI run loop (#180) — each green on its own branch, the composed tree never run until dispatched by hand:

```
run 33987924703  workflow_dispatch  main  completed/success  ed917bf
```

Main is fine. But nobody knew that, and finding out required knowing to ask.

Without this, a bad compose surfaces as an unrelated PR going red — the most expensive way to find it, since the author debugs their own change first. Tonight already produced three failures on PRs that belonged to something else (#179, #185, the #174 chain), and each cost a tick to attribute.

Paths are deliberately not filtered on the push trigger: on a PR the question is "does this change affect the runtime", on main it is "is the tree good", and a docs-only merge can land on a tree someone else broke.

Evidence at the merged head d1cb32a:
- signoff: local 3-lens preswarm, maintainability / history / structure all REVIEW_PASSED
- CI: run 33988314276 success on d1cb32a, verified by headSha and event=pull_request
- YAML parsed and asserted: triggers are `pull_request`, `push`, `workflow_dispatch`; push is branch-scoped to main; the pull_request path filter is unchanged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant