Skip to content

feat(cli): 2.F — CI --json NDJSON machine-output contract (+ ADR-0049, 2.D-Done bookkeeping)#42

Merged
cemililik merged 5 commits into
mainfrom
development
Jun 22, 2026
Merged

feat(cli): 2.F — CI --json NDJSON machine-output contract (+ ADR-0049, 2.D-Done bookkeeping)#42
cemililik merged 5 commits into
mainfrom
development

Conversation

@cemililik

@cemililik cemililik commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

What & why

Phase-2 workstream 2.F — the CI / non-interactive --json machine-output contract for relavium run, the next node on the M3 spine after 2.D. Plus the post-merge 2.D-Done roadmap bookkeeping (PR #41 merged).

Under relavium run --json, the CLI now emits a stable contract a CI job can pipe and assert on (ADR-0049):

  • stdout is a pure RunEvent NDJSON stream — one event serialized verbatim per line, in sequenceNumber order, no wrapper/header/version line (the envelope is already canonical per sse-event-schema.md).
  • The terminal run:completed event is the final result line (it already carries outputs + totals) — no separate summary line.
  • All diagnostics go to stderr — incl. the {type:'error'} CLI-fault envelope, which moves from stdout (2.D) to stderr; stdout stays empty on a pre-run fault and the exit code is the signal.
  • Masked secrets pass through verbatim ({secret:true,ref}), never unwrapped.
  • NDJSON is --json-gated (no-TTY/CI disable the TUI but don't force NDJSON); exit codes 0/1/2/3 flow from the terminal event.

The contract is scoped to a run (ADR-0049 §6): --help/--version/a bare invocation are exit-0 human-on-stdout meta-ops.

Commits

  • docs(roadmap) — mark 2.D Done (PR feat(cli): 2.D — relavium run wired to the @relavium/core engine #41) across the status surfaces; advance the spine pointer to 2.F.
  • docs(decisions)ADR-0049 (the machine-output contract) + realign the phase-2-cli.md §2.F task spec to it.
  • feat(cli) — the 2.F implementation (the fault-envelope stdout→stderr move + renderer formalization).
  • fix(cli) — review follow-up (a stale comment + the --json run:cancelled test).

Process (ADR-gated, with adversarial verification)

This went through the full discipline: a 6-investigator doc review → 3 design decisions surfaced for maintainer approval → ADR-0049 written, externally reviewed (3 reviews), lockstep-fixed, and approved before any code → implementation → an adversarial verify pass that found 8 real issues (all fixed, incl. a $?-after-! cmd tutorial bug and a false-pass in the schema test fixed with round-trip equality) → 3 more external reviews (2× ship-it + 1 low) → this follow-up.

Notable design calls (each rejected its alternatives in the ADR): the fault envelope → stderr (aligns with error-handling.md: run failures are run:failed, never type:error); run:completed reused as the result line (no synthetic summary line, which would break the every-line-is-a-RunEvent acceptance); --json-gated (not auto-on under CI, which surprises a human piping output).

Not touched

No @relavium/shared change (no new event type), no new exit code, no new dependency, no packages/core change, no @relavium/llm seam involvement. The contract reuses the already-canonical RunEvent envelope and realizes ADR-0036 (terminal-event invariant) + ADR-0047 (renderer-over-one-bus).

Test coverage

The full --json terminal matrix asserts stdout-purity (every line a schema-valid RunEvent, via round-trip equality that catches a stray field the non-strict schema would otherwise strip) with faults isolated to stderr: completed→0, failed→1, cancelled→1, paused→3. Plus a verbatim masked-secret round-trip and the meta-op carve-out pinned.

Toolchain

pnpm run ci green end-to-end: lint, typecheck (+ tools), 115 CLI tests (full monorepo green), build, format:check, the @relavium/llm seam fence, and engine-deps. Leakwatch clean on the new code (one pre-existing 2.B test false-positive, untouched).

Docs

commands.md gains a "--json machine-output contract" subsection + a refined Output-modes table; the run-a-workflow-in-ci tutorial is expanded (NDJSON consumption, the correct RELAVIUM_<PROVIDER>_API_KEY env var, a corrected exit-code-gating shell snippet). The exit-3 note is corrected to run:paused (the aggregate suspension).

🤖 Generated with Claude Code

Summary by Sourcery

Formalize the CLI relavium run --json machine-output contract so CI consumers receive a stable RunEvent NDJSON stream on stdout with diagnostics confined to stderr, and document this behavior via ADR-0049 and updated CLI/roadmap docs.

New Features:

  • Define a stable --json NDJSON output mode where relavium run streams canonical RunEvent objects in sequenceNumber order on stdout and uses the terminal event as the result line.
  • Clarify that --json is an explicit opt-in for NDJSON output while non-TTY or CI environments without --json use the plain human renderer.

Bug Fixes:

  • Route structured CLI error envelopes under --json to stderr instead of stdout so stdout remains reserved for RunEvent streams.
  • Preserve the run:cancelled terminal event as part of the NDJSON stream with the correct exit code mapping and no stderr fault pollution.
  • Correct CLI and tutorial documentation around exit code 3 semantics and shell examples for checking exit codes and handling error output.
  • Fix the documented provider API key environment variable name for CI runs to use the RELAVIUM_<PROVIDER>_API_KEY convention.

Enhancements:

  • Strengthen JSON renderer and run-command tests to assert every stdout line is a schema-valid, round-trippable RunEvent in monotonic sequenceNumber order, including failure and cancellation cases.
  • Refine the renderer abstraction and error-rendering behavior so all output modes share a single event bus while diagnostics are cleanly separated from the run stream.
  • Record ADR-0049 to capture the CLI machine-output contract design and align the phase-2 CLI roadmap tasks with the finalized behavior.
  • Update roadmap and high-level project docs to mark 2.D as done, describe 2.F --json CI mode, and clarify the M3 milestone path.

Documentation:

  • Extend the CLI reference to describe the --json machine-output contract, output modes, and exit-code behavior including stderr diagnostics for faults.
  • Expand the CI tutorial to show consuming NDJSON output, using the correct API key environment variables, and handling exit codes and stderr in shell scripts.

Tests:

  • Add and update tests for relavium run --json and the JSON renderer to cover successful, failed, cancelled, and meta-op cases while asserting stdout purity and stderr-only fault envelopes.

Summary by CodeRabbit

  • Bug Fixes
    • Corrected --json fault output routing: diagnostics and pre-run faults now reliably go to stderr, while stdout is a pure NDJSON RunEvent stream.
  • Documentation
    • Documented the finalized --json machine-output contract (stdout NDJSON, stderr diagnostics, masked secrets, terminal run:completed as final result) and updated exit-code semantics.
    • Updated the CI “run in CI” tutorial, CLI reference docs, roadmap status notes, and added ADR 0049 to the decision index.
  • Tests
    • Strengthened --json NDJSON tests with strict schema validation, event ordering checks, correct stdout/stderr separation, and improved cancellation/error scenarios.

cemililik and others added 4 commits June 22, 2026 17:09
PR #41 merged `relavium run` wired to @relavium/core (2.D — the M3 keystone) plus
the `defaultProviders()` seam registry and the exported `MAX_SOURCE_CHARS` cap.
Flip the status surfaces now that it's on main (roadmap-done-after-merge):

- phase-2-cli.md: 2.D heading → ✅ Done (PR #41); status line advances the spine
  pointer to 2.F and notes the 2.E (ink TUI) + 2.H (durable history) run-surface
  feeders now open.
- current.md: record 2.D Done (PR #41); "Next on the spine" → 2.F then 2.K (M3).
- CLAUDE.md / README.md: status paragraphs note `relavium run` has landed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Record the `--json` machine-output contract for the CLI (workstream 2.F): under
`--json`, stdout is a pure `RunEvent` NDJSON stream (one event per line, in
sequenceNumber order, no wrapper); the terminal `run:completed` IS the final result
line (no synthetic summary); the `{type:'error'}` CLI-fault envelope moves to stderr
(it was on stdout in 2.D); NDJSON is `--json`-gated (no-TTY/CI disable the TUI but do
not force NDJSON); and the exit code is derived from the terminal event (0/1/3, with a
pre-run fault = 2). No new envelope, no @relavium/shared change, no new exit code —
the contract reuses the already-canonical RunEvent envelope and realizes ADR-0036 +
ADR-0047.

Also realign the phase-2-cli.md §2.F task spec to this decision (it previously said
"auto-engage under no-TTY / CI=true" and named `human_gate:paused` for exit 3 — both
now corrected to `--json`-gated NDJSON and the aggregate `run:paused`), so the
implementer's spec and the ADR agree. The commands.md envelope subsection + the CI
tutorial expansion are 2.F implementation deliverables.

Refs: ADR-0036, ADR-0047

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Realize ADR-0049 for `relavium run --json`: stdout is a stable, pure NDJSON stream
of RunEvents (one event verbatim per line, in sequenceNumber order, no wrapper), the
terminal run:completed IS the result line (outputs + totals), and ALL diagnostics —
including the structured `{type:'error'}` CLI-fault envelope — go to stderr.

- render-error.ts: the CLI-fault envelope moves stdout → stderr under `--json` (it was
  on stdout in 2.D); stdout stays empty on a pre-run fault, exit code is the signal.
- renderer.ts: formalize createJsonRenderer as the 2.F NDJSON renderer (verbatim
  RunEvent/line, masked-secret passthrough, stdout-only); fix the docstring doc links.
- The contract is scoped to a workflow run (ADR-0049 §6): `--help`/`--version`/a bare
  invocation stay exit-0 human-on-stdout meta-ops, `--json` notwithstanding.

Tests: the 3 boundary assertions rewritten (fault envelope on stderr, stdout empty);
a `--json` run:failed test (the failure stays a RunEvent on stdout, no stderr fault);
per-line round-trip EQUALITY (catches a stray field the non-strict schema would strip);
a verbatim masked-secret round-trip; the meta-op carve-out pinned (--json --help/
--version/bare → stdout, exit 0); renderer stdout-purity.

Docs: commands.md gains a "--json machine-output contract" subsection + a refined
Output-modes table (NDJSON is --json-gated) + the status note marked 2.F-done + the
exit-3 note corrected to run:paused; the run-a-workflow-in-ci tutorial is expanded
(NDJSON consumption, the correct RELAVIUM_<PROVIDER>_API_KEY env var, a corrected
exit-code-gating shell snippet). No new envelope, no @relavium/shared change, no new
exit code.

Refs: ADR-0049, ADR-0036, ADR-0047

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three reviews of 2.F (PASS / ship-it) surfaced one real finding and one matrix gap:

- program.ts: the configureOutput.writeErr comment still said the --json envelope is
  re-rendered "on stdout"; after 2.F it goes to stderr (ADR-0049). Corrected to match
  render-error.ts and the run.ts comment.
- Complete the --json terminal matrix: add an integration test that a SIGINT-cancelled
  run under --json keeps stdout a pure RunEvent NDJSON stream ending in run:cancelled
  with stderr empty (the sibling of the run:completed/run:failed/run:paused cases). The
  SIGINT stalling-engine setup is extracted to a shared makeStallingCancelEngine() helper,
  reused by the existing SIGINT test (no behavior change there).

The media-only-park `run:paused`→exit-3 observation the review raised is already tracked
in deferred-tasks (the ~2.S item) and is unreachable in 2.F; no action.

Refs: ADR-0049

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Reviewer's Guide

Implements the finalized 2.F CLI machine-output contract for relavium run --json by formalizing the NDJSON renderer semantics (stdout-only RunEvent stream, stderr diagnostics), tightening tests around RunEvent schema and exit codes, and updating roadmap/ADR/docs to describe the new behavior and 2.D completion.

Sequence diagram for relavium run --json NDJSON stream and stderr diagnostics

sequenceDiagram
  actor CI as CI_job
  participant CLI as relavium_run
  participant Engine as WorkflowEngine
  participant Renderer as JsonRunRenderer
  participant Out as stdout
  participant Err as stderr

  CI->>CLI: relavium run wf.relavium.yaml --json
  CLI->>Engine: start(workflow, inputs)
  Engine-->>CLI: RunHandle.events (RunEvent stream)
  loop For_each_RunEvent
    CLI->>Renderer: onEvent(RunEvent)
    Renderer->>Out: JSON_line(RunEvent)\n
  end
  note over Out: Pure NDJSON RunEvent stream

  alt Pre_run_CLI_fault (invalid_invocation)
    CLI->>Err: JSON { type: error, code, message }\n
    CLI-->>CI: exit 2
    Out-->>CI: (empty)
  else Terminal_run_completed
    Engine-->>CLI: run:completed (outputs + totals)
    CLI-->>CI: exit 0
  else Terminal_run_failed_or_cancelled
    Engine-->>CLI: run:failed / run:cancelled
    CLI-->>CI: exit 1
  else Non_interactive_run_paused
    Engine-->>CLI: run:paused
    CLI-->>CI: exit 3
  end
Loading

File-Level Changes

Change Details Files
Finalize --json NDJSON behavior for relavium run, ensuring stdout is a pure RunEvent stream and CLI faults are emitted as structured stderr diagnostics.
  • Adjust run top-level error handling to emit {type:'error', code, message} envelopes to stderr instead of stdout when --json is set, leaving stdout empty on CLI faults.
  • Refine commander error handling so parse errors are re-rendered as JSON fault envelopes only under --json, written to stderr while preserving human stderr output for non-JSON runs.
  • Clarify program construction to suppress commander human stderr only when the caller will emit the JSON fault envelope, avoiding duplicate diagnostics.
apps/cli/src/process/render-error.ts
apps/cli/src/run.ts
apps/cli/src/program.ts
Strengthen relavium run and renderer tests to assert the NDJSON machine contract: every stdout line is exactly a schema-valid RunEvent in sequenceNumber order, terminal events drive exit codes, and stderr is used for diagnostics.
  • Extend IO test helpers to capture both stdout and stderr so tests can assert output purity and diagnostic routing.
  • Add/expand tests for runCommand under --json to validate RunEvent round-trip equality against RunEventSchema, sequenceNumber monotonicity, and correct terminal event types for completed, failed, and cancelled runs.
  • Introduce a reusable stalling engine helper for SIGINT/cancel tests, and add a --json cancellation test that asserts a run:cancelled terminal event on stdout with no stderr faults.
  • Add tests that --help, --version, and bare invocations remain human-on-stdout exit-0 meta-operations even with --json.
  • Update CLI fault tests so all structured error envelopes are expected on stderr with stdout empty under --json.
  • Enhance JSON renderer tests to verify one line per event, schema round-trip equality, sequence ordering, terminal run:completed as the result line, and verbatim serialization of masked secrets.
apps/cli/src/commands/run.test.ts
apps/cli/src/render/renderer.test.ts
apps/cli/src/run.test.ts
Document and ADR-encode the --json machine-output contract, realigning roadmap and reference docs to the final behavior and marking 2.D as done.
  • Add ADR-0049 describing the --json NDJSON machine-output contract, including stdout RunEvent stream semantics, stderr diagnostics, exit-code mapping, and scoping to workflow runs.
  • Update the CLI commands reference to define the --json machine-output contract section, refine output mode table (interactive/plain/NDJSON), and document exit-code behavior and stderr fault envelopes under --json.
  • Expand the CI tutorial to show consumption of the NDJSON stream with jq, correct API key environment variable naming, correct exit-3 run:paused description, and a fixed shell snippet that handles exit codes and separates stdout/stderr.
  • Mark 2.D as Done in the phase-2 CLI roadmap, adjust the 2.F task spec to the --json-gated NDJSON and terminal run:completed semantics, and update the current roadmap, README, and CLAUDE docs to reflect 2.D completion and 2.F as the active workstream.
  • Register ADR-0049 in the ADR index.
docs/decisions/0049-cli-machine-output-contract.md
docs/decisions/README.md
docs/reference/cli/commands.md
docs/tutorials/cli/run-a-workflow-in-ci.md
docs/roadmap/phases/phase-2-cli.md
docs/roadmap/current.md
README.md
CLAUDE.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b1c096f8-b662-413f-97c9-8448b3facb7d

📥 Commits

Reviewing files that changed from the base of the PR and between de36423 and c33f59e.

📒 Files selected for processing (4)
  • apps/cli/src/commands/run.test.ts
  • apps/cli/src/process/render-error.ts
  • apps/cli/src/render/renderer.test.ts
  • apps/cli/src/run.test.ts
🚧 Files skipped from review as they are similar to previous changes (4)
  • apps/cli/src/process/render-error.ts
  • apps/cli/src/render/renderer.test.ts
  • apps/cli/src/run.test.ts
  • apps/cli/src/commands/run.test.ts

📝 Walkthrough

Walkthrough

Formalizes the relavium run --json machine-output contract by routing CLI JSON fault envelopes from stdout to stderr, adding ADR-0049 to document the NDJSON stream contract, updating all affected tests to assert the stderr/stdout separation, and aligning reference docs, roadmap, and tutorial accordingly.

Changes

CLI --json machine-output contract

Layer / File(s) Summary
stdout→stderr fault routing in renderError
apps/cli/src/process/render-error.ts, apps/cli/src/program.ts, apps/cli/src/render/renderer.ts, apps/cli/src/run.ts
renderError now emits the { type:'error', code, message } JSON envelope via io.writeErr instead of io.writeOut. Comments in program.ts, renderer.ts, and run.ts are updated to describe stdout-only NDJSON behavior and the stderr routing for pre-run faults.
ADR-0049: CLI machine-output contract
docs/decisions/0049-cli-machine-output-contract.md, docs/decisions/README.md
New ADR defines stdout as a pure RunEvent NDJSON stream (one per line, sequenceNumber-ordered, run:completed as final result line), stderr as the destination for all diagnostics including structured pre-run fault envelopes, explicit --json opt-in semantics, exit-code derivation, and rejected alternatives.
run.ts / run.test.ts: stderr envelope assertions
apps/cli/src/run.test.ts
Adds CLI_VERSION import, introduces a test confirming --help/--version/bare invocation stay as human text on stdout regardless of --json, and reworks all structured JSON error envelope assertions to parse from err() with stdout empty, covering create --json failure, --json with bad flags, and commander parse errors.
renderer.test.ts: strict NDJSON contract tests
apps/cli/src/render/renderer.test.ts
captureIo() gains separate out() and err() accessors. JSON renderer tests are replaced with a sequentialRun() fixture-based approach: assertions validate RunEventSchema round-trip per line, monotonic sequenceNumber ordering, run:completed as sole terminal line, and verbatim preservation of engine-masked secret shapes.
commands/run.test.ts: NDJSON + SIGINT expansion
apps/cli/src/commands/run.test.ts
Adds RunEventSchema import, refactors captureIo to expose err(), introduces makeStallingCancelEngine helper with an AbortSignal-driven slow node, strengthens happy/fail NDJSON assertions with monotonic sequenceNumber and terminal event type checks, and adds a SIGINT cancellation NDJSON test asserting run:cancelled as the final event.
Docs, roadmap, and status updates
docs/reference/cli/commands.md, docs/roadmap/phases/phase-2-cli.md, docs/roadmap/current.md, docs/tutorials/cli/run-a-workflow-in-ci.md, CLAUDE.md, README.md
CLI reference output-mode table and exit-code docs are expanded. Phase-2 roadmap marks 2.D done and rewrites the 2.F NDJSON/CI contract. Current roadmap updates the "Next on the spine" ordering. CI tutorial steps and "What just happened" section are finalized. Status paragraphs in CLAUDE.md and README.md are updated.

Sequence Diagram(s)

sequenceDiagram
  participant Consumer as CI Consumer
  participant CLI as relavium run --json
  participant Engine as `@relavium/core` Engine
  participant Stdout as stdout (NDJSON)
  participant Stderr as stderr (diagnostics)

  Consumer->>CLI: relavium run workflow.yaml --json
  CLI->>Engine: execute DAG
  Engine-->>CLI: RunEvent (run:started, sequenceNumber=1)
  CLI->>Stdout: {"type":"run:started","sequenceNumber":1,...}
  Engine-->>CLI: RunEvent (cost:updated, sequenceNumber=N)
  CLI->>Stdout: {"type":"cost:updated","sequenceNumber":N,...}
  Engine-->>CLI: RunEvent (run:completed, sequenceNumber=M)
  CLI->>Stdout: {"type":"run:completed","sequenceNumber":M,...}
  CLI-->>Consumer: exit 0

  alt Pre-run CLI fault (e.g. missing key)
    CLI->>Stderr: {"type":"error","code":"missing_key","message":"..."}
    Note right of Stdout: stdout stays empty
    CLI-->>Consumer: exit 2
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • HodeTech/Relavium#41: Wired relavium run to the engine, establishing the NDJSON renderer and terminal-event behavior that this PR's commands/run.test.ts and renderer.test.ts expansions now assert against.

Poem

🐇 Hop hop, the streams are clear now!
Faults scurry off to stderr's den,
While stdout sings pure NDJSON zen.
Each RunEvent lands in order true,
sequenceNumber climbs right on through.
The rabbit types --json with delight—
Clean contracts make the pipelines bright! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 54.55% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly describes the main implementation: 2.F CI machine-output contract with NDJSON, ADR-0049, and 2.D completion bookkeeping. It accurately reflects all major components of this substantial changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch development

Comment @coderabbitai help to get the list of available commands and usage tips.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The captureIo helper is now duplicated (and growing) in multiple test files; consider extracting a shared test utility (e.g. test/helpers/io.ts) to keep the IO capture shape consistent and avoid diverging behavior as you add more cases.
  • In the NDJSON tests you call RunEventSchema.parse twice per line (once for the equality assertion and once for the typed return); you could parse once into a local variable and reuse it to keep the tests a bit clearer and avoid redundant work.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The `captureIo` helper is now duplicated (and growing) in multiple test files; consider extracting a shared test utility (e.g. `test/helpers/io.ts`) to keep the IO capture shape consistent and avoid diverging behavior as you add more cases.
- In the NDJSON tests you call `RunEventSchema.parse` twice per line (once for the equality assertion and once for the typed return); you could parse once into a local variable and reuse it to keep the tests a bit clearer and avoid redundant work.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements the CLI machine-output contract (ADR-0049) for Workstream 2.F, ensuring that under --json, stdout contains only the pure RunEvent NDJSON stream in sequence order, while all diagnostics and CLI-fault envelopes are routed to stderr. The changes include updates to the JSON renderer, error rendering logic, CLI commands, tests, and documentation. Feedback on the changes highlights an issue where enabling both --json and --verbose prints raw stack traces to stderr as plain text, which violates the JSON contract and could break parsers; nesting the stack trace inside the JSON envelope is recommended.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines 20 to 23
if (opts.json) {
io.writeOut(
io.writeErr(
JSON.stringify({ type: 'error', code: userFacing.code, message: userFacing.message }) + '\n',
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When --json and --verbose are both enabled, the raw stack trace is appended to stderr as plain text (lines 27-29), which violates the pure JSON contract of the stream and will break JSON parsers attempting to read stderr under --json.

To preserve the JSON contract, the stack trace should be nested inside the JSON envelope when opts.json is active, rather than printed as raw text.

For example:

export function renderError(
  value: unknown,
  opts: { readonly json: boolean; readonly verbose: boolean },
  io: CliIo,
): void {
  const userFacing = toUserFacing(value);
  if (opts.json) {
    const payload: Record<string, unknown> = {
      type: 'error',
      code: userFacing.code,
      message: userFacing.message,
    };
    if (opts.verbose && value instanceof Error && value.stack !== undefined) {
      payload.stack = value.stack;
    }
    io.writeErr(JSON.stringify(payload) + '\n');
  } else {
    io.writeErr(`relavium: ${userFacing.message}\n`);
    if (opts.verbose && value instanceof Error && value.stack !== undefined) {
      io.writeErr(value.stack + '\n');
    }
  }
}

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
apps/cli/src/run.test.ts (1)

61-73: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Strengthen the meta-op contract test and avoid version pin brittleness.

Line 68 hard-codes 0.0.0, and this block doesn’t assert stderr is empty for help/version/bare. A semver-shaped assertion + err() === '' checks would keep the test stable while fully enforcing the stdout/stderr contract.

Proposed patch
 it('keeps --help / --version / bare as human-on-stdout exit-0 meta-ops even under --json', async () => {
-  const help = makeIo();
-  expect(await run(argv('--json', '--help'), help.io)).toBe(0);
-  expect(help.out()).toContain('Usage: relavium');
+  const help = makeIo();
+  expect(await run(argv('--json', '--help'), help.io)).toBe(0);
+  expect(help.out()).toContain('Usage: relavium');
+  expect(help.err()).toBe('');

   const version = makeIo();
   expect(await run(argv('--json', '--version'), version.io)).toBe(0);
-  expect(version.out()).toContain('0.0.0');
+  expect(version.out()).toMatch(/\b\d+\.\d+\.\d+\b/);
+  expect(version.err()).toBe('');

   const bare = makeIo();
   expect(await run(argv('--json'), bare.io)).toBe(0);
   expect(bare.out()).toContain('Usage: relavium');
+  expect(bare.err()).toBe('');
 });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/cli/src/run.test.ts` around lines 61 - 73, The test is hard-coding the
version check to `0.0.0` which creates brittleness when the version changes, and
it is not verifying that stderr is empty for the help, version, and bare
meta-operations. Replace the hard-coded version assertion with a semver pattern
match (e.g., a regex that matches semantic version format like X.Y.Z) to keep
the test stable across version changes. Additionally, add assertions using err()
on each of the three test cases (help, version, and bare) to ensure that stderr
is empty for all three, fully enforcing the stdout/stderr contract that the test
name promises. Apply these changes to the help, version, and bare test blocks
within the it block.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@apps/cli/src/run.test.ts`:
- Around line 61-73: The test is hard-coding the version check to `0.0.0` which
creates brittleness when the version changes, and it is not verifying that
stderr is empty for the help, version, and bare meta-operations. Replace the
hard-coded version assertion with a semver pattern match (e.g., a regex that
matches semantic version format like X.Y.Z) to keep the test stable across
version changes. Additionally, add assertions using err() on each of the three
test cases (help, version, and bare) to ensure that stderr is empty for all
three, fully enforcing the stdout/stderr contract that the test name promises.
Apply these changes to the help, version, and bare test blocks within the it
block.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7140d198-fa8c-4b32-a291-b6a1f22fede3

📥 Commits

Reviewing files that changed from the base of the PR and between 0c98abe and de36423.

📒 Files selected for processing (15)
  • CLAUDE.md
  • README.md
  • apps/cli/src/commands/run.test.ts
  • apps/cli/src/process/render-error.ts
  • apps/cli/src/program.ts
  • apps/cli/src/render/renderer.test.ts
  • apps/cli/src/render/renderer.ts
  • apps/cli/src/run.test.ts
  • apps/cli/src/run.ts
  • docs/decisions/0049-cli-machine-output-contract.md
  • docs/decisions/README.md
  • docs/reference/cli/commands.md
  • docs/roadmap/current.md
  • docs/roadmap/phases/phase-2-cli.md
  • docs/tutorials/cli/run-a-workflow-in-ci.md

- render-error.ts: under `--json`, the `--verbose` raw stack no longer goes to stderr
  (it would mix non-JSON text into the structured diagnostic; error-handling.md — a
  stack is not machine output). The stack stays a human-mode affordance; under `--json`
  the `{type:'error'}` envelope is the whole stderr diagnostic. Pinned by a new test
  asserting stderr is a single JSON line under `--json --verbose`.
- run.test.ts: the meta-op test now asserts `err() === ''` for --help/--version/bare
  (fully enforcing the stdout-only contract its name promises) and checks the version
  against the CLI_VERSION constant instead of a hard-coded '0.0.0' (tracks the source of
  truth; a regex would re-introduce a Sonar S5852 hotspot the package deliberately avoids).
- commands/run.test.ts + renderer.test.ts: parse each NDJSON line once and reuse it
  (was RunEventSchema.parse twice per line); use toHaveLength for the SIGINT
  listener-count assertions (Sonar).

Skipped: extracting a shared captureIo test helper — a maintainability refactor across
several files, out of scope for this minimal review pass; the copies are small and stable.

Refs: PR #42 review

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cemililik
cemililik merged commit ca13ebe into main Jun 22, 2026
9 checks passed
@sonarqubecloud

Copy link
Copy Markdown

cemililik added a commit that referenced this pull request Jun 22, 2026
PR #42 merged the CLI `--json` machine-output contract (2.F) + ADR-0049 + the 2.D-Done
bookkeeping. Flip the status surfaces now that it's on main (roadmap-done-after-merge):

- phase-2-cli.md: 2.F heading → ✅ Done (PR #42); status line records it (behind
  ADR-0049) and advances the spine pointer to 2.K (the regression harness that reaches M3).
- current.md: record 2.F Done (PR #42); "Next on the spine" → 2.K.
- CLAUDE.md / README.md: status paragraphs note the `--json` CI output contract has landed.

The 2.E (ink TUI) and 2.H (durable run history) run-surface feeders remain open.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cemililik added a commit that referenced this pull request Jun 22, 2026
Adopt the CLI as the engine's canonical end-to-end regression gate (completes M3 with
2.D + 2.F). An in-process vitest e2e suite (apps/cli/src/harness/) runs committed example
workflows through the REAL `relavium run … --json` path (the default engine over the real
createCliHost) and asserts on the NDJSON RunEvent stream + exit code — deterministic and
offline on every PR, inside the existing required `test` CI gate.

- Fixtures (apps/cli/src/harness/fixtures/, non-agent → no LLM/key/network): sequential,
  fan-out (parallel + object_merge), conditional (condition + node:skipped, both arms via
  n=3→lo and n=15→hi), human-gate (run:paused → exit 3), failure (a throwing transform →
  run:failed → exit 1). Together they cover every authorable non-agent node type and all
  three run exit codes (0/1/3).
- Assertion is keyed by a `type:nodeId` SIGNATURE (not just event type), so a wrong-branch /
  wrong-node regression is caught — exact ordered sequence for deterministic fixtures, the
  signature multiset + anchors for the `parallel` fan-out (whose branches interleave by async
  timing). Plus gap-free sequenceNumber (ADR-0036), each line round-trips to exactly a
  canonical RunEvent, and stderr is empty (the stdout-pure --json contract, ADR-0049).
- Shared captureIo extracted to apps/cli/src/test-support.ts; the three other test copies
  (run.test, commands/run.test, render/renderer.test) now import it.

No new ADR, no new dependency, no @relavium/shared change — 2.K composes ADR-0036/0047/0049
+ the @relavium/llm conformance/replay substrate + the 1.U m2-e2e-harness seed.

Docs: a new docs/reference/cli/regression-harness.md (the canonical fixture + scenario
format) + a testing.md section; the §2.K work note records the maintainer-approved scope —
the gate-RESUME scenario is deferred to 2.G+2.H and agent-replay + the nightly live lane to
later (both need the replay-provider wiring), with the run-to-gate→exit-3 half shipping now.

Also marks 2.F Done (PR #42) across the status surfaces (post-merge bookkeeping, riding along).

Refs: ADR-0036, ADR-0047, ADR-0049

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant