Skip to content

M2 — 1.U end-to-end harness + engine-hardening pass + cross-process cost persistence#27

Merged
cemililik merged 6 commits into
mainfrom
development
Jun 16, 2026
Merged

M2 — 1.U end-to-end harness + engine-hardening pass + cross-process cost persistence#27
cemililik merged 6 commits into
mainfrom
development

Conversation

@cemililik

@cemililik cemililik commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

The post-PR-#26 batch — it lands the M2 critical-path milestone (the engine proven end-to-end) and a hardening pass. Four commits, all on the engine packages; full gate green, pnpm coverage EXIT 0 (engine ≥90% line+branch floor holds), Leakwatch clean.

What's in it

1. 8087d85 — docs: mark 1.V + 1.AC Done (PR #26), complete 1.m4
Reconciled every status home after PR #26 merged (1.V AgentSession + 1.AC budget governor). 1.AC was the last 1.m4 component → 1.m4 complete; 1.V opens the Lane-C sub-spine (1.m5). M2 was not promoted (it is gated by 1.U — see next).

2. 933a86f — test: the 1.U end-to-end harness suite (the M2 proof)
The proof the engine works end-to-end before any surface exists — composes 1.K/1.N/1.O/1.P/1.Q/1.R/1.S/1.T behind the @relavium/llm seam on the in-memory ExecutionHost. Zero platform imports, no live network/keys, deterministic; no new engine code or ADR (already-exported symbols only). A scenario suite:

  • happy-path — literal 3-node input → agent(+tool) → output: streaming, per-attempt cost, gap-free sequenceNumber validating against the canonical RunEventSchema.
  • flagshipinput → agent → human_gate → output: in ONE run across a process boundary — forced provider error → node retryfailover to the second chain entry → per-attempt cost; pause at the gate (durable checkpoint); a fresh engine resumes via resumeFromCheckpoint, completes output, reproduces the final output gap-free.
  • determinism — a re-run yields an identical event signature + output.

Design note (maintainer-approved): the resume demonstration uses a human gate as the durable mid-run checkpoint, because the Phase-1 engine resumes only from a gate/budget pause — a gate-less interrupted run is reconciled to run:failed (ADR-0036; verified by a resume-model review + an adversarial refutation). The literal 3-node topology is preserved by the happy-path member; recorded as the §1.U Harness shape clarification.

3. 15a5b0e — test: engine-hardening pass (5 verified deferred items)
No production behavior change (tests + test-support + a CI job + an AgentTurnLimits doc-comment):

  • Combined tool-loop DoS bound — the bounds are NOT multiplicative (maxToolTurns+1 is the egress ceiling; maxToolCorrections is a monotonic sub-budget). Documented + an interleaving test.
  • DeepSeek reasoning replay — confirmed correct (captured inbound, dropped on egress; reasoning_content is output-only, replay 400s); locked with a test.
  • Conformance tool-loop + cache-hit scenarios across all 4 providers (new replayFetchSequence; cache-read folds into the canonical Usage).
  • Advisory repo-root pnpm coverage CI job (enforces the testing.md ≥90% floor; non-required while the core-branch margin is thin).

4. eeb3f32 — fix(core): restore cumulative cost on cross-process resume
Closes the cost-event-persistence gap: the durable node:completed now carries an optional cumulativeCostMicrocents (a snapshot of the run-wide running total), folded on resume with a monotonic Math.max that reconciles with the budget:paused restore. A run paused at any gate now resumes with the right spend (was ~0 at a plain human gate). Additive, forward-compatible field — no ADR.

After merge

Per the roadmap-done-after-merge rule, 1.U → ✅ Done and M2 → reached will be marked in a follow-up once this merges (phase-1 milestone table + dependency matrix + README global spine + CLAUDE.md/current.md).

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added an end-to-end Node harness covering retry/failover, durable checkpoint/resume, and human-gate pause/resume.
    • Added run-wide cumulative cost persistence to checkpoint/resume boundaries.
    • Expanded LLM conformance fixtures to include multi-turn tool loops and provider replay support for sequences.
  • Bug Fixes
    • Ensured cumulative cost restoration remains monotonic and order-independent on resume.
    • Prevented DeepSeek internal reasoning content from being sent on subsequent requests.
    • Strengthened tool-loop correction-budget enforcement and cache-read token observability checks.
  • CI
    • Added a non-blocking, advisory coverage job/threshold lane.
  • Documentation
    • Updated roadmap and milestone status across project docs.

cemililik and others added 4 commits June 16, 2026 07:49
…to 1.U/M2

PR #26 merged the pre-egress budget governor (1.AC, ADR-0028) and the AgentSession
agent-first entry point (1.V, ADR-0024) together. Reconcile every status home to the
post-merge state.

- 1.AC was the last 1.m4 component → **1.m4 is complete** (the full engine stack:
  node handlers, gate, checkpoint/resume, retry, tools, sandbox, budget governor).
- 1.V opens the Lane-C agent-first sub-spine (1.m5, still open: 1.W/1.X/1.Y/1.Z/1.AA).
- M2 is NOT promoted: it is gated solely by the end-to-end Node harness (1.U), which is
  still open. 1.AC completing 1.m4 only unblocks 1.U — now the next critical-path task.

Files:
- phase-1-engine-and-llm.md: banner; 1.AC section header + Acceptance (Met); 1.V bullet
  Done marker + scope-note past-tense; 1.m4 milestone row ✅; dep-matrix 1.AC/1.U/1.V rows.
- current.md: date bump; M2-next sentence (1.m4 done, only 1.U remains); next-steps tail
  (1.AC+1.V Done, 1.U next + Lane C 1.W/1.X); PR-#12 follow-up note.
- CLAUDE.md / AGENTS.md / README.md: Status paragraphs brought current through PR #26.
- deferred-tasks.md: date; 1.V merge note; close the per-attempt pre-egress gate item
  (landed with 1.AC); re-home the multi-tool-ordering item (1.V now reuses the core).

Leakwatch clean; no code touched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1.U is the proof the engine works end-to-end before any surface exists. It composes
the already-landed pieces behind the @relavium/llm seam (1.K FallbackChain, 1.N run
loop + RunEventBus, 1.O AgentRunner, 1.P node handlers, 1.Q human gate, 1.R
checkpoint/resume, 1.S node retry, 1.T ToolRegistry) on the in-memory ExecutionHost
reference — zero platform imports, no live network/keys, deterministic. No engine
code and no new ADR: the harness uses only already-exported @relavium/core symbols.

A scenario suite (the seed the Phase-2 CLI regression harness 2.K grows from):
- happy-path — the literal 3-node input -> agent(+tool call) -> output: live token
  streaming, per-attempt cost (attributed to the model), a gap-free sequenceNumber
  stream that validates against the canonical RunEventSchema.
- flagship — input -> agent -> human_gate -> output: in ONE run across a process
  boundary, the agent's forced provider error -> node retry (ADR-0040) -> failover to
  the second chain entry (1.K), per-attempt cost attributed to the fallback model;
  then a pause at the gate (the durable mid-run checkpoint persisted to the
  SQLite-shaped store); then a FRESH engine resumes via resumeFromCheckpoint, runs
  output to run:completed, and reproduces the final output with sequenceNumber
  continuing gap-free. The agent is NOT re-run on resume (its output is restored).
- determinism — a re-run yields an identical event signature + final output (the
  no-wall-clock/no-RNG ban the risk table binds to the harness).

Design decision (maintainer-approved): the resume demonstration uses a human gate as
the durable suspend point, because the Phase-1 engine resumes ONLY from a gate/budget
pause — a gate-less interrupted run is reconciled to run:failed (ADR-0036; verified by
the resume-model review + an adversarial refutation). The literal 3-node topology is
preserved by the happy-path member; the flagship adds the gate. Recorded as the §1.U
"Harness shape" clarification in the phase plan (no new ADR — composes decided
mechanisms). Exit criterion #1 annotated to point resume at the gated flagship.

Per-attempt cost is asserted as distinct cost:updated events (not only the terminal
cumulative); all LLM cost is incurred pre-gate, so the plain-human-gate cost-restore
deferral (deferred-tasks.md) is off this path.

1.U / M2 will be marked Done only after this PR merges (roadmap-done-after-merge).
Full gate (format/lint/typecheck/test/build) green — core 683 tests; Leakwatch clean.

Refs: §1.U, M2, ADR-0036, ADR-0040, ADR-0003

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A maintainer-approved hardening batch from the deferred-tasks.md triage. No production
behavior changes: tests + test-support + a CI job + an AgentTurnLimits doc-comment + the
deferred-tasks ledger. The five items, each verified against current code first:

1. Combined tool-loop DoS bound (core/agent-turn). The "product" framing was imprecise —
   the bounds are NOT multiplicative: maxToolTurns is the egress ceiling (≤ maxToolTurns+1
   provider calls); maxToolCorrections is a monotonic sub-budget that only ends the turn
   EARLY. Documented on AgentTurnLimits + an interleaving test (correctable/genuine/
   correctable/correctable → tool_failed at turn 4, far under maxToolTurns; corrections
   accumulate across the interleaved genuine round; egress stays bounded).

2. DeepSeek reasoning replay (llm/openai). Confirmed correct: reasoning_content is captured
   inbound but intentionally dropped on egress (output-only — echoing it 400s; deepseek-
   reasoner needs no prior reasoning). Locked with a test that a prior-turn reasoning part
   never reaches the request body. No seam-shape carrier needed (unlike Anthropic-redacted /
   Gemini-thoughtSignature, which stay deferred).

3+5. Conformance tool-loop + cache-hit scenarios (llm/conformance, the 1.F follow-up). Added
   a multi-turn tool loop across all four provider suites — a new replayFetchSequence (+ a
   replayFor router; the Gemini transport indexes per call) drives two generate() calls so
   turn 2 exercises the adapter lowering a tool_result back onto the wire (the call→result→
   continuation path every agent node runs). Added an optional cacheReadTokens expectation
   asserted in textGenerate (DeepSeek's fixture records prompt_cache_hit_tokens: 4 → folds
   into the one canonical Usage). New replay helpers covered by replay.test.ts.

4. Coverage in CI (ci.yml). Added a repo-ROOT `pnpm coverage` job (makes the root-relative
   per-glob ≥90% line+branch thresholds authoritative). ADVISORY (separate, non-required job
   like peer-dep-gate) while the core-package branch margin is thin (90.29% vs 90%); promote
   to required once stable under CI's Node 22.

Full gate (format/lint/typecheck/test/build) green — core 684 tests, llm 315; pnpm coverage
EXIT 0 (engine floor holds); Leakwatch clean.

Refs: deferred-tasks.md, ADR-0030, ADR-0039, ADR-0028, testing.md

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…node:completed)

Closes the deferred cost-event-persistence gap (1.AC/1.R). `cost:updated` is streamed
(#nodeEmit → bus), not persisted, so reconstructCheckpointState never saw it — a run
resumed cross-process restarted its cumulativeCostMicrocents near 0, under-reporting
run:completed.totalCostMicrocents and under-blocking the budget governor after resume.
A budget-gate pause was already mitigated (budget:paused.spentMicrocents); a PLAIN
human-gate pause still lost the cost.

Fix (maintainer-approved least-invasive variant — carry on node:completed, not persist
cost:updated):
- @relavium/shared: NodeCompletedEventSchema gains an optional cumulativeCostMicrocents
  (the run-wide running total snapshotted at the node boundary). Additive +
  forward-compatible (old logs without it still parse) → no ADR, no new event, no
  hot-path durable write, no delivery-ordering change.
- engine: #completeNode populates it from #cumulativeCostMicrocents (which, by the
  boundary, already includes this node's cost — its cost:updated fired during execution).
- checkpoint.ts: the fold restores the cumulative from node:completed with a monotonic
  Math.max that reconciles, order-independently, with the existing budget:paused restore.

So a run paused at ANY gate (plain human or budget) now resumes with the right spend; a
gate-less crashed-mid-run is reconciled to run:failed (not resumed), so its loss is moot.

Tests: checkpoint.test.ts pins the plain-human-gate restore from a durable node:completed
(no cost:updated rows, the real log shape); the 1.U flagship harness now asserts the
post-resume run:completed.totalCostMicrocents reflects the pre-gate cost (was ~0 before).

Full gate (format/lint/typecheck/test/build) green; pnpm coverage EXIT 0 (engine floor
holds); Leakwatch clean.

Refs: deferred-tasks.md (cost-event persistence), 1.AC/1.R, ADR-0003, ADR-0028

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @cemililik, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@coderabbitai

coderabbitai Bot commented Jun 16, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a61b6374-73c7-4983-8131-9f6062368481

📥 Commits

Reviewing files that changed from the base of the PR and between 1613ef3 and f180bd6.

📒 Files selected for processing (8)
  • .github/workflows/ci.yml
  • CLAUDE.md
  • docs/roadmap/phases/phase-1-engine-and-llm.md
  • packages/core/src/engine/checkpoint.test.ts
  • packages/core/src/engine/checkpoint.ts
  • packages/core/src/engine/m2-e2e-harness.e2e.test.ts
  • packages/llm/src/conformance/spec.ts
  • vitest.config.ts
✅ Files skipped from review due to trivial changes (3)
  • vitest.config.ts
  • CLAUDE.md
  • docs/roadmap/phases/phase-1-engine-and-llm.md
🚧 Files skipped from review as they are similar to previous changes (4)
  • packages/core/src/engine/checkpoint.ts
  • .github/workflows/ci.yml
  • packages/llm/src/conformance/spec.ts
  • packages/core/src/engine/m2-e2e-harness.e2e.test.ts

📝 Walkthrough

Walkthrough

Adds durable cumulativeCostMicrocents to the node:completed event schema, writes it from engine#settleCompleted, and restores it monotonically in checkpoint.ts on resume. Introduces a 435-line M2 end-to-end Node harness test suite with happy-path, flagship checkpoint/failover/resume, and determinism scenarios. Extends the LLM conformance layer with replayFetchSequence/replayFor for multi-turn replay, toolLoop fixtures for all four providers, and a DeepSeek reasoning-egress regression test. Adds an advisory CI coverage job and updates roadmap/status documentation.

Changes

Cost Persistence, Agent-Turn Bounds, and M2 E2E Harness

Layer / File(s) Summary
NodeCompletedEvent cost field and engine snapshot
packages/shared/src/run-event.ts, packages/core/src/engine/engine.ts
Adds optional cumulativeCostMicrocents to NodeCompletedEventSchema (backwards-compatible) and writes the current run-wide value from engine#settleCompleted into the durable node:completed payload.
Checkpoint cost restoration on resume
packages/core/src/engine/checkpoint.ts, packages/core/src/engine/checkpoint.test.ts
applyNodeEvent conditionally restores acc.cumulativeCostMicrocents from event.cumulativeCostMicrocents via Math.max for monotonic order-independence and reconciles with budget:paused.spentMicrocents to prevent undercounting; adds three test scenarios for human-gate resume, higher budget spend reconciliation, and monotonic Math.max validation.
Agent-turn loop-bounds documentation and interleaving test
packages/core/src/engine/agent-turn.ts, packages/core/src/engine/agent-turn.test.ts
Expands block comment to clarify maxToolTurns vs maxToolCorrections non-multiplicative semantics and worst-case provider-call bound; reorganizes toolUseTurn helper earlier in test file for reuse and adds a test scripting interleaved correctable/genuine tool rounds asserting tool_failed after correction budget exhaustion with exactly one successful agent:tool_result event.
M2 end-to-end harness test suite
packages/core/src/engine/m2-e2e-harness.e2e.test.ts
Defines deterministic LLM provider stubs replaying scripted StreamChunk sequences, in-memory echo tool registry, two canonical workflows (happy path and flagship with human_gate/retry/failover), engine wiring with node executor and provider resolver, async drive helper coordinating event consumption and timer backoff, validation utilities for schema/sequenceNumber/cost extraction. Implements three tests: happy path validates tokens/tool/cost/schema; flagship drives process #1 to human-gate pause with retry/failover then resumes in fresh process #2 asserting restored output, cost persistence, and gap-free sequence continuity; determinism re-runs happy path twice comparing event signatures.
Advisory coverage CI job and roadmap/status documentation
.github/workflows/ci.yml, AGENTS.md, CLAUDE.md, README.md, docs/roadmap/*
Adds an advisory pnpm coverage GitHub Actions job with frozen-lockfile install, cache restore, and pnpm turbo run build before coverage enforcement; updates all status and roadmap documents to reflect 1.m4 completion (PR #26), shift critical path to 1.U Node harness (M2), defer user-triggered retry-from-node to Phase 2, and mark previously deferred items as resolved.

LLM Conformance Multi-Turn Replay and Tool-Loop Fixtures

Layer / File(s) Summary
Multi-turn replay infrastructure
packages/llm/src/conformance/replay.ts, packages/llm/src/conformance/replay.test.ts
replay.ts adds replayFetchSequence (sequential per-call replay with over-fetch guard and JSON body validation) and replayFor (type-guard dispatch between one-shot and sequence); replay.test.ts covers ordered multi-call, over-fetch rejection, invalid JSON, and single-vs-array routing.
Conformance spec: toolLoop fixture, cacheReadTokens, MakeReplayAdapter broadening
packages/llm/src/conformance/spec.ts
Adds optional cacheReadTokens to textGenerate expectations, extends ConformanceFixtures with toolLoop, broadens MakeReplayAdapter to accept single or array RecordedResponse, conditionally asserts cacheReadTokens in generate test, and adds a two-step tool-loop conformance test validating schema, final text normalization, and no trailing tool calls.
Provider toolLoop fixtures, cacheReadTokens, and conformance adapter wiring
packages/llm/src/conformance/fixtures/*, packages/llm/src/conformance/*conformance.test.ts, packages/llm/src/adapters/openai.test.ts
Adds toolLoop fixtures for all four providers; expands DeepSeek cacheReadTokens expectation; updates OpenAI/Anthropic/DeepSeek conformance adapters to use replayFor; rewires Gemini makeReplayAdapter to call-indexed replay; adds DeepSeek reasoning-egress regression test asserting reasoning_content is dropped on egress.

Sequence Diagram(s)

sequenceDiagram
  participant Process1 as Process `#1`
  participant Engine1 as WorkflowEngine (P1)
  participant Store as Durable Store
  participant Process2 as Process `#2`
  participant Engine2 as WorkflowEngine (P2)
  
  rect rgba(100, 150, 255, 0.5)
    note over Process1,Store: Flagship run with retry/failover to human gate
    Process1->>Engine1: run(flagshipWorkflow, input)
    Engine1->>Engine1: primaryProvider emits retryable error
    Engine1->>Engine1: node:retrying → wait for timer arming
    Engine1->>Engine1: fireTimers() → failover to fallbackProvider
    Engine1->>Engine1: agent succeeds, advance to human_gate
    Engine1->>Store: node:completed (cumulativeCostMicrocents)
    Engine1->>Store: human_gate:paused (run:paused)
  end
  
  rect rgba(100, 220, 150, 0.5)
    note over Process2,Engine2: Resume in fresh process instance
    Process2->>Engine2: resumeFromCheckpoint(store)
    Engine2->>Store: replay log → applyNodeEvent + cumulativeCostMicrocents
    Engine2->>Store: reconstruct checkpoint with restored cost
    Engine2->>Engine2: continue from paused human_gate
    Engine2-->>Process2: run:completed (gap-free sequenceNumber, cost persisted)
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~65 minutes

Possibly related PRs

  • HodeTech/Relavium#2: Both PRs modify packages/shared/src/run-event.ts to extend the run-event Zod contract—this PR adds optional NodeCompletedEventSchema.cumulativeCostMicrocents, while the retrieved PR adds optional cost:updated.attemptNumber—so they are part of the same event-schema evolution.
  • HodeTech/Relavium#26: This PR's engine checkpoint/resume changes (adding cumulativeCostMicrocents to node:completed and restoring it in applyNodeEvent/reconstructCheckpointState) are directly aligned with PR #26's pre-egress budget governor and budget-gate resume work, which also required fixing cumulative budget/cost state propagation across pause/resume boundaries.
  • HodeTech/Relavium#8: Both PRs extend LLM conformance infrastructure—this PR adds multi-call replay helpers (replayFetchSequence, replayFor) and multi-turn tool-loop fixtures that build directly on the conformance harness foundation introduced by PR #8.

Poem

🐇 Hoppity hop through the checkpoint gate,
Cost snapshots durable—resume won't wait!
Multi-turn tool loops, tested with care,
Reasoning content? Dropped into thin air.
The M2 harness flags a fresh start anew,
Gap-free sequences all the way through! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title accurately and specifically highlights the three main deliverables: M2 milestone (end-to-end harness), engine hardening pass, and cross-process cost persistence. It clearly summarizes the primary changes without being vague or overly broad.
Docstring Coverage ✅ Passed Docstring coverage is 91.67% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch development

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request completes milestone 1.m4 and advances toward M2 by implementing the end-to-end Node harness (1.U) and resolving several engine-hardening deferred tasks. Key changes include restoring cumulative run costs across process boundaries by snapshotting the running total onto durable node:completed events, adding a sequential replay fetch utility to support multi-turn tool loops in the conformance suite, and documenting/testing the non-multiplicative nature of tool-loop limits. The review feedback suggests improving debuggability in the sequential replay utility by including the total number of expected recordings in the error message when the sequence is exhausted.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +75 to +78
if (recorded === undefined) {
return Promise.reject(
new Error(`replayFetchSequence: no recorded response for call #${String(call)}`),
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When a test fixture sequence is exhausted, including the total number of expected recordings in the error message makes debugging fixture mismatches significantly easier.

Suggested change
if (recorded === undefined) {
return Promise.reject(
new Error(`replayFetchSequence: no recorded response for call #${String(call)}`),
);
if (recorded === undefined) {
return Promise.reject(
new Error(
'replayFetchSequence: no recorded response for call #' +
String(call) +
' (total recordings: ' +
recordings.length +
')',
),
);
}

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/roadmap/phases/phase-1-engine-and-llm.md (1)

894-895: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Remove the stray + before the ADR-0032 link.

This looks like an accidental diff marker and will render a literal plus sign in the published doc.

Suggested fix
-+ [ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md), designed in
+[ADR-0032](../../decisions/0032-desktop-rust-media-de-inline-amends-0018.md), designed in
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/roadmap/phases/phase-1-engine-and-llm.md` around lines 894 - 895, In the
file docs/roadmap/phases/phase-1-engine-and-llm.md at lines 894-895, remove the
literal plus sign character that appears immediately before the [ADR-0032] link.
This character appears to be an accidental diff marker that will render as a
literal plus sign in the published documentation instead of being hidden as a
version control marker.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/core/src/engine/m2-e2e-harness.e2e.test.ts`:
- Around line 75-84: The scriptedProvider function's stream method currently
returns an empty array when call exceeds the scripts array length due to the
nullish coalescing operator. To fail fast on unexpected stream calls, replace
the fallback behavior to throw an error when scripts[call] is undefined instead
of silently returning an empty array. This will immediately expose unintended
LLM invocations during testing rather than allowing them to proceed silently.

In `@packages/llm/src/adapters/openai.test.ts`:
- Around line 582-584: Remove the unsafe `as` cast on the `messages` variable
assignment in the regression test. Instead of casting sent['messages'] with the
`as` operator, use a proper type guard such as Array.isArray() to safely
validate that the messages property is an array before accessing it, or
restructure the code to properly type the sent object from its source to
eliminate the need for the cast entirely. Ensure the logic still correctly finds
the assistant message and validates its content without bypassing TypeScript
type checking.

---

Outside diff comments:
In `@docs/roadmap/phases/phase-1-engine-and-llm.md`:
- Around line 894-895: In the file docs/roadmap/phases/phase-1-engine-and-llm.md
at lines 894-895, remove the literal plus sign character that appears
immediately before the [ADR-0032] link. This character appears to be an
accidental diff marker that will render as a literal plus sign in the published
documentation instead of being hidden as a version control marker.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8e5a907d-0eb1-4bbf-b76d-df1fc76bce3e

📥 Commits

Reviewing files that changed from the base of the PR and between 998f633 and eeb3f32.

📒 Files selected for processing (26)
  • .github/workflows/ci.yml
  • AGENTS.md
  • CLAUDE.md
  • README.md
  • docs/roadmap/current.md
  • docs/roadmap/deferred-tasks.md
  • docs/roadmap/phases/phase-1-engine-and-llm.md
  • packages/core/src/engine/agent-turn.test.ts
  • packages/core/src/engine/agent-turn.ts
  • packages/core/src/engine/checkpoint.test.ts
  • packages/core/src/engine/checkpoint.ts
  • packages/core/src/engine/engine.ts
  • packages/core/src/engine/m2-e2e-harness.e2e.test.ts
  • packages/llm/src/adapters/openai.test.ts
  • packages/llm/src/conformance/anthropic.conformance.test.ts
  • packages/llm/src/conformance/deepseek.conformance.test.ts
  • packages/llm/src/conformance/fixtures/anthropic.ts
  • packages/llm/src/conformance/fixtures/deepseek.ts
  • packages/llm/src/conformance/fixtures/gemini.ts
  • packages/llm/src/conformance/fixtures/openai.ts
  • packages/llm/src/conformance/gemini.conformance.test.ts
  • packages/llm/src/conformance/openai.conformance.test.ts
  • packages/llm/src/conformance/replay.test.ts
  • packages/llm/src/conformance/replay.ts
  • packages/llm/src/conformance/spec.ts
  • packages/shared/src/run-event.ts

Comment thread packages/core/src/engine/m2-e2e-harness.e2e.test.ts
Comment thread packages/llm/src/adapters/openai.test.ts Outdated
cemililik and others added 2 commits June 16, 2026 13:08
Verified each finding against current code; fixed the still-valid ones.

- CI (the red check): the advisory coverage job ran `pnpm coverage` after install-only,
  but `@relavium/*` package `exports` resolve only to `dist`, and `pnpm coverage` is a
  repo-ROOT vitest run (not a turbo task) so it does NOT get the `test` task's `^build`.
  A fresh checkout therefore had no `dist` → "Failed to resolve entry for @relavium/shared".
  It passed locally only because dist was already built. Fix: build workspaces before
  `pnpm coverage`. Confirmed: coverage EXIT 0, floor holds (each package still covers its
  OWN src via relative imports, so the src-accuracy of the floor is unchanged).
- m2-e2e-harness scriptedProvider: throw on an unscripted stream call instead of silently
  returning an empty turn — an unintended extra LLM invocation is now a loud harness bug.
- openai.test (DeepSeek drop-on-replay): replace the unsafe `as` on sent['messages'] with
  an Array.isArray + isRecord type guard (no cast, no unsafe member access).
- agent-turn.test: dedup — hoist the shared `toolUseTurn` to the describe top and drop the
  interleaving test's identical local `turn` (Sonar duplicate).
- replay.ts: include the recorded count in replayFetchSequence's over-fetch error message.
- phase-1 docs: fix two line-leading `+ ` (the flagged ADR-0031 + ADR-0032 join at L894 and
  the identical exact-FQDN-allowlist + SSRF join at L823) — a line-leading `+ ` renders as a
  stray bullet on GitHub; moved the `+` to the end of the prior line.

Full gate (format/lint/typecheck/test/build) green; `pnpm coverage` EXIT 0; Leakwatch clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…che)

A multi-dimensional Sonnet review (7 dimensions, adversarial refute-by-default verify)
surfaced 18 findings; 16 confirmed, 2 refuted. Fixed 15 (skipped 1 — see end). No product
behavior changed: test strengthening, code/doc comments, a CI cache step.

Test hardening (the meaningful ones):
- checkpoint.test.ts: the new node:completed cost-restore test was tautological for the
  Math.max reconciliation (a bare assignment would pass). Added two interaction tests over
  BOTH durable cost sources — incl. one (budget:paused 900 → node:completed 800 → 900) that
  gives Math.max teeth (a bare-assign would yield 800 and fail it). Clarified that the
  pre-existing "cost tallies" test exercises the fold's cost:updated arm (a defensive branch;
  cost:updated is streamed, not persisted — the production path is node:completed).
- m2-harness: tightened the under-constrained cost-count asserts to exact counts
  (happy-path === 2 tool+answer turns; flagship === 1 successful fallback — a double-charge
  or missing tool-turn cost would now fail); replaced the tautological node:retrying
  `.retryable === true` with the classified `.error.code === 'provider_unavailable'`; added a
  direct assertion that the checkpointed `work` node is NOT re-dispatched on resume; collapsed
  nodeOutput's double array scan to a single find.

Docs / comments (accuracy):
- checkpoint.ts: the budget:paused comment called cost-event persistence "the deferred
  general fix" — it landed; rewrote it + the CheckpointState JSDoc to describe both durable
  sources + the Math.max reconciliation.
- phase-1: exit criterion #1 no longer self-contradicts ("3-node" now names only the
  happy-path member; resume rides the gated flagship); added 1.AB to the §1.U composing-list
  (the harness needs the ExpressionSandbox). Same 1.AB addition in the harness header.
- CLAUDE.md: the status blurb was one giant **…** span with nested **…** that GFM can't
  render; scoped the bold to "Status:" so the inner emphases render.
- spec.ts: documented the toolLoop test's scope (asserts the call→result→continuation FLOW;
  the provider tool_result WIRE shape is asserted by the per-adapter unit tests).
- vitest.config.ts: corrected the stale "coverage not yet wired into CI / tracked in
  deferred-tasks" comments (it is now the advisory ci.yml job).

CI:
- The coverage job now restores the .turbo cache (like `ci`), so its build is a warm hit
  rather than a cold rebuild.

Skipped: the nit that commit 15a5b0e used a bare `test:` type without a scope — it is already
pushed; a history rewrite on a shared branch is unwarranted for a nit (future commits scope).

Full gate (format/lint/typecheck/test/build) green; `pnpm coverage` EXIT 0 (floor holds);
Leakwatch clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@cemililik
cemililik merged commit 64adf7e into main Jun 16, 2026
9 checks passed
cemililik added a commit that referenced this pull request Jun 17, 2026
…#27 merge

PR #27 merged, so per roadmap-done-after-merge the M2 milestone is reached: the 1.U Node
harness proves @relavium/core runs a workflow end-to-end (live streaming + per-node-boundary
checkpointing + cross-process resume + node retry + provider failover, per-attempt cost,
gap-free sequenceNumber). The Phase-1 engine critical path is complete.

- phase-1-engine-and-llm.md: banner advanced (M2 reached, critical path complete, remaining
  Phase-1 work is the additive 1.m5/1.m6 sub-spines, Phase 2 unblocked); §1.U header + the
  Acceptance marked ✅ Done/Met (PR #27); the M2 milestone-table row → M2 ✅ (achieved
  2026-06-16); the dependency-matrix 1.U row → Done (PR #27).
- roadmap/README.md: the global milestone spine M2 row → M2 ✅ *(achieved 2026-06-16, PR #27)*,
  mirroring the M0/M1 format.
- current.md: the milestone sentence now declares M2 reached (next global checkpoint M3); the
  immediate-next-steps tail records 1.U Done + M2 reached + the additive remainder + Phase-2
  unblocked.
- CLAUDE.md / README.md / AGENTS.md: status paragraphs advanced to M2 reached.

Not over-claimed: Phase 1 is NOT "complete" — only its engine CRITICAL PATH is. The agent-first
sub-spine (1.m5: 1.W/1.X/1.Y/1.Z/1.AA) and the multimodal sub-spine (1.m6: 1.AE–1.AH) remain as
additive, off-critical-path Phase-1 work. M2 is gated solely by 1.U (now landed); those sub-spines
explicitly do not gate it.

Docs-only; Leakwatch clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cemililik added a commit that referenced this pull request Jun 18, 2026
Consolidates an independent multi-dimensional review with a second reviewer's findings on
PR #32; every confirmed BLOCKER/HIGH and the actionable MEDIUM/LOW items are folded in.

Security (blocker/high):
- errors.ts: InvalidBaseUrlError no longer interpolates the raw base URL into its message or
  stores it on `.url` — a `user:pass@host` base URL leaked the credential to logs/events. It
  now keeps only a credential-free scheme+host summary (reusing the shared extractHttpsHost).
- content.ts: the shared SSRF range-primitive (isPrivateOrLocalHost) now NORMALIZES before the
  range check, closing real bypasses — numeric IPv4 (decimal 2130706433 / hex 0x7f000001 /
  octal 0177.0.0.1 / inet_aton short forms), the FQDN trailing dot (`localhost.`), and IPv6
  forms (`0::1`, fully-expanded loopback, IPv4-mapped/NAT64 via a proper 8-group parser).
  extractHttpsHost strips trailing dots; urlHasCredentials is HTTPS-only (consistent).
- openai.ts: a url-source media part is no longer forwarded to the provider as `image_url.url`
  (ADR-0031 §A7 — a media url is fetched by host/engine, never the adapter); url/handle are
  rejected with a typed bad_request, matching Anthropic/Gemini.

Correctness/honesty (medium):
- Capability matrices are honest at 1.AE: document:false (all three) + video:false (Gemini) —
  base64 video/document are ceiling-blocked and handle/url are deferred to 1.AF, so advertising
  them was "advertised-but-unsendable".
- openai.ts: audio/mpeg (canonical MP3 MIME) maps to format 'mp3'; an unsupported audio subtype
  (ogg/flac/…) is rejected, never silently coerced to 'wav'; PDF no longer mis-mapped to image_url.
- anthropic.ts + openai.ts: media on an assistant turn throws bad_request instead of a silent drop.
- capabilities.ts: documented that requiredCapabilities() coarsely requires `vision` for any
  media (per-modality FallbackChain gating lands in 1.AF).
- registry.ts: imports the shared extractHttpsHost/hasSmugglingChar instead of a duplicate copy
  (one primitive, never a second parser); the false "shared by enforceHttpEgress" docstring is now true.
- shared.ts: gateModality is typed against CapabilityFlags['media']['input'], not Record<string,boolean>.
- Anthropic content lowering deduped to one helper; deferred-tasks.md 1.AE PR ref corrected (#27#32).

Tests: SSRF negatives for every new encoding (numeric/trailing-dot/compressed+expanded IPv6) +
over-block guards; OpenAI positive media wire (image_url + input_audio) on generate AND stream;
audio/mpeg→mp3 + audio-subtype reject; document-gate + assistant-media reject; InvalidBaseUrlError
credential-redaction; the coarse requiredCapabilities media behavior. Full turbo gate green;
format/seam/shared-purity clean; Leakwatch 0.

Refs: ADR-0031, ADR-0029, ADR-0037, security-review.md, 1.AE
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant