ci: split Verify into a fast leg + 2 integration shards behind the required gate; pin the browser; retry setup; nightly pools (#576)

[ScriptedAlchemy]

Part of #576 (Rstest audit) — the test-level P1/P2 items. Out of scope here, as agreed on the issue: the config-level items (worker-root teardown, setup-file loading, timeouts, stale-dist guard, adapter libId, pool membership — follow-up to #570), the contract-matrix split and mcp-lineage import (after #569), and CI shape (ci.yml, separate PR).

Six lanes, one file set each, integrated on one branch. Every mechanism below was re-derived from the current code, not copied from the issue; two of the issue's diagnoses are corrected (items 3 and 6).

1. P1 — runtime-client-surface-proxy.test.ts waited the real 15 s twice

Mechanism. upstreamRequestTimeout = 15_000 was a module constant (src/dev/runtime-client-surface-proxy.ts:26) armed before every upstream dispatch — the bootstrap entry fetch and every proxied asset. bounds an upstream HTTP request before headers arrive and keeps a completed 502 response intact when a response body stalls after headers both drive an upstream that never completes, so each case sat on that deadline: 15.0 s + 15.0 s, 30.6 s of a 33 s file, and the unit pool's wall was bound by this one file regardless of worker count (issue: 46.2 / 43.1 / 45.5 s at 16 / 32 / 95 workers).

Fix. RuntimeClientSurfaceProxy.open(...) takes a trailing options: RuntimeClientSurfaceProxyOptions = {} with upstreamRequestTimeoutMs?: number; the default is exported as defaultRuntimeClientSurfaceUpstreamRequestTimeoutMs = 15_000 and validated (positive safe integer ≤ the 2^31−1 setTimeout ceiling — Node collapses larger delays to 1 ms, which would silently drop the bound; out-of-range values throw TypeError before the proxy opens). Production callers (workbench-server.ts) pass four arguments and keep 15 s. The two tests pass { upstreamRequestTimeoutMs: 500 }, keep their exact assertions (status: 502; { body: 'Not Found', status: 502 } + socket closed within 250 ms), and tighten their outer bound from within(pending, 16_000) to 2_000, so a proxy that ignored the option now fails instead of passing at 15 s. A new fast test asserts the default once and rejects 0, -1, 1.5, NaN, Infinity, 2^31.

Proof (shared 96-core host, 1-min load noted; same command shape before/after, unmodified vs. patched scratch worktree):

	reporter total	wall	load
unit pool before ×3	42.7 / 42.5 / 43.2 s	44.9 / 44.5 / 45.5 s	23–58
unit pool after ×2	39.9 / 37.1 s	42.1 / 39.1 s	37–44
unit pool after, on the merged branch (gate)	see gate table

The two cases: 15.0 s → 0.21 s each; file host span 30.6 s → 1.1 s. The pool's wall is now bound by the next file (eval-native-mount.test.ts, 25.8 s), as the issue predicted. 10× file loop: 10/10, 3.1–3.8 s per run.

2. P1 — dev-host-install.test.ts re-sync test raced a real ProjectWatcher

Mechanism. The test built its DevCoordinator without createWatcher, so the coordinator installed the real ProjectWatcher (coordinator.ts:244, #startEffect :331-338, onInvalidation → this.rebuild): chokidar over the fixture root, 100 ms debounce (watcher.ts:90-93). The test's own writeFiles therefore scheduled an unrequested rebuild C, queued behind the explicit rebuild B (coordinator.ts:288-295) and started from B's onExit (:421-424). Every successful build mints a fresh epoch id and publishes artifact.available; the install manager turns each into a sync() that rewrites DEV_INSTALL_MARKER, and settled() (host-install-manager.ts:391-393) only awaits the syncs pending at call time. Normally C fails or is rejected as sameInputs because the test's broken write lands while C compiles; when host sync of B is slow relative to C (a loaded runner), C completes first, its sync moves the marker, and :370 sees epochId: C ≠ markerBeforeFailure — the exact CI failure. Reproduced deterministically with an instrumented harness outside the repo: with a 2.5 s delayed epoch acquisition the marker moves (FAIL (marker b306cf8d -> 75fa7946)); with a stubbed watcher it does not.

Fix. Four lines: createWatcher: () => ({ close: async () => undefined }) — the DevelopmentWatcher shape dev-coordinator.test.ts already uses (:917, 955, 1052, 1094). The coordinator now rebuilds only on the test's explicit rebuild() calls; assertions unchanged. Not "drain": there is no idle signal on the coordinator, so a drain would be a wall-clock wait around inotify + debounce.

Proof. Single test ×20 before: 0 failures (does not reproduce natively on this host — fast disk); ×20 after: 20/20, 11.8–16.8 s per run (fixture build dominates), load 27–58. Whole file under the integration config: 7/7 (Claude and Codex variants ran), 106 s; under the root config the CI host-install-proofs job uses: 7/7, 105 s. Integrator's 10× loop on the merged branch: see gate table.

3. P1 — scaffold-packed-matrix.e2e asserted "failedTests": 0, a key the default reporter never prints

Mechanism (corrects the issue). The key does exist — in Rstest's md reporter, which initCli selects automatically when it detects an AI-agent environment (CURSOR_AGENT, CLAUDECODE, CODEX_*, …) and no reporter is configured. The scaffolded project inherits the developer's environment, so under an agent the template's npm run test:routes printed the md summary block and the assertion passed for its author; in CI (GITHUB_ACTIONS, no agent variables) the reporters are ['default', 'github-actions'] and the key never appears — red every nightly, and pnpm test:evidence (the next step) has never run in CI. Second latent failure the issue did not list: the default reporter expands per-test lines only for single-file runs, so the cli-tool projection pool (two files) fails toContain('greets through the routed CLI shell …') at line 96 before it reaches line 98. Real default-reporter output of a scaffolded pool, for the record:

 ✓ tests/projection/cli-dispatch.test.ts (5)
 ✓ tests/projection/script-dispatch.test.ts (3)

 Test Files 2 passed
      Tests 8 passed
   Duration 1.36s (build 75ms, tests 1.29s)

Fix. Test-only; the templates are untouched. tests/support/scaffold-fixture.ts gains expectPassedPool(projectRoot, script, testNames): runs npm run <script> -- --reporter=json (npm appends the flag to the template's rstest --config … script; an explicit CLI reporter overrides both the agent md swap and the CI default), locates the report on stdout (the only thing there that opens a line with {), reads a non-zero exit rather than throwing (the report is already written and names the failing test), and asserts in order: failing tests [], failing files [], tests[].name ⊇ testNames (a dropped or empty pool reports status: 'fail', tests: 0 — and failedTests: 0, so the names are what catch it), then { status: 'pass', summary: { failedTests: 0 } }. Three call sites replace the six toContains; the two hunks stay clear of #569's edits in the same file.

Proof. CI-faithful (env -u CURSOR_AGENT CI=true GITHUB_ACTIONS=true) pnpm test:packed:release packages/create-agent-bundle/tests/scaffold-packed-matrix.e2e.test.ts: before → exit 1 (expected ' Rstest v0.11.10\n\n ✓ tests/route-u…' to contain '"failedTests": 0' at line 50; line 96 on the cli-tool test); after → exit 0, Test Files 1 passed, Tests 2 passed, 1m15s wall (load 41→49); again under the agent environment: exit 0, 1m11s. Negative proofs against sabotaged scaffolded copies: an appended failing it → the failing-tests assertion prints the entry with its errors[]; a deleted tests/projection/ → expected [] to deeply equal ArrayContaining [...]. Note for the next person: pnpm test:packed:release -- <file> forwards -- to pnpm exec rstest, where cac treats it as end-of-options and drops the filter — pass the file without --.

What the evidence artifact looks like now. pnpm test:evidence (the step that has never run in CI) passes: 1 file / 1 test, 51 s wall including pnpm build (34 s of test). runtime-playground-capture.test.ts runs packages/workbench/scripts/capture-runtime-playground.mjs into mkdtemp(join(tmpdir(), 'agent-bundle-runtime-capture-')) under the worker's isolated tmp root and removes it in finally — nothing survives the run and nothing is uploaded. Reproduced by invoking the script directly: five 1440×900 PNGs (desktop.png — populated Runtime playground with "All outputs are from the current runtime generation (generation-3). No stale views."; hmr-before.png; hmr-after.png; compile-error.png — Diagnostics tab showing AB8206 RSC runtime source build failed beside the last-good generation-2 banner; recovered.png) and evidence.json (1.4 kB): appMarkerVisible: true, appRefreshPreservedDocument: true, appVisibleBefore/After/Recovered: true, compactRunGeneration: "generation-2", compactRunId (== compileErrorRunId), compileErrorDiagnosticsVisible: true, compileErrorGeneration: "generation-2", compileErrorHistoryUnchanged: true, compileErrorLastGoodVisible: true, compileErrorLayout: { diagnostics: { top, bottom, viewportHeight: 900 }, lastGood: { … } }, documentTimeOriginBefore === documentTimeOriginAfter (HMR without reload), desktopControlColumns: 4, generationBefore/After/Recovered: "generation-1"/"generation-2"/"generation-3", lastGoodGenerationDuringError: "generation-2", lastGoodPreserved: true, providerSessionId, runBefore/runAfter (distinct UUIDs), recovered: true, sandboxOpaqueOrigin: true, viewports: { desktop: { width: 1440, height: 900 } }. ci.yml still needs test:evidence as its own step with if: ${{ !cancelled() }} — that is the CI-shape PR.

4. P1 — mcp-app-real.e2e inferred readiness from a notification count, then waited 120 s for a /close that could never come

Mechanism. The gate is McpAppFrameRelay.#resourceProvided (packages/workbench/src/mcp/mcp-app-frame.tsx): it flips only when the sandbox proxy posts ui/notifications/sandbox-proxy-ready, and close() branches on it — unset → #forceClose() → DELETE /api/mcp/apps/<id>; set → POST …/close. The test counted ui/notifications/initialized /messages requests across both previews (toBe(2)), a different channel from the relay flag and not tied to the reopened binding, then waitForRequest('/close', { timeout: 30_000 * timeScale }) — 120 s in CI. When the relay it closed had not flipped, only a DELETE was ever sent and the wait consumed the whole budget (2m16s failures on ~15 % of Node 22.19 main pushes).

Fix. Option (2) from the issue, the one that observes the actual gating state, plus (3) and (4). The relay publishes its lifecycle on the outer iframe as data-mcp-app-relay-state ∈ loading | ready | closing | closed (#publishFrameState(), called on exactly the events that change #state/#resourceProvided: start(), the proxy-ready flip, close(), #completeClose(); no React state, timers, or polling — the same imperative-attribute pattern applyMcpAppFramePolicy already uses on this iframe). McpAppFrameIframe.setAttribute is optional so the unit-test fakes publish nothing. The test: captures the reopened /apps POST response to derive the binding path (per-binding instead of a count); await expect(outerFrame).toHaveAttribute('data-mcp-app-relay-state', 'ready', { timeout: browserTimeout }) before clicking; then waits for that binding's own initialized; after the click, polls the recorded requests for the first close-path request for that binding and throws immediately with Expected the reopened App preview to close gracefully (POST …/close); the relay sent DELETE /api/mcp/apps/<id> instead. if it was the force path; asserts the teardown acknowledgement; every wait is browserTimeout (32 s in CI) instead of 120 s. AppRouteRequest records method (moved up from RuntimeAppRouteRequest, which already had it).

Proof. Workbench rebuilt (data-mcp-app-relay-state present in packages/workbench/dist/static/js/index.js); e2e file 10×: 10/10 (40/40 tests), 34.6–47.8 s per run, close-step test 4.0–5.5 s, load 34–50; a passing toHaveAttribute(..., 'ready') in every run is direct evidence the relay publishes on the real production path. mcp-app-frame.test.ts (read-only, includes the two force-delete cases and the markup test): 19/19. Integrator's 10× loop on the merged branch: see gate table.

5. P1 — mcp-app-preview-browser.test.ts read a Node-side request log synchronously after an in-page signal

Mechanism. bootstrapRequests is appended by the bootstrap server's request handler (:21-25) — i.e. after Chrome has opened the connection and Node's parser has emitted 'request'. Every in-page signal the test waited on fires before that: create:* is pushed inside the mocked fetch before the create response is even parsed, two React commits and a network round-trip ahead of the GET; factory is pushed in the passive effect of the commit that arms the iframe src, so the GET is at best concurrently in flight. waitForFunction (raf polling) sees the event on the next frame, replies over CDP, the test reads the array — and on a 2-worker 4-core runner the CDP replies occasionally win: expected [ '/runtime-bootstrap' ] to deeply equal [ '/runtime-bootstrap', …(1) ].

Fix. One local helper next to the assertions, expectBootstrapRequests(requests, expected) = await expect.poll(() => requests.length, { timeout: 5_000 }).toBe(expected.length) then the exact toEqual (an over-fetch still fails with the array diff). Applied at the four sites that observe a freshly issued bootstrap GET (old lines 247, 332 — the CI flake — 446, 476); the four "still N" sites after a settled state stay synchronous (polling there would turn a genuine over-fetch into a 5 s wait with a worse message). Nothing in the fixture/setup region changed, so this merges cleanly with the shared Rsbuild-fixture extraction in flight.

Proof. 40 runs: 39/40 — the one failure was a mid-edit of mcp-app-frame.tsx by lane 4 landing in the fixture bundle (SyntaxError at fixture init, before any assertion); the following 36 runs against the completed edit all passed, 5.7–6.7 s typical. A scratch Playwright script mirroring the create: shape lagged the Node log 2/300 unpinned and 1/300 pinned to two CPUs — consistent with CI's twice-in-36 h. Integrator's 10× loop on the merged branch: see gate table.

6. P2 — packed-outage-ledger Date.now() fences

Mechanism (corrects the issue's second shape). Shape 1 (unexpected pre-outage failures: GET /api/logs/replay?after=0 … net::ERR_ABORTED): the Logs page issues the replay from its mount effect and aborts it from the effect cleanup; the e2e left Logs on the heading alone, so on a loaded runner the navigation aborted the replay 22 ms in, before any headers, and the allowlist accepted a replay abort only with a 2xx status (ledger:149-156). Shape 2 (unknown post-recovery failure: […after=0…, …after=5…, /api/logs/stream?after=36]): not the after=0→5 pair — the code shows that message can only fire after the fresh-B filter and its per-failure assertion passed, and the two-reader shape (transport stream?after=0 right after the POST; session controller stream?after=N after the trace refresh; both aborted before the DELETE) was already accepted. The unclaimed entry was the Logs navigation stream: its request and response events arrived in one Playwright batch, the awaited continuation ran synchronously into the next iteration's openedAt = Date.now() — which is the previous record's leftAt — so leftAt === at and the exclusive window request.at < navigation.leftAt excluded it; the message then dumped every post-recovery failure, making the recognised ones look guilty. Both shapes reproduce deterministically by replaying the exact CI ledger entries through the old validator.

Fix. Exactly two shapes are newly accepted, each documented in the ledger: (a) a pre-outage GET /api/logs/replay net::ERR_ABORTED with no response (pre-header navigation abort; a non-2xx answer is still rejected); (b) a navigation-owned request observed in the same millisecond as the departure stamp (request.at <= navigation.leftAt). The fresh-B close window is derived from wire entries — opens at the completion of the session's last POST …/operations, closes at the completion of its DELETE (both awaited via expect.poll) — instead of Date.now() around the click; the e2e waitForResponses the Logs replay (and requires .ok()) before navigating away; recognised failures are a Set and the assertion reports only unclaimed entries. Everything else still fails: replay abort after non-2xx, replay abort via POST, non-abort replay errors, unknown-path pre-header aborts, fresh-B stream abort without headers / before the last operation completed / after the DELETE completed / a third abort / non-canonical cursor, navigation abort delivered before departure, navigation-shaped request after departure, navigation stream abort without headers, any unclaimed post-recovery abort — 13 guard shapes exercised against the new validator.

Proof. Old validator vs. CI entries: shape 1 throws unexpected pre-outage failures, shape 2 throws the same three-entry dump as CI; controls (200 replay; leftAt + 1 ms; the after=0/after=5 pair alone) pass. New validator: both shapes pass, all 13 guards still throw. packed-release.e2e 10× against pre-packed tarballs: 10/10, 39.4–45.1 s per run, load 49→18. packed-outage-ledger.test.ts untouched and green. Integrator's 10× loop on the merged branch: see gate table.

Gate (merged branch @@GATE_SHA@@, after git merge origin/main and a fresh pnpm build)

@@GATE_TABLE@@

Changeset

patch for agent-bundle — production source changed in item 1 (RuntimeClientSurfaceProxy.open options). RuntimeClientSurfaceProxy is exported only from the internal src/dev/index.ts barrel, not from src/index.ts or any exports entry, so no docsite page describes it; no docs change. packages/workbench (item 4) is private.

Self-review

Reviewer: change-risk-reviewer on gpt-5.6-sol-medium, two passes against origin/main.

Pass 1 (48d19b7):

1. Blocker — a failed rstest could be reported green. The shard step's pnpm … | tee ran under the default bash -e shell (no pipefail), so the pipeline returned tee's status; the following banner grep passed and the gate could go green on red shards. Fixed: set -euo pipefail in the step (88d470b). CI run 2 then correctly failed the step — for a second reason: sed | grep -q under pipefail dies with SIGPIPE (141) when grep exits on the match; fixed with one grep -qP that tolerates rstest's colour codes (8d27683). Run 3 green.
2. Should-fix — local gate omitted two pools it was documented as running. scripts/local-ci.mjs ran test:unit + test:integration only. Fixed: Verify legs now run test:route-unit and test:projection as well; docs/local-ci.md table updated (88d470b).
3. Q1 (any test file or leg dropped?): no finding — fast + integration-1|2 cover unit/route-unit/projection/integration; the shard guard proves exact partitioning; the Node 26 include adds exactly one job on PRs and nothing on main (9 combinations).
4. Q2 (gate passes with a skipped shard?): no finding at the wiring level — failure/cancelled/non-docs skipped all fail; only a docs-only pull_request may pass on skipped. (The pipefail hole above was the only path, now closed.)
5. Nit — the worker-isolation test does not assert PLAYWRIGHT_BROWSERS_PATH=0 is preserved. Dismissed: ??= semantics; covered by reading the one-line implementation.
6. Docs/changeset: no finding — CI/test infrastructure only; no publishable source changes; skip-changeset appropriate.

Pass 2 (8d27683, after the fixes): no findings. Confirmed: the shard step fails on a failed rstest and passes only with the exact banner (set -euo pipefail, single grep -P, 20-min job timeout bounds hangs); no remaining path lets Verify gate pass without every shard passing; channel: 'chromium' is the documented full-build/new-headless selection and playwright install chromium installs that build; the /api/artifacts/epochs/<id> allowlist entry is justified (same-origin GET ERR_ABORTED only, the MCP page's own AbortController on unmount — server/HTTP failures surface as other errors); --pool.maxWorkers applies to the route-unit/projection configs and the new local-ci step ids collide with nothing.

Codex (chatgpt-codex-connector) reviewed 48d19b7 with no comments. Nightly proof: nightly.yml cannot be dispatched until it exists on main (gh workflow run → 404); it will be dispatched right after merge and its evidence / mcp-conformance results linked in a follow-up comment.

@@SELF_REVIEW@@

[changeset-bot]

⚠️ No Changeset found

Latest commit: 457d2c4

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-05T02:25:30.916945Z	48d19b7	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[pkg-pr-new]

npm i https://pkg.pr.new/ScriptedAlchemy/agent-bundle@583

npm i https://pkg.pr.new/ScriptedAlchemy/agent-bundle/create-agent-bundle@583

npm i https://pkg.pr.new/ScriptedAlchemy/agent-bundle/rsc-markdown-stream@583

npm i https://pkg.pr.new/ScriptedAlchemy/agent-bundle/@agent-bundle/runtime@583

commit: 457d2c4

[ScriptedAlchemy]

Post-merge proofs (squash dd3f550b4):

• First main push on the new matrix — run 33941144447: all 9 Verify (leg, Node) jobs + Verify gate green, 7.0 min wall (last 10 main runs before: 13.4–16.3 min). Legs: fast 3.9–4.6 min, integration-1 5.1–5.3 min, integration-2 4.9–6.5 min.
• Nightly, dispatched from main — run 33941185524:
  • Evidence capture (Node 22.19) — green (Test Files 1 passed, 1.4 min): first hosted run of rstest.evidence.config.ts ever.
  • MCP conformance (Node 22.19) — green (MCP conformance 0.1.16 / spec 2025-11-25: 7 passed, 23 failed, 0 skipped, 23 = expected-failure allowlist; results artifact uploaded).
  • Packed release matrix (Node 22.19) — red on exactly the known assertion (scaffold-packed-matrix.e2e.test.ts:98, expected '…test:routes…' to contain '"failedTests": 0'), non-blocking as documented in nightly.yml; goes green when test/576-p1-test-fixes lands.