test-speed audit: ranked time sinks and classification (static audit, post-#118 speed pass plan)

[ScriptedAlchemy]

Static-only audit of where test time goes and whether it is earned, fixable, or slop. No measurement pools were run (machine reserved for gate queues). Numbers are either [measured] — extracted from tonight's local-ci gate logs (96 cores, 4 concurrent legs, AGENT_BUNDLE_TEST_TIME_SCALE=4, green run laneA attempt 3 + failed attempt 2 + in-flight laneC) — or [estimated] from declared budgets and file structure, and are labeled per row.

1. Measured pool shape (tonight's gate logs)

Pool / step	Green run [measured]	Failing run [measured]	Shape
test:unit	37–39s (1795/1799, 131 files, 32 workers)	60–63s with 3 fails	healthy, ~21ms/test
test:integration	2m19s–2m28s (557/559, 51 files, 4 workers/leg)	6m52s–7m06s	failure-burn triples cost
examples:check	5m14s (187/193, 2 suites + 5 example builds, serial)	10m34s (rsc fail), 39s (fail-fast)	gate critical path
check:release (packed pool + audits)	4m16s (22/23, 9 files, 1 worker)	—	gate critical path
eval:spot	17s	—	fine
Gate wall time	9m52s green / 10m43s failed	—	gates leg ≈ 9m47s of it

Also measured: a serial re-run of the 6 heaviest browser files (examples-real, mcp-app-real, overview, runtime-playground-capture, runtime-playground-hmr, runtime-playground) = 193.7s for 28 tests (rstest report, 02:41 UTC). Failing tests burn ~7.5–8.2s each (scaled poll budgets), confirming failures — not green-path waste — are what blows runs up.

2. Ranked heavy hitters and classification

#	File (pool)	Cost	Evidence	Classification
1	workbench/tests/packed-release.e2e.test.ts (packed)	est. 90–180s of check:release; 360s budget	[estimated from budget/structure]	earning-its-runtime (see §3)
2	workbench/tests/runtime-playground-capture.test.ts (integration)	est. 60–120s; 600s budget, largest in repo	[estimated]; member of measured 193.7s/6-file set	fixable — misplaced proof level (see §4)
3	examples:check serialization (gates leg)	5m14s	[measured]	fixable — serial-shape artifact: --workspace-concurrency=1 over 5 disjoint example dirs; rsc example additionally pins maxWorkers: 1
4	workbench/tests/examples-real.e2e.test.ts (integration)	4 journeys, 90–150s budgets	[estimated]; in measured 6-file set	earning — AGENTS.md-mandated browser acceptance (populated + stale-diagnostic + repair)
5	workbench/tests/overview.e2e.test.ts (integration)	21 tests, ~15 server boots	[estimated from structure]	earning, margin to share read-only servers; #118-owned file — defer
6	workbench/tests/mcp-app-real.e2e.test.ts (integration)	4 tests incl. one 180s budget	[estimated]	earning; #118-owned — defer
7	create-agent-bundle/tests/scaffold-packed.e2e.test.ts (packed)	est. 60–120s: scaffold + auto-install + full check inside scaffold	[estimated]	earning — the one per-PR scaffold journey (matrix already nightly-only)
8	agent-bundle/tests/packed-consumer.test.ts (packed)	est. 30–60s: own rslib build + own npm pack + 2 installs	[estimated]	earning — deliberately NOT the shared tarball: packs from a deletable copy to prove no path references leak back to the pack source (documented in-file)
9	agent-bundle/tests/dev-workbench.test.ts (integration)	30 tests, ~20 dev-server boots	[estimated]	mostly earning — lifecycle/fencing tests kill their own server, per-test boot inherent; #118-owned — defer
10	agent-bundle/tests/public-api-packed.test.ts + dev-workbench-packaging.test.ts (packed)	3 + 2 npm installs of the same tarball into separate consumers	[estimated ~10–30s recoverable]	fixable — repeated installs, where flags match (plain vs --omit=dev differ, so only partial)

3. The 1,136-line packed+browser journey: earning its runtime

Full read of packed-release.e2e.test.ts. What it proves that no smaller packed or browser test does:

• install isolation (realpath walk of every runtime dep out of the workspace), PATH-clamped children;
• all 13 Agent API tools called against the installed tarball with set-equality (called === expectedAgentApiToolNames);
• multi-epoch lifecycle A→B(good)→stale(invalid, diagnostics retained)→C(repaired) — the exact stale-diagnostic + repair flow AGENTS.md requires browser acceptance to cover, against the packed product;
• native run cancellation with process-tree death verification (descendantProcessIds);
• foreground kill → outage UI → restart → session recovery, validated against a full network ledger (packed-outage-ledger);
• 9-route desktop navigation floor post-recovery; shutdown hygiene (no leaked temp roots, processes, dev.lock, or token in output).

Its only fixed sleep is a 1s quiet fence — a negative proof (nothing new fails after recovery) that cannot be event-driven. Splitting the journey would multiply npm installs and server boots; one install + one server amortized across ~30 phases is the cheap shape. The 360s budget is headroom (timeScale-multiplied), not runtime. Verdict: keep as-is. Marginal trim: a few page-render assertions (Skills/Artifacts headings) also proven per-page by overview.e2e, but they ride an already-paid page load — no real win.

4. The one genuine misplacement: evidence-capture harness in the per-PR pool

runtime-playground-capture.test.ts ('captures identity-backed HMR, last-good, recovery, and desktop browser evidence', 600s budget) spawns a subprocess that drives a full HMR/recovery browser journey to produce screenshots + evidence.json — a documentation/evidence artifact. The behavioral contracts it exercises (HMR activation, last-good retention, recovery) are already proven interactively by runtime-playground.e2e.test.ts and runtime-playground-hmr.e2e.test.ts in the same pool. Recommendation: move the capture test to the gates leg or nightly (evidence regenerates when the flow changes, not per PR-pool run). Est. win: 60–120s per integration pool run [estimated from budget/structure]. Caution: its helper workbench/tests/helpers/runtime-playground-fixture.ts is #118-owned, so sequence after #118 lands.

5. Patterns swept for and NOT found (why the slop hypothesis fails)

• Fixed sleeps: repo-wide grep found only 8 literal sleeps ≥50ms in tests; all are justified negative-proof observation windows of 25ms–1s (state stays pending during cleanup; ignored watcher paths stay silent; sandbox side effects stay absent; post-recovery quiet fence) or event-driven fallbacks. Zero "sleep then assert the event happened" instances.
• Budgets that run to the poll interval: eventually/eventuallyPasses/within/expect.poll all early-exit on success (1–50ms poll steps). Budgets only burn on failure — measured: failing runs cost 3× green (integration 7m vs 2m2Xs).
• Repeated builds/packs: the shared-pack machinery is genuinely one build + two tarballs per run; the only self-packing test (packed-consumer) documents a contract reason. Installs marked cache-safe use --prefer-offline; the networked ones (release-audit, native smoke, scaffold matrix) are deliberate consumer-reality proofs per perf(test): serve packed-pool installs from the npm cache with --prefer-offline #109.
• Spawn-per-assertion: not found; spawn-heavy files batch many assertions per spawned process (e.g. cli.test.ts runs MCP + hook + artifact assertions from one packed consumer).
• Dead weight: skipped native-host tests are env-gated it.skip with no fixture setup paid; no trivial-getter integration tests found.
• Proof-level duplication: only two small findings — the CLI-manifest-producer contract proved at both built (public-api.test.ts) and packed (public-api-packed.test.ts) level, and cli.test.ts holding a packed-consumer proof inside the integration pool (perf(test): remove the CI serial-worker pin with flakes fixed at the source #118-owned file). Both are seconds, not minutes.

6. #118 overlap markers

git diff origin/main...origin/perf/ci-parallel-unpin --name-only = 31 files. Heavy hitters above that are #118-owned and must wait: cli.test.ts, dev-workbench.test.ts, dev-workbench-packaging.test.ts, overview.e2e.test.ts, mcp-app-real.e2e.test.ts, public-api-packed.test.ts, release-audit.test.ts, script-playground-service.test.ts, rsc-runtime-optional-packaging.test.ts, all rstest.*.config.ts, tests/support/shared-pack.ts, tests/support/time-scale.ts, workbench/tests/helpers/runtime-playground-fixture.ts. NOT #118-owned (safe immediately after it lands, or now if isolated): runtime-playground-capture.test.ts itself (helper is owned — sequence after), examples/*/package.json + root examples:check script, packed-release.e2e.test.ts (no change recommended anyway).

7. Recommended post-#118 implementation order

1. Parallelize examples:check (root script --workspace-concurrency=2..3; examples are disjoint dirs — packed-consumer's comment already relies on npm-cache concurrency safety). Est. 2–3 min off the gate wall [estimated]; it is the measured critical path (5m14s of a 9m52s wall). Minimal perf(test): remove the CI serial-worker pin with flakes fixed at the source #118 contact (root package.json only).
2. Move the 600s evidence-capture test out of the per-PR integration pool (gates leg or nightly). Est. 60–120s per integration run [estimated]. Touch after perf(test): remove the CI serial-worker pin with flakes fixed at the source #118 lands (shared fixture helper).
3. Consolidate packed-pool installs where flags match (public-api-packed: 3 installs → 1–2). Est. 10–30s off check:release [estimated]. perf(test): remove the CI serial-worker pin with flakes fixed at the source #118-owned file — fold into the post-perf(test): remove the CI serial-worker pin with flakes fixed at the source #118 pass.
4. Optional dedupe: drop or unit-ify the built-level CLI-manifest-producer test now that the packed level pins it; revisit cli.test.ts's packed-consumer test placement. Seconds each.
5. Do not split the packed-release journey, re-serialize anything, or re-litigate npm-cache flags (perf(test): serve packed-pool installs from the npm cache with --prefer-offline #109) — those shapes are earning their cost.

8. Verdict on the hypothesis

Not slop. Green-path hygiene is unusually good: event-driven polls everywhere, one shared build+pack per run, no fake-timer debt, batched spawns, gated skips that cost nothing. The perceived slowness decomposes into (a) serial-shape artifacts — examples:check one-at-a-time (biggest lever, ~5m14s measured) and the CI-side integration single-worker shape #118 already unpins; (b) inherent proof cost — real rsbuild/rslib builds, real npm installs, real Chrome, mandated release journeys; and (c) failure burn — a flaky run costs ~3× a green one under scaled budgets (measured 7m vs 2m2Xs integration), which makes #118's seven timing-race fixes the highest-value speed change already in flight. One true misplacement found (the 600s capture harness), plus a couple of seconds-scale dedupes.

[ScriptedAlchemy]

Post-#118 speed pass — implementation status for the recommended order in §7:

1. Parallelize examples:check — PR perf(examples): parallelize examples:check with scaled budgets (#128 rec 1) #146. --workspace-concurrency=3 over the five disjoint example dirs, plus an AGENT_BUNDLE_TEST_TIME_SCALE floor of 2 (mirrors the integration pool's multi-worker poolTimeScale; never lowers local-ci's 4 or hosted CI's pinned 4). Verified green end to end: 3m47s wall including the root build, vs the 5m14s measured serial step. A scale-1 trial run at concurrency 3 reproduced exactly one budget timeout (dev-invocation.integration.test.ts, 45_000 * timeScale), confirming the floor is load-bearing.
2. Move the 600s evidence-capture harness out of the per-PR integration pool — PR perf(tests): move the 600s evidence-capture journey to a nightly pool (#128 rec 2) #148. New nightlyEvidenceTestFiles + rstest.evidence.config.ts + root test:evidence, wired into the nightly packed-matrix CI job. Verified: the full capture journey passes under the new pool; typecheck/lint green. Sequenced after perf(test): remove the CI serial-worker pin with flakes fixed at the source #118 as required (the fixture helper is untouched — pool membership and new wiring only).
3. Consolidate packed-pool installs (public-api-packed) and 4. optional dedupes — not taken in this pass; both are seconds-scale and can fold into the next touch of those files.

[ScriptedAlchemy]

Audit complete — closing with implemented + declined items

Static audit purpose satisfied; recommended implementation order from §7 has a clear terminal state.

§7 implementation status

#	Recommendation	Outcome	Evidence
1	Parallelize examples:check	Shipped	#146 — --workspace-concurrency=3; measured ~3m47s vs 5m14s serial
2	Move 600s evidence-capture harness out of per-PR integration pool	Shipped	#148 — test:evidence nightly pool
3	Consolidate packed-pool installs (public-api-packed)	Declined this pass	Seconds-scale; deferred to next touch of that file (per prior status comment)
4	Optional dedupes (built vs packed CLI-manifest, cli.test.ts placement)	Declined this pass	Seconds-scale each

Verdict unchanged

Hypothesis not slop — green-path hygiene is good; biggest lever (#1) landed; the one true misplacement (#2) moved to nightly; packed-release journey correctly kept intact.

No further audit actions planned under this issue. Closing.

[ScriptedAlchemy]

Closing: audit recommendations 1–2 shipped (#146, #148); 3–4 explicitly declined. No further action under this tracker.