perf(test): round two — one shared release tarball per packed run, concurrent template scaffolds, narrow defineConfig stub - #82
Merged
Conversation
The packed pool re-derived the same release artifacts per test file: seven full workspace `pnpm build`s (public-api-packed, release-audit x2, dev-workbench-packaging, rsc-runtime-optional-packaging, packed-release.e2e via the harness — the pool never set AGENT_BUNDLE_PACKAGE_PREBUILT) plus ten `npm pack`s and two copy+rslib rebuilds in the scaffolder e2e. `test:packed` now builds once, packs agent-bundle and create-agent-bundle once (scripts/run-packed-tests.mjs), and hands the shared tarballs plus the prebuilt seams to the pool through AGENT_BUNDLE_SHARED_PACK_DIR (tests/support/shared-pack.ts, with a lazy build-and-pack fallback for ad-hoc single-file runs). The stale-asset pruning test keeps its forced rebuild — the rebuild is the behavior under test. Census is unchanged (8 files, 22 passed / 1 skipped); the pool drops from 5m41s to 3m30s on the same machine and no longer rebuilds the workspace at all.
Each template test scaffolds, installs, and checks its own project under the shared runner, so nothing but the memoized pack fixture is shared. Running them concurrently cuts the scaffolder e2e from ~57s to ~19s wall on a development machine; the census is unchanged (same three tests).
seedEvalProject's node_modules/agent-bundle shim re-exported defineConfig from src/index.ts — the whole package entry. Seeded configs load through Jiti with the module cache off, so every project's config load re-transpiled the entire package graph: a flat ~5s floor under each of the 35 eval-service tests (3m9s for the file; the slowest file in the CI-serial integration leg at ~150s) and under every other eval-project consumer. Re-export from src/core/types.ts — defineConfig's defining module and the exact symbol src/index.ts re-exports — the same way project-fixture.ts already stubs it. eval-service drops from 3m9s to 15s locally with an identical 35-test census; eval-cli, eval-workbench, agent-api, dev-artifact-service, eval-native-mount, and evals-real inherit the same floor removal.
…production Review follow-ups from the shared-pack rework: the run-level build and both npm pack invocations now run with NODE_ENV=production (matching the production-build guarantee release-audit's in-test builds used to provide), the build moved into scripts/run-packed-tests.mjs so `test:packed` has one owner for the sequence, and the ad-hoc fallback in shared-pack.ts memoizes a single process-wide build so concurrent callers (the scaffolder e2e requests both packages at once) cannot race two workspace builds.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
commit: |
…eftovers Review follow-ups across the packed pool: one canonical installedEnvironment and npm-install flag list exported from tests/support/shared-pack.ts (five copies deleted; the workbench harness re-exports so its consumers are untouched), the two run-level npm packs in scripts/run-packed-tests.mjs now run concurrently, packed-consumer's two disjoint consumer installs run concurrently and the test documents why it deliberately bypasses the shared tarball (its deletable pack source proves the tarball holds no path references back to the pack root), dead packageRoot/packedServerStartupBudget leftovers and the harness's inline dynamic import are gone, the stale mobile-era locals in the desktop navigation walk are renamed with the redundant viewport call dropped, and public-api-packed's 15s budget joins its siblings at 30s (it wraps a real npm install plus a tsc run).
Frequency policy for the packed pool: per-PR release gates keep the single-cycle consumer proofs (pack once, install once per contract) plus one full scaffolder journey — the minimal template, which covers the installed scaffolder bin, template scaffold, scaffolder-driven npm install, project check, and clean validate. The mcp-server and cli-tool template runs move to scaffold-packed-matrix.e2e.test.ts, which runs in `test:packed:release` (pre-publish check:release and a new nightly CI schedule), not per PR. The shared scaffold fixture moves to tests/support/scaffold-fixture.ts; test names are unchanged, so the release-boundary pool census is identical to the old per-PR pool census, and the per-PR pool drops exactly the two matrix templates. CI's release-gates job now runs check:release:ci (per-PR pool); the nightly packed-matrix job runs the full check:release.
…set of the per-PR hosted gate
…o concurrent local-ci legs don't collide on /tmp
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the "70-minute test run" actually was
Job compute is healthy; the wall clock is runner-queue time. In the 75-minute CI run on
feat/prebuilt-payload-adapters(33390725959), every job finished in ≤11.5 min of compute, but Examples check and Verify (Node 26) sat queued for 64 minutes — each push starts 7 CI jobs plus a Package-preview job, and account-level runner concurrency saturates. Reducing total runner-minutes is the in-repo lever; this PR cuts the biggest consumers (3× Verify legs, release-gates) without touching coverage.Fixes (each its own commit, behavior-preserving)
test:packedrun (13b0c470,9b49dd7d) — the packed pool re-derived the same artifacts per test file: 7 full workspace builds (the pool never setAGENT_BUNDLE_PACKAGE_PREBUILT, several helpers ignored it, plus in-testpnpm builds) and ~10npm packs + 2 copy-and-rebuild cycles.test:packednow builds once (NODE_ENV=production) and packs agent-bundle + create-agent-bundle once (scripts/run-packed-tests.mjs), sharing the tarballs throughAGENT_BUNDLE_SHARED_PACK_DIR/tests/support/shared-pack.ts(lazy build-and-pack fallback for ad-hoc single-file runs). The stale-asset pruning test keeps its forced rebuild — that rebuild is the behavior under test.41858abc) — the three scaffolder e2e templates each own their project directory; running them viait.concurrentcuts the file from 37.5s to 15.4s (57s → 19s standalone).defineConfigstub in eval fixtures (744a65f8) —seedEvalProject'snode_modules/agent-bundleshim re-exporteddefineConfigfromsrc/index.ts; seeded configs load through Jiti with the module cache off, so every seeded project re-transpiled the whole package graph (~5s flat floor under each test). Re-exporting fromsrc/core/types.ts(the defining module, same symbol — the existingproject-fixture.tspattern) removes the floor:eval-service.test.ts206s → 11.8s, and eval-cli/evals-real/cli/agent-api/eval-workbench/dev-artifact-service inherit it.Measured before/after (paired runs, same machine, alternating order; background load recorded — fixed runs generally ran under heavier load, so deltas are conservative)
test:unittest:packed(reporter-matched, build excluded)pnpm checkend-to-end (fixed branch)pnpm check:releaseend-to-end (fixed branch)Biggest per-file integration wins (CI-serial shape): eval-service 206→11.8s, eval-cli 61→2.7s, evals-real 35.9→8.7s, cli 50.3→25.2s, agent-api 16.6→6.0s. Packed per-file: scaffold-packed 37.5→15.4s, release-audit 33.3→15.5s, public-api-packed 28.7→16.8s, dev-workbench-packaging 27.3→19.3s, rsc-runtime-optional 13.7→6.3s. Expected CI effect: each Verify leg's 12.8-min
pnpm testdrops ~4–5 min ×3 legs, easing the queue for every push.Census proof (name-level)
No changeset: nothing published changes (test/support/scripts layer only). Conflict zones respected: no edits to
src/build/rslib.ts,build.test.ts,hooks.test.ts(PR #80) orpacked-consumer.test.ts,overview.e2e.test.ts,runtime-playground.e2e.test.ts,examples/rsc-agent-runtime(PR #71).Flagged for a future pass (not done here — this round cuts no coverage)
overview.e2e.test.tsis now the dominant integration file (106s of the 540s serial pool) — natural round-three target, but PR feat: prebuilt payload adapter mode (RFC #50 Phase 3) #71 touches it today.dev-workbench-packaging's forced in-placedistrebuild racing release-audit's audit-script pack; restructuring would collide with PR refactor(build): serve generated entries through VirtualModulesPlugin at dedicated virtual paths #80's build-plugin rewrite.npm install --prefer-offlinein packed consumers (~2s × ~10 installs) — skipped to avoid touching install semantics in release-audit paths.Adoption follow-ups (this PR was adopted after its authoring agent was aborted)
Merges from main: #71 and #80 merged during review; both are merged in and the full gate suite re-ran green on the merged head.
Simplify/deslop pass (
f45dade4): one canonicalinstalledEnvironment+ npm-install flag list exported fromtests/support/shared-pack.ts(five copies deleted; the workbench harness re-exports); the two run-levelnpm packs and packed-consumer's two disjoint installs run concurrently; deadpackageRoot/packedServerStartupBudgetleftovers and the harness's inline dynamic import removed; stale mobile-era locals in the packed-release desktop navigation walk renamed and a redundant viewport call dropped;public-api-packed's 15s budget joined its siblings at 30s (it wraps a realnpm installplus atscrun).packed-consumer.test.tsnow documents why it deliberately bypasses the shared tarball: it packs from a deletable copy and deletes the pack source after install, proving the tarball holds no path references back to the pack root — a hermeticity property the shared (workspace-packed) tarball cannot test.Packed-pool frequency policy (
0bf2d54e)Real npm work stays real, but each proof now has an owner:
pnpm test/pnpm check): zero packed npm work, unchanged (the one remaining cycle iscli.test.ts's pack+install in the integration pool — flagged for round 3, entangled with that file's local helpers).check:release:ci): the single-cycle consumer proofs (shared tarball: one build + two packs per run; one install per distinct consumer contract) plus one full scaffolder journey (minimal template: installed bin → scaffold → scaffolder-driven install → check → validate). Per-PR packed pool: 21 tests (20 passed / 1 skipped).check:releasepre-publish + nightlypacked-matrixCI job): adds the scaffolder template matrix (scaffold-packed-matrix.e2e.test.ts: mcp-server, cli-tool). Full pool census is name-identical to the old per-PR pool: 23 tests (22 passed / 1 skipped).What a template-matrix regression would now hit before the release gate: nothing per-PR beyond the minimal-template smoke — an mcp-server- or cli-tool-template-specific break (template manifest drift, stdio-entry wrapping, bin/lib/dts conventions inside a scaffolded project) surfaces at the nightly run or pre-publish
check:release. The framework-side halves of those behaviors remain covered per-PR bypacked-consumer.test.ts(framework-build project: bin/lib/dts + stdio lifecycle from the installed tarball) and the artifact validators.Measured on the same machine (load ~45–60): old full pool on main 6m00s; new per-PR pool 3m02s; new release pool 3m05s (template matrix is concurrent, so its wall cost is small — the per-PR win is runner-minutes, npm work, and flake surface).