perf(build): lex compiler bundles once instead of re-parsing every module with acorn

[ScriptedAlchemy]

Summary

Artifact validation dominated agent-bundle build: on examples/host-test ~28 s of a ~40 s build. The cause was validateJavaScriptModules (src/build/validate-artifact-modules.ts) re-parsing every emitted bundle in full with acorn for its side effect alone — the AST was discarded and only the AB6005 "has invalid syntax" branch depended on it — and doing so on every pass: the pre-manifest content pass, the post-manifest strict pass, after rslib.ts had already lexed the same bytes for its residual-reserved-import check.

This PR keeps every check and changes what proves the syntax:

• Compiler bundles are lexed, not re-parsed. A module the framework compiled (manifest kind bundle) is the bundler's own output; the ESM lexer that drives the import-graph walk is its only syntax pass (ModuleSyntaxCheck lexed). It rejects unterminated strings, templates, comments, and regexps and unbalanced braces. Copied and generated modules the framework did not compile keep the full acorn parse (parsed). Prebuilt payloads stay opaque as before. acorn stays in package.json for that branch.
• Each module's imports are read once per process. New src/build/module-imports.ts reads imports through one function and remembers the result by <check>:<sha256> (bounded map). The digest is the one inspectArtifact already computed, so the post-compile self-containment check in rslib.ts and both validation passes of one build share one lex of a multi-megabyte bundle.
• Import-graph walk unchanged: every import a literal specifier naming a Node built-in or resolving to a regular manifest file inside the artifact, no non-literal dynamic imports, no bare package names. A build whose tools hatch could have rewritten the emitted assets keeps the full parse for its bundles too. AB6005 codes and messages are unchanged.

Also folded in (routed from the orchestration lane; behavior-preserving, artifact bytes unchanged): virtualModulesPluginConstructor was duplicated verbatim in rslib.ts and mcp-apps.ts — now one helper in meta.ts taking (rspack, packageName, purpose), each path still checking its own engine object with its own message text; and enforceInvariants in rslib.ts appended the RSC-manifest DefinePlugin and the VirtualModulesPlugin in two separate spreads — now one append, same position, same order.

Profile

node --cpu-prof on a full agent-bundle build of examples/host-test, self time by module:

	Before (main)	After
Sampled total	40.2 s	11.7 s
acorn/dist/acorn.mjs	22.6 s	—
GC (acorn AST churn)	3.7 s	negligible
es-module-lexer wasm	1.6 s (3 lexes per bundle)	0.56 s (1 lex per bundle)
SHA-256 hashing	—	0.55 s
@rspack/core	2.3 s	2.3 s

So ~28 s of the 40 s build was module validation, matching the report; after the change the bundler itself is the largest item.

Timings

Wall time, warm cache, one run each (AGENT_BUNDLE_WORKBENCH_PREBUILT=1):

	Before (main)	After
host-test build	40 s	12.5 s
host-test validate --artifact	17 s	3.6 s
audiobook-curator build	12.7 s	6.8 s

The parse-once/concurrency follow-ups were not needed after this.

Proof of identical results

artifact-validator.test.ts (280 assertions) and the build/mcp/hooks/package-build/api/cli integration pools pass unchanged in outcome. Two tests whose fixture corrupted a compiler bundle with export const broken = ; — a bare statement error a bundler never emits, which the lexer accepts — now corrupt it with an unterminated template literal, which the lexer rejects; a new test pins the kind-based gating (copied and generated modules parsed in full, bundles lexed). module-imports.test.ts pins the import records, the digest cache, and what each check level rejects.

Artifact diff for the folded simplification: examples/host-test and examples/audiobook-curator built before and after that commit (on the pre-#518 base) differed only in the MCP entries' then-unfixed .artifact.stage-XXXXXX token and those files' sha256 in the manifest; every other byte was identical. Rebased over #518, build-reproducibility.test.ts passes on this branch: two builds emit byte-identical artifacts.

Locally: unit 3205/3205; integration build, hooks, mcp, package-build, cli-routes-build, api, plugin-bundle, cli, artifact-validator, prebuilt-payload, target-stages (277/277); pnpm lint, pnpm typecheck, pnpm docs:site:build green.

Docs

website/docs/{en,zh}/guide/distribution/validation.mdx ("Content-addressed, not path-present"): how emitted modules are walked and how the syntax check follows who produced the module. Changeset: .changeset/validate-artifact-lex-once.md (patch).

Review status

Per the maintainer's instruction this PR carries no comments; review threads are answered here.

• Rebased onto main 61ff1f1dd (over chore(build): official Rsbuild plugin adoption — drop the standalone publint gate, register the framework-owned plugin set #509, refactor(workbench): fail the Agent Document and host discovery atoms with their typed client errors #515) before opening; rebased again onto 5b9d3b72c after fix(build): byte-reproducible artifacts — root the generated-module namespace at the project root #518 and feat(routes): publish routes.d.ts through Effect FileSystem with an ensuringRemoved staging file (FileSystem phase 1, module 3) #520 merged (one rslib.ts conflict, resolved by keeping fix(build): byte-reproducible artifacts — root the generated-module namespace at the project root #518's project-rooted virtualRegistryModules(cwd, …) with this PR's rslibVirtualModulesPlugin()). Rebased once more onto 5a86149d9 after test(build): pass the project root to the composed configs in the plugin-registry test #523 (the main typecheck fix, fix(build): byte-reproducible artifacts — root the generated-module namespace at the project root #518 × chore(build): official Rsbuild plugin adoption — drop the standalone publint gate, register the framework-owned plugin set #509) and feat(cli): route first-party CLI terminal I/O through Effect Terminal/Stdio; spell routed-CLI input errors in CLI terms (#465) #505 merged; the duplicate test-fix commit dropped out.
• Last Codex-reviewed head: 3f9e91b. Later heads (e44e45567 review fixes, 72e8ddaed rebase) carry no review request under the no-comments rule; merging on green CI per the reviewer-quota fallback.
• Codex 3f9e91b P2 ×3, all fixed in e44e45567:
  • Keep syntax validation for post-processed bundles (validate-artifact-modules.ts:126): agreed for the hatch case. ValidateArtifactOptions.bundleSyntaxCheck (lexed | parsed) now chooses how a kind: 'bundle' module is checked; build selects parsed whenever tools.rspack or tools.rsbuild is configured, since a hatch runs after Rspack parsed the source and can rewrite the asset, and lexed otherwise (the bundler's own bytes). Both validation passes of a build receive the same choice. Pinned by build.test.ts "parses emitted bundles in full when a tools hatch could have rewritten them" (a raw BannerPlugin banner export const broken = ; fails the build with AB6005) and the parsed branch of the kind-gating validator test; documented in both locales.
  • Key cached imports to the bytes actually read (:144): the validator now always reads the module and keys the cache by sha256Hex of those bytes rather than the inspection's digest, so a rewrite between inspection and read is never answered from the cache; hashing costs ~0.3 s on host-test, whose build still takes 12.4 s.
  • Document the Node built-in exception (validation.mdx:31): both locales now state that an import may name a Node built-in (node:fs, fs) or resolve as a relative / file: specifier to a regular manifest file inside the artifact.

[changeset-bot]

🦋 Changeset detected

Latest commit: 72e8dda

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
agent-bundle	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-04T04:15:26.926390Z	3f9e91b	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f9e91b3f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[pkg-pr-new]

npm i https://pkg.pr.new/ScriptedAlchemy/agent-bundle@521

npm i https://pkg.pr.new/ScriptedAlchemy/agent-bundle/create-agent-bundle@521

npm i https://pkg.pr.new/ScriptedAlchemy/agent-bundle/@agent-bundle/runtime@521

commit: 72e8dda