feat(test): script-dispatch and workbench-surface proof levels; routed cli-tool template (#370, #102 migration 6)

.changeset/script-dispatch-and-workbench-surface.md

@@ -0,0 +1,6 @@
+---
+"agent-bundle": minor
+"create-agent-bundle": patch
+---
+
+Add `runScript`, `scriptJson`, `scriptNdjson`, and `inspectWorkbenchSurface` to `agent-bundle/test`, and move the `cli-tool` template onto the routed CLI. `runScript` (the `script-dispatch` proof level) runs a conventional `src/scripts/*` module through its generated executable's contract: a rendered `.tsx` script through the rendered-script shell with piped Markdown, TTY, `--json`, and `--ndjson` output and the project's conventional `src/providers/*` mounted with the `script` invocation (a `process.exit` in rendered code fails the run as the executable's shell reports its render worker's exit, never ending the test process), a plain `.ts` script as a Node process of its own with the `main` envelope, `process.exit`, exit code, stdout, stderr, optional `stdin`, and the compiled `agent-bundle/meta` identity (no `AB4760` outside a compiled surface); `testManifest().scripts` (a new required member of `AgentBundleTestManifest`, so a hand-built manifest literal must now supply it) lists only the compiled scripts that ship — a nested (`AB4808`) or configuration-conflicting (`AB4809`) conventional script is never a `runScript` target — and every failure names the script route, execution form, and proof level. `inspectWorkbenchSurface` (the `workbench-surface` proof level) returns the route manifest, grouped route catalog, state declaration, lifecycle-replay fixtures, and page availability the Workbench would show for a project, without a browser or dev server, and reports `manifest-unavailable` with the compiler's error diagnostics (for example `AB4100`) for a project the compiler rejects. `ScriptRouteProps` types rendered script components. `create-agent-bundle`'s `cli-tool` template replaces the hand-written `src/cli.ts` with a routed `src/cli/greet.ts` command and a conventional `src/scripts/hello.ts`, proved by a generated projection pool at the `cli-dispatch` and `script-dispatch` levels. (#398)

docs/entry-conventions.md

@@ -360,7 +360,10 @@ await renderRoute('tool:curator/status', {
 ```
 
 The same seam accepts `actor`, `workspace`, and `capabilities`; tests can use
-`unavailable(...)` to pin a transport's honest absence semantics.
+`unavailable(...)` to pin a transport's honest absence semantics. `invokeCli`
+(routed commands) and `runScript` (conventional scripts) accept the same
+`context` for their rendered surfaces and open the request scope with the
+surface-specific `invocation.kind` the generated executable would use.
 
 ### Migration nudges
 

packages/agent-bundle/README.md

@@ -423,13 +423,15 @@ kind, and the module provenance.
 Conventional request context providers (`src/providers/*`, see
 [entry conventions](../../docs/entry-conventions.md#request-context-providers-power-tier))
 are mounted automatically for every manifest-backed helper — `renderRoute`,
-`renderRouteEvents`, `invokeCli`, and the in-memory MCP helpers — exactly as the
+`renderRouteEvents`, `invokeCli`, `runScript` (rendered scripts), and the
+in-memory MCP helpers — exactly as the
 generated request scopes mount them: discovered from the compiled manifest,
 executed once per request in the same deterministic key order, handed the same
 surface-specific `invocation` (`tool`, `event`, `cli`, `script`), and failing the
 request closed when a factory throws. `providers.processLifetime` is scoped the
-way the artifact scopes it: each `invokeCli` call and each `renderRoute` render
-is a fresh simulated executable (hit 1, new `instanceId`), while one open
+way the artifact scopes it: each `invokeCli` call, each `runScript` run, and
+each `renderRoute` render is a fresh simulated executable (hit 1, new
+`instanceId`), while one open
 `openInMemoryMcpServer` session shares a single identity across every request
 it handles, like the artifact's warm Flight worker. Pass `context.providers` to opt out: an explicit map is mounted
 verbatim and no conventional provider runs, which is how a test stubs a provider
@@ -480,6 +482,8 @@ is never a receipt for another.
 | `route-unit` | `renderRoute`, `renderRouteEvents` | a route module renders to the document (and render-event stream) it claims |
 | `mcp-in-memory` | `openInMemoryMcpServer`, `invokeMcpTool`, `readMcpResource`, `getMcpPrompt`, `listMcpSurface`, `runContractMatrix` | the real generated MCP server's protocol contract, over the SDK's in-memory transport |
 | `cli-dispatch` | `invokeCli`, `cliJson`, `cliNdjson` | a plain or rendered argv vector resolved and run through the routed CLI's own shell, including rendered Markdown, explicit TTY, JSON, and NDJSON modes, in-process |
+| `script-dispatch` | `runScript`, `scriptJson`, `scriptNdjson` | a conventional `src/scripts/*` module run through its generated executable's contract: a rendered `.tsx` script through the rendered-script shell in-process (piped Markdown, explicit TTY, `--json`, `--ndjson`), a plain `.ts` script through the `main` process envelope as a Node process of its own over the source — fresh module state, real `process.exit`, its own argv, exit code, and streams — without bundling |
+| `workbench-surface` | `inspectWorkbenchSurface`, `workbenchSurfaceFromRouteGraph` | what the dev server would hand the Workbench for this project — the route manifest, the grouped route catalog, the state declaration, lifecycle-replay fixtures per host, and page availability — from the same compiler pass and projection functions, with no browser and no dev server |
 | `packed-stdio` | `openPackedMcpServer`, `runPackedContractMatrix` | a built artifact's generated entry running as a real process over stdio |
 | `packed-deleted-source` | `removeProjectSource`, `openPackedMcpServer({ deletedSource })`, `runPackedContractMatrix` | the packed stdio process still runs after project source and configuration are removed and verified absent |
 | `host-install` | `openInstalledHostMcpServer`, `runInstalledHostContractMatrix` | a built bundle staged into an isolated host root, discovered in the emitted host format, and spawned from the installed layout |
@@ -505,6 +509,106 @@ const tty = await invokeCli(['library', 'report', './books'], { tty: true });
 expect(tty.stdout).toContain('\r\u001B[2K');
 ```
 
+`runScript` is the same idea for the `src/scripts/*` convention. The manifest
+carries every conventional script with its extension contract
+(`testManifest().scripts`), and the helper runs the module through what its
+generated `scripts/<name>.mjs` would do — never by bundling it: a rendered
+`.tsx` script runs in-process through the same shell the executable uses,
+and a plain `.ts` script runs as a Node process of its own, as the
+executable does (see below):
+
+```ts
+import { runScript, scriptJson, scriptNdjson } from 'agent-bundle/test';
+
+// script-dispatch, rendered .tsx script: `--json` / `--ndjson` are reserved by
+// the framework, everything else reaches the component's `argv` prop.
+const summary = await runScript('summary', ['./books', '--json']);
+expect(summary.exitCode).toBe(0);
+expect(scriptJson(summary)).toMatchObject({ arguments: ['./books'] });
+expect(scriptNdjson(await runScript('summary', ['./books', '--ndjson'])).at(-1)?.type).toBe('complete');
+
+// script-dispatch, plain .ts script exporting main(argv): the envelope adopts
+// a numeric return as the exit code; stdout and stderr are captured.
+const checksum = await runScript('checksum', ['./books']);
+expect(checksum.exitCode).toBe(0);
+expect(checksum.stdout).toBe('Fixture checksum: 7\n');
+```
+
+A plain script runs as a Node process of its own over the source module, so
+the process contract is Node's rather than a simulation of it: every run
+evaluates the module afresh (module-level state never survives between runs,
+as it never survives between processes), `process.argv` is
+`[node, <source>, ...argv]`, `process.exit` ends the script for real — work
+queued after it never runs, whether or not the script caught the call —
+process-level APIs such as `process.chdir` work and affect only the script, a
+numeric `main` return goes through the real `process.exitCode` setter (`300`
+reports `44`; `1.5` exits 1 with the setter's `RangeError`), a signal that
+ends the process reports as `128 +` its number, and the test process's own
+argv, exit code, cwd, and streams are never touched, so plain runs may overlap
+each other and any other test. The builder's static export scan decides
+between the `main` envelope and a self-executing module, and a non-callable
+`main` fails the way the generated executable fails. The process resolves
+relative `.js` specifiers to their TypeScript sources, transforms `.ts` with
+Node's own type transform, lowers the `.tsx` and `.jsx` helpers a plain
+script imports with the bundler's SWC — the same lowering the generated
+executable was built with — and serves `agent-bundle/meta` as the identity the build stamps from the
+manifest's `plugin`. Explicit `scripts:` configuration entries are bundled
+entries rather than routes and stay with the packed level. A rendered script
+composes the project's root layout (a script belongs to no server, so no
+server layout applies) and mounts the project's conventional providers with the `script` invocation the
+generated executable passes (`context.providers` substitutes a fixture map, as
+everywhere); a plain script opens no request scope, so it accepts no `context`
+at all. A rendered script's declared state mounts on a disposable root for
+the run, as at every harness level (`renderRoute`, `invokeCli`): the
+`AGENT_BUNDLE_PLUGIN_ROOT` / `.agent-bundle` anchor a `workspace-durable`
+store keeps between executable runs is the packed artifact's, and the packed
+level proves it; a test that needs one store across several rendered runs
+passes the same `context.state` and `context.noticeLedger` bindings to each.
+`stdin` pipes input to a plain script (omitted,
+it reads end-of-file at once); `process.execArgv` is empty as under plain
+`node`; an aborted `signal` sends SIGTERM and, should the script trap it,
+kills the process after a one-second grace before the run rejects. A rendered
+script's own `console` and stream writes during the run land on the
+invocation's `stderr` — the generated executable forwards its render worker's
+stdout and stderr there — so `stdout` holds machine output only and nothing
+escapes into the test runner. `process.exit` from rendered code is that
+worker's exit, never the test process's: the run fails as the executable's
+shell reports it (`Generated render worker exited with code N.` on `stderr`,
+exit code 1, `0` included), the call unwinds the caller, and whatever code
+that catches it writes afterwards is discarded, as a worker that has exited
+writes nothing. Once a rendered run's `signal` aborts, no state mount or
+module load that has not begun is started on its behalf. Every `ScriptInvocation` carries
+`provenance.execution` (`rendered-shell`, `main-envelope`, or
+`self-executing`) beside the level.
+
+`inspectWorkbenchSurface` answers "what would the Workbench show for this
+project?" without a browser. It runs the dev server's own preparation as the
+Workbench server constructs it — `development` mode for a configuration
+factory that branches on `context.mode`, the selected `configPath` for both
+the compiler pass and eval-suite discovery — and the same projection functions
+the dev server serves — `GET /api/routes/manifest` and `GET /api/lifecycles`
+byte for byte — then applies the Workbench's own grouping and navigation
+rules:
+
+```ts
+import { inspectWorkbenchSurface, workbenchPageLabel } from 'agent-bundle/test';
+
+const surface = await inspectWorkbenchSurface({ root: projectRoot });
+expect(surface.catalog.groups.map((group) => group.label)).toContain('curator · Tools');
+expect(surface.catalog.stateDefinition).toMatchObject({ driver: 'sqlite', lifetime: 'workspace-durable' });
+expect(surface.lifecycles[0]?.targets.map((target) => target.target)).toEqual(['claude', 'codex']);
+expect(surface.pages.map(workbenchPageLabel)).not.toContain('Playground');
+```
+
+`counts` are the artifact inventory the Workbench counts, derived without a
+build: one instance per hook, MCP server, or script declaration per selected
+target it names (a declaration whose `targets` select none of the project's
+targets is emitted nowhere and counts nothing), plus the declared Skills, eval
+suites, and targets. Page availability depends only on whether each count is
+zero and on what the compiled graph declares. Host discovery, live MCP probes, published epochs,
+and the RSC runtime page are artifact- or process-bound and are not projected
+here.
+
 `expectEvents` asserts over a render-event stream. `toContainSequence` is
 sequence-tolerant — an extra `progress` or `replace` frame is legal and cannot
 turn a passing render red — while a missing frame, a reordering, or a regressed

packages/agent-bundle/fixtures/route-harness/src/badge.tsx

@@ -0,0 +1,6 @@
+import type { ReactElement } from 'react';
+
+/** A JSX helper shared by the fixture's plain `badge` script; not a route. */
+export const Badge = ({ label }: { readonly label: string }): ReactElement => (
+  <span className="badge">{label}</span>
+);

packages/agent-bundle/fixtures/route-harness/src/ribbon.d.ts

@@ -0,0 +1,3 @@
+import type { ReactElement } from 'react';
+
+export declare const Ribbon: (props: { readonly label: string }) => ReactElement;

packages/agent-bundle/fixtures/route-harness/src/ribbon.jsx

@@ -0,0 +1,6 @@
+/**
+ * A JavaScript JSX helper (`.jsx`, imported with its extension) shared by the
+ * fixture's plain `badge` script; not a route. The bundler lowers it through
+ * the React plugin, as it does the `.tsx` helper beside it.
+ */
+export const Ribbon = ({ label }) => <em className="ribbon">{label}</em>;

packages/agent-bundle/fixtures/route-harness/src/scripts/badge.ts

@@ -0,0 +1,17 @@
+import { Badge } from '../badge.js';
+import { Ribbon } from '../ribbon.jsx';
+
+/**
+ * A plain script that imports a `.tsx` helper — and, with `--ribbon`, a
+ * JavaScript `.jsx` one — the way a project shares presentational pieces
+ * between its rendered and plain scripts: the bundler lowers the JSX for the
+ * generated executable, and the harness must do the same for the source it
+ * runs.
+ */
+export const main = (argv: readonly string[]): number => {
+  const label = argv.filter((argument) => !argument.startsWith('--')).join(' ') || 'unlabelled';
+  const element = argv.includes('--ribbon') ? Ribbon({ label }) : Badge({ label });
+  const { children, className } = element.props as { readonly children: string; readonly className: string };
+  process.stdout.write(`<${String(element.type)} class="${className}">${children}</${String(element.type)}>\n`);
+  return 0;
+};

packages/agent-bundle/fixtures/route-harness/src/scripts/banner.ts

@@ -0,0 +1,5 @@
+/**
+ * A self-executing plain script: no `main` export, so the artifact bundles the
+ * module as-is and its top-level code runs when the process evaluates it.
+ */
+process.stdout.write(`banner: ${process.argv.slice(2).join(' ')}\n`);

packages/agent-bundle/fixtures/route-harness/src/scripts/blank.tsx

@@ -0,0 +1,6 @@
+/**
+ * A rendered script module that evaluates but exports no component: the
+ * generated executable's render worker reports the shape failure through its
+ * event stream, so the process writes the failure to stderr and exits 1.
+ */
+export const notAComponent = true;

packages/agent-bundle/fixtures/route-harness/src/scripts/broken.tsx

@@ -0,0 +1,16 @@
+import type { ScriptRouteProps } from 'agent-bundle';
+
+/**
+ * A rendered script whose module fails to evaluate. The generated executable
+ * loads it inside its render worker and the failure reaches the shell as an
+ * event-stream error: stderr carries the message and the process exits 1.
+ */
+const tally = globalThis as { routeHarnessBrokenLoads?: number };
+tally.routeHarnessBrokenLoads = (tally.routeHarnessBrokenLoads ?? 0) + 1;
+
+const loadFailure = ((): Error | undefined => new Error('broken script failed to load'))();
+if (loadFailure !== undefined) throw loadFailure;
+
+export default function Broken(_props: ScriptRouteProps) {
+  return null;
+}

packages/agent-bundle/fixtures/route-harness/src/scripts/checksum.ts

@@ -0,0 +1,97 @@
+/**
+ * A plain script with the `main` process-envelope contract: the generated
+ * `scripts/checksum.mjs` awaits `main(process.argv.slice(2))` and adopts a
+ * numeric return as the exit code. No renderer, no request context.
+ */
+
+/** Module state: a fresh process starts at zero, a cached module would not. */
+let calls = 0;
+
+export const main = async (argv: readonly string[]): Promise<number | undefined> => {
+  calls += 1;
+  if (argv.includes('--explode')) {
+    throw new Error('checksum exploded');
+  }
+  if (argv.includes('--calls')) {
+    process.stdout.write(`checksum call ${String(calls)} in ${process.argv[1]!}\n`);
+    return 0;
+  }
+  if (argv.includes('--exit-then-hang')) {
+    // A real process is gone at the exit; the never-settling await after it
+    // (and the write) can only happen if the exit was merely simulated.
+    try {
+      process.exit(6);
+    } catch {
+      process.stdout.write('checksum survived process.exit\n');
+    }
+    await new Promise(() => undefined);
+    return 0;
+  }
+  if (argv.includes('--exec-argv')) {
+    // `node scripts/checksum.mjs` carries no Node flags; neither may this run.
+    process.stdout.write(`checksum execArgv ${JSON.stringify(process.execArgv)}\n`);
+    return 0;
+  }
+  if (argv.includes('--stdin')) {
+    let input = '';
+    for await (const chunk of process.stdin.setEncoding('utf8')) input += chunk as string;
+    process.stdout.write(`checksum read ${String(input.length)} byte(s): ${input.trim()}\n`);
+    return 0;
+  }
+  if (argv.includes('--hang')) {
+    // A script that never finishes on its own; only the harness ending the
+    // process ends this run.
+    setInterval(() => undefined, 1000);
+    await new Promise(() => undefined);
+    return 0;
+  }
+  if (argv.includes('--ignore-sigterm')) {
+    // A script that traps termination and carries on; only a harness that
+    // reaps its process can end this run.
+    process.on('SIGTERM', () => { process.stdout.write('checksum ignored SIGTERM\n'); });
+    process.stdout.write('checksum trapping SIGTERM\n');
+    // Keep the event loop alive; a pending promise alone would let Node exit.
+    setInterval(() => undefined, 1000);
+    await new Promise(() => undefined);
+    return 0;
+  }
+  if (argv.includes('--chdir')) {
+    // Process-level APIs a worker thread refuses; a process of its own has
+    // them, and changing directory there leaves the harness's alone.
+    process.chdir('..');
+    process.stdout.write(`checksum cwd ${process.cwd()}\n`);
+    return 0;
+  }
+  if (argv.includes('--exit-code-property')) {
+    process.stdout.write('checksum set process.exitCode\n');
+    process.exitCode = 4;
+    return undefined;
+  }
+  if (argv.includes('--process-exit')) {
+    process.stdout.write('checksum called process.exit\n');
+    process.exit(5);
+  }
+  if (argv.includes('--swallow-exit')) {
+    // A real process is gone after this call; nothing below can happen there.
+    try {
+      process.exit(3);
+    } catch {
+      process.stdout.write('checksum survived process.exit\n');
+    }
+    return 0;
+  }
+  const returned = argv.find((argument) => argument.startsWith('--return='));
+  if (returned !== undefined) {
+    return Number(returned.slice('--return='.length));
+  }
+  if (argv.includes('--delay')) {
+    await new Promise((resolve) => setTimeout(resolve, 40));
+  }
+  const total = argv.filter((argument) => !argument.startsWith('--')).reduce((sum, argument) => sum + argument.length, 0);
+  process.stdout.write(`Fixture checksum: ${String(total)}\n`);
+  if (total === 0) {
+    process.stderr.write('No arguments to checksum.\n');
+    return 2;
+  }
+  return 0;
+};

packages/agent-bundle/fixtures/route-harness/src/scripts/constant.ts

@@ -0,0 +1,9 @@
+/**
+ * A plain script whose `main` export is not callable. The builder's static
+ * export scan still selects the process envelope, which re-verifies the export
+ * at runtime and throws — so the generated executable evaluates the module,
+ * then always exits 1.
+ */
+process.stdout.write('constant evaluated\n');
+
+export const main = 'not callable';

packages/agent-bundle/fixtures/route-harness/src/scripts/identity.ts

@@ -0,0 +1,11 @@
+import { meta, name, version } from 'agent-bundle/meta';
+
+/**
+ * A plain script that reports the project identity the build stamps into
+ * `agent-bundle/meta`. Outside a compiled surface the published entry throws,
+ * so this only runs where the generated identity module is served.
+ */
+export const main = (): number => {
+  process.stdout.write(`${name}@${version} ${meta.packageName ?? '-'} ${meta.packageVersion ?? '-'}\n`);
+  return 0;
+};