diff --git a/.changeset/script-dispatch-and-workbench-surface.md b/.changeset/script-dispatch-and-workbench-surface.md new file mode 100644 index 000000000..40801e8e0 --- /dev/null +++ b/.changeset/script-dispatch-and-workbench-surface.md @@ -0,0 +1,6 @@ +--- +"agent-bundle": minor +"create-agent-bundle": patch +--- + +Add `runScript`, `scriptJson`, `scriptNdjson`, and `inspectWorkbenchSurface` to `agent-bundle/test`, and move the `cli-tool` template onto the routed CLI. `runScript` (the `script-dispatch` proof level) runs a conventional `src/scripts/*` module through its generated executable's contract: a rendered `.tsx` script through the rendered-script shell with piped Markdown, TTY, `--json`, and `--ndjson` output and the project's conventional `src/providers/*` mounted with the `script` invocation (a `process.exit` in rendered code fails the run as the executable's shell reports its render worker's exit, never ending the test process), a plain `.ts` script as a Node process of its own with the `main` envelope, `process.exit`, exit code, stdout, stderr, optional `stdin`, and the compiled `agent-bundle/meta` identity (no `AB4760` outside a compiled surface); `testManifest().scripts` (a new required member of `AgentBundleTestManifest`, so a hand-built manifest literal must now supply it) lists only the compiled scripts that ship — a nested (`AB4808`) or configuration-conflicting (`AB4809`) conventional script is never a `runScript` target — and every failure names the script route, execution form, and proof level. `inspectWorkbenchSurface` (the `workbench-surface` proof level) returns the route manifest, grouped route catalog, state declaration, lifecycle-replay fixtures, and page availability the Workbench would show for a project, without a browser or dev server, and reports `manifest-unavailable` with the compiler's error diagnostics (for example `AB4100`) for a project the compiler rejects. `ScriptRouteProps` types rendered script components. `create-agent-bundle`'s `cli-tool` template replaces the hand-written `src/cli.ts` with a routed `src/cli/greet.ts` command and a conventional `src/scripts/hello.ts`, proved by a generated projection pool at the `cli-dispatch` and `script-dispatch` levels. (#398) diff --git a/docs/entry-conventions.md b/docs/entry-conventions.md index e8fb66945..f0abda9c6 100644 --- a/docs/entry-conventions.md +++ b/docs/entry-conventions.md @@ -360,7 +360,10 @@ await renderRoute('tool:curator/status', { ``` The same seam accepts `actor`, `workspace`, and `capabilities`; tests can use -`unavailable(...)` to pin a transport's honest absence semantics. +`unavailable(...)` to pin a transport's honest absence semantics. `invokeCli` +(routed commands) and `runScript` (conventional scripts) accept the same +`context` for their rendered surfaces and open the request scope with the +surface-specific `invocation.kind` the generated executable would use. ### Migration nudges diff --git a/packages/agent-bundle/README.md b/packages/agent-bundle/README.md index 665fd767d..cbe6af1a0 100644 --- a/packages/agent-bundle/README.md +++ b/packages/agent-bundle/README.md @@ -423,13 +423,15 @@ kind, and the module provenance. Conventional request context providers (`src/providers/*`, see [entry conventions](../../docs/entry-conventions.md#request-context-providers-power-tier)) are mounted automatically for every manifest-backed helper — `renderRoute`, -`renderRouteEvents`, `invokeCli`, and the in-memory MCP helpers — exactly as the +`renderRouteEvents`, `invokeCli`, `runScript` (rendered scripts), and the +in-memory MCP helpers — exactly as the generated request scopes mount them: discovered from the compiled manifest, executed once per request in the same deterministic key order, handed the same surface-specific `invocation` (`tool`, `event`, `cli`, `script`), and failing the request closed when a factory throws. `providers.processLifetime` is scoped the -way the artifact scopes it: each `invokeCli` call and each `renderRoute` render -is a fresh simulated executable (hit 1, new `instanceId`), while one open +way the artifact scopes it: each `invokeCli` call, each `runScript` run, and +each `renderRoute` render is a fresh simulated executable (hit 1, new +`instanceId`), while one open `openInMemoryMcpServer` session shares a single identity across every request it handles, like the artifact's warm Flight worker. Pass `context.providers` to opt out: an explicit map is mounted verbatim and no conventional provider runs, which is how a test stubs a provider @@ -480,6 +482,8 @@ is never a receipt for another. | `route-unit` | `renderRoute`, `renderRouteEvents` | a route module renders to the document (and render-event stream) it claims | | `mcp-in-memory` | `openInMemoryMcpServer`, `invokeMcpTool`, `readMcpResource`, `getMcpPrompt`, `listMcpSurface`, `runContractMatrix` | the real generated MCP server's protocol contract, over the SDK's in-memory transport | | `cli-dispatch` | `invokeCli`, `cliJson`, `cliNdjson` | a plain or rendered argv vector resolved and run through the routed CLI's own shell, including rendered Markdown, explicit TTY, JSON, and NDJSON modes, in-process | +| `script-dispatch` | `runScript`, `scriptJson`, `scriptNdjson` | a conventional `src/scripts/*` module run through its generated executable's contract: a rendered `.tsx` script through the rendered-script shell in-process (piped Markdown, explicit TTY, `--json`, `--ndjson`), a plain `.ts` script through the `main` process envelope as a Node process of its own over the source — fresh module state, real `process.exit`, its own argv, exit code, and streams — without bundling | +| `workbench-surface` | `inspectWorkbenchSurface`, `workbenchSurfaceFromRouteGraph` | what the dev server would hand the Workbench for this project — the route manifest, the grouped route catalog, the state declaration, lifecycle-replay fixtures per host, and page availability — from the same compiler pass and projection functions, with no browser and no dev server | | `packed-stdio` | `openPackedMcpServer`, `runPackedContractMatrix` | a built artifact's generated entry running as a real process over stdio | | `packed-deleted-source` | `removeProjectSource`, `openPackedMcpServer({ deletedSource })`, `runPackedContractMatrix` | the packed stdio process still runs after project source and configuration are removed and verified absent | | `host-install` | `openInstalledHostMcpServer`, `runInstalledHostContractMatrix` | a built bundle staged into an isolated host root, discovered in the emitted host format, and spawned from the installed layout | @@ -505,6 +509,106 @@ const tty = await invokeCli(['library', 'report', './books'], { tty: true }); expect(tty.stdout).toContain('\r\u001B[2K'); ``` +`runScript` is the same idea for the `src/scripts/*` convention. The manifest +carries every conventional script with its extension contract +(`testManifest().scripts`), and the helper runs the module through what its +generated `scripts/.mjs` would do — never by bundling it: a rendered +`.tsx` script runs in-process through the same shell the executable uses, +and a plain `.ts` script runs as a Node process of its own, as the +executable does (see below): + +```ts +import { runScript, scriptJson, scriptNdjson } from 'agent-bundle/test'; + +// script-dispatch, rendered .tsx script: `--json` / `--ndjson` are reserved by +// the framework, everything else reaches the component's `argv` prop. +const summary = await runScript('summary', ['./books', '--json']); +expect(summary.exitCode).toBe(0); +expect(scriptJson(summary)).toMatchObject({ arguments: ['./books'] }); +expect(scriptNdjson(await runScript('summary', ['./books', '--ndjson'])).at(-1)?.type).toBe('complete'); + +// script-dispatch, plain .ts script exporting main(argv): the envelope adopts +// a numeric return as the exit code; stdout and stderr are captured. +const checksum = await runScript('checksum', ['./books']); +expect(checksum.exitCode).toBe(0); +expect(checksum.stdout).toBe('Fixture checksum: 7\n'); +``` + +A plain script runs as a Node process of its own over the source module, so +the process contract is Node's rather than a simulation of it: every run +evaluates the module afresh (module-level state never survives between runs, +as it never survives between processes), `process.argv` is +`[node, , ...argv]`, `process.exit` ends the script for real — work +queued after it never runs, whether or not the script caught the call — +process-level APIs such as `process.chdir` work and affect only the script, a +numeric `main` return goes through the real `process.exitCode` setter (`300` +reports `44`; `1.5` exits 1 with the setter's `RangeError`), a signal that +ends the process reports as `128 +` its number, and the test process's own +argv, exit code, cwd, and streams are never touched, so plain runs may overlap +each other and any other test. The builder's static export scan decides +between the `main` envelope and a self-executing module, and a non-callable +`main` fails the way the generated executable fails. The process resolves +relative `.js` specifiers to their TypeScript sources, transforms `.ts` with +Node's own type transform, lowers the `.tsx` and `.jsx` helpers a plain +script imports with the bundler's SWC — the same lowering the generated +executable was built with — and serves `agent-bundle/meta` as the identity the build stamps from the +manifest's `plugin`. Explicit `scripts:` configuration entries are bundled +entries rather than routes and stay with the packed level. A rendered script +composes the project's root layout (a script belongs to no server, so no +server layout applies) and mounts the project's conventional providers with the `script` invocation the +generated executable passes (`context.providers` substitutes a fixture map, as +everywhere); a plain script opens no request scope, so it accepts no `context` +at all. A rendered script's declared state mounts on a disposable root for +the run, as at every harness level (`renderRoute`, `invokeCli`): the +`AGENT_BUNDLE_PLUGIN_ROOT` / `.agent-bundle` anchor a `workspace-durable` +store keeps between executable runs is the packed artifact's, and the packed +level proves it; a test that needs one store across several rendered runs +passes the same `context.state` and `context.noticeLedger` bindings to each. +`stdin` pipes input to a plain script (omitted, +it reads end-of-file at once); `process.execArgv` is empty as under plain +`node`; an aborted `signal` sends SIGTERM and, should the script trap it, +kills the process after a one-second grace before the run rejects. A rendered +script's own `console` and stream writes during the run land on the +invocation's `stderr` — the generated executable forwards its render worker's +stdout and stderr there — so `stdout` holds machine output only and nothing +escapes into the test runner. `process.exit` from rendered code is that +worker's exit, never the test process's: the run fails as the executable's +shell reports it (`Generated render worker exited with code N.` on `stderr`, +exit code 1, `0` included), the call unwinds the caller, and whatever code +that catches it writes afterwards is discarded, as a worker that has exited +writes nothing. Once a rendered run's `signal` aborts, no state mount or +module load that has not begun is started on its behalf. Every `ScriptInvocation` carries +`provenance.execution` (`rendered-shell`, `main-envelope`, or +`self-executing`) beside the level. + +`inspectWorkbenchSurface` answers "what would the Workbench show for this +project?" without a browser. It runs the dev server's own preparation as the +Workbench server constructs it — `development` mode for a configuration +factory that branches on `context.mode`, the selected `configPath` for both +the compiler pass and eval-suite discovery — and the same projection functions +the dev server serves — `GET /api/routes/manifest` and `GET /api/lifecycles` +byte for byte — then applies the Workbench's own grouping and navigation +rules: + +```ts +import { inspectWorkbenchSurface, workbenchPageLabel } from 'agent-bundle/test'; + +const surface = await inspectWorkbenchSurface({ root: projectRoot }); +expect(surface.catalog.groups.map((group) => group.label)).toContain('curator · Tools'); +expect(surface.catalog.stateDefinition).toMatchObject({ driver: 'sqlite', lifetime: 'workspace-durable' }); +expect(surface.lifecycles[0]?.targets.map((target) => target.target)).toEqual(['claude', 'codex']); +expect(surface.pages.map(workbenchPageLabel)).not.toContain('Playground'); +``` + +`counts` are the artifact inventory the Workbench counts, derived without a +build: one instance per hook, MCP server, or script declaration per selected +target it names (a declaration whose `targets` select none of the project's +targets is emitted nowhere and counts nothing), plus the declared Skills, eval +suites, and targets. Page availability depends only on whether each count is +zero and on what the compiled graph declares. Host discovery, live MCP probes, published epochs, +and the RSC runtime page are artifact- or process-bound and are not projected +here. + `expectEvents` asserts over a render-event stream. `toContainSequence` is sequence-tolerant — an extra `progress` or `replace` frame is legal and cannot turn a passing render red — while a missing frame, a reordering, or a regressed diff --git a/packages/agent-bundle/fixtures/route-harness/src/badge.tsx b/packages/agent-bundle/fixtures/route-harness/src/badge.tsx new file mode 100644 index 000000000..6d54727b3 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/badge.tsx @@ -0,0 +1,6 @@ +import type { ReactElement } from 'react'; + +/** A JSX helper shared by the fixture's plain `badge` script; not a route. */ +export const Badge = ({ label }: { readonly label: string }): ReactElement => ( + {label} +); diff --git a/packages/agent-bundle/fixtures/route-harness/src/ribbon.d.ts b/packages/agent-bundle/fixtures/route-harness/src/ribbon.d.ts new file mode 100644 index 000000000..325d1055e --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/ribbon.d.ts @@ -0,0 +1,3 @@ +import type { ReactElement } from 'react'; + +export declare const Ribbon: (props: { readonly label: string }) => ReactElement; diff --git a/packages/agent-bundle/fixtures/route-harness/src/ribbon.jsx b/packages/agent-bundle/fixtures/route-harness/src/ribbon.jsx new file mode 100644 index 000000000..207773e06 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/ribbon.jsx @@ -0,0 +1,6 @@ +/** + * A JavaScript JSX helper (`.jsx`, imported with its extension) shared by the + * fixture's plain `badge` script; not a route. The bundler lowers it through + * the React plugin, as it does the `.tsx` helper beside it. + */ +export const Ribbon = ({ label }) => {label}; diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/badge.ts b/packages/agent-bundle/fixtures/route-harness/src/scripts/badge.ts new file mode 100644 index 000000000..4838b4e75 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/badge.ts @@ -0,0 +1,17 @@ +import { Badge } from '../badge.js'; +import { Ribbon } from '../ribbon.jsx'; + +/** + * A plain script that imports a `.tsx` helper — and, with `--ribbon`, a + * JavaScript `.jsx` one — the way a project shares presentational pieces + * between its rendered and plain scripts: the bundler lowers the JSX for the + * generated executable, and the harness must do the same for the source it + * runs. + */ +export const main = (argv: readonly string[]): number => { + const label = argv.filter((argument) => !argument.startsWith('--')).join(' ') || 'unlabelled'; + const element = argv.includes('--ribbon') ? Ribbon({ label }) : Badge({ label }); + const { children, className } = element.props as { readonly children: string; readonly className: string }; + process.stdout.write(`<${String(element.type)} class="${className}">${children}\n`); + return 0; +}; diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/banner.ts b/packages/agent-bundle/fixtures/route-harness/src/scripts/banner.ts new file mode 100644 index 000000000..5040d216f --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/banner.ts @@ -0,0 +1,5 @@ +/** + * A self-executing plain script: no `main` export, so the artifact bundles the + * module as-is and its top-level code runs when the process evaluates it. + */ +process.stdout.write(`banner: ${process.argv.slice(2).join(' ')}\n`); diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/blank.tsx b/packages/agent-bundle/fixtures/route-harness/src/scripts/blank.tsx new file mode 100644 index 000000000..c9b1bbd8f --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/blank.tsx @@ -0,0 +1,6 @@ +/** + * A rendered script module that evaluates but exports no component: the + * generated executable's render worker reports the shape failure through its + * event stream, so the process writes the failure to stderr and exits 1. + */ +export const notAComponent = true; diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/broken.tsx b/packages/agent-bundle/fixtures/route-harness/src/scripts/broken.tsx new file mode 100644 index 000000000..a7a792c3f --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/broken.tsx @@ -0,0 +1,16 @@ +import type { ScriptRouteProps } from 'agent-bundle'; + +/** + * A rendered script whose module fails to evaluate. The generated executable + * loads it inside its render worker and the failure reaches the shell as an + * event-stream error: stderr carries the message and the process exits 1. + */ +const tally = globalThis as { routeHarnessBrokenLoads?: number }; +tally.routeHarnessBrokenLoads = (tally.routeHarnessBrokenLoads ?? 0) + 1; + +const loadFailure = ((): Error | undefined => new Error('broken script failed to load'))(); +if (loadFailure !== undefined) throw loadFailure; + +export default function Broken(_props: ScriptRouteProps) { + return null; +} diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/checksum.ts b/packages/agent-bundle/fixtures/route-harness/src/scripts/checksum.ts new file mode 100644 index 000000000..ee13e3efb --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/checksum.ts @@ -0,0 +1,97 @@ +/** + * A plain script with the `main` process-envelope contract: the generated + * `scripts/checksum.mjs` awaits `main(process.argv.slice(2))` and adopts a + * numeric return as the exit code. No renderer, no request context. + */ + +/** Module state: a fresh process starts at zero, a cached module would not. */ +let calls = 0; + +export const main = async (argv: readonly string[]): Promise => { + calls += 1; + if (argv.includes('--explode')) { + throw new Error('checksum exploded'); + } + if (argv.includes('--calls')) { + process.stdout.write(`checksum call ${String(calls)} in ${process.argv[1]!}\n`); + return 0; + } + if (argv.includes('--exit-then-hang')) { + // A real process is gone at the exit; the never-settling await after it + // (and the write) can only happen if the exit was merely simulated. + try { + process.exit(6); + } catch { + process.stdout.write('checksum survived process.exit\n'); + } + await new Promise(() => undefined); + return 0; + } + if (argv.includes('--exec-argv')) { + // `node scripts/checksum.mjs` carries no Node flags; neither may this run. + process.stdout.write(`checksum execArgv ${JSON.stringify(process.execArgv)}\n`); + return 0; + } + if (argv.includes('--stdin')) { + let input = ''; + for await (const chunk of process.stdin.setEncoding('utf8')) input += chunk as string; + process.stdout.write(`checksum read ${String(input.length)} byte(s): ${input.trim()}\n`); + return 0; + } + if (argv.includes('--hang')) { + // A script that never finishes on its own; only the harness ending the + // process ends this run. + setInterval(() => undefined, 1000); + await new Promise(() => undefined); + return 0; + } + if (argv.includes('--ignore-sigterm')) { + // A script that traps termination and carries on; only a harness that + // reaps its process can end this run. + process.on('SIGTERM', () => { process.stdout.write('checksum ignored SIGTERM\n'); }); + process.stdout.write('checksum trapping SIGTERM\n'); + // Keep the event loop alive; a pending promise alone would let Node exit. + setInterval(() => undefined, 1000); + await new Promise(() => undefined); + return 0; + } + if (argv.includes('--chdir')) { + // Process-level APIs a worker thread refuses; a process of its own has + // them, and changing directory there leaves the harness's alone. + process.chdir('..'); + process.stdout.write(`checksum cwd ${process.cwd()}\n`); + return 0; + } + if (argv.includes('--exit-code-property')) { + process.stdout.write('checksum set process.exitCode\n'); + process.exitCode = 4; + return undefined; + } + if (argv.includes('--process-exit')) { + process.stdout.write('checksum called process.exit\n'); + process.exit(5); + } + if (argv.includes('--swallow-exit')) { + // A real process is gone after this call; nothing below can happen there. + try { + process.exit(3); + } catch { + process.stdout.write('checksum survived process.exit\n'); + } + return 0; + } + const returned = argv.find((argument) => argument.startsWith('--return=')); + if (returned !== undefined) { + return Number(returned.slice('--return='.length)); + } + if (argv.includes('--delay')) { + await new Promise((resolve) => setTimeout(resolve, 40)); + } + const total = argv.filter((argument) => !argument.startsWith('--')).reduce((sum, argument) => sum + argument.length, 0); + process.stdout.write(`Fixture checksum: ${String(total)}\n`); + if (total === 0) { + process.stderr.write('No arguments to checksum.\n'); + return 2; + } + return 0; +}; diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/constant.ts b/packages/agent-bundle/fixtures/route-harness/src/scripts/constant.ts new file mode 100644 index 000000000..3c33685c4 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/constant.ts @@ -0,0 +1,9 @@ +/** + * A plain script whose `main` export is not callable. The builder's static + * export scan still selects the process envelope, which re-verifies the export + * at runtime and throws — so the generated executable evaluates the module, + * then always exits 1. + */ +process.stdout.write('constant evaluated\n'); + +export const main = 'not callable'; diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/identity.ts b/packages/agent-bundle/fixtures/route-harness/src/scripts/identity.ts new file mode 100644 index 000000000..2dd7b19f1 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/identity.ts @@ -0,0 +1,11 @@ +import { meta, name, version } from 'agent-bundle/meta'; + +/** + * A plain script that reports the project identity the build stamps into + * `agent-bundle/meta`. Outside a compiled surface the published entry throws, + * so this only runs where the generated identity module is served. + */ +export const main = (): number => { + process.stdout.write(`${name}@${version} ${meta.packageName ?? '-'} ${meta.packageVersion ?? '-'}\n`); + return 0; +}; diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/stalled.tsx b/packages/agent-bundle/fixtures/route-harness/src/scripts/stalled.tsx new file mode 100644 index 000000000..10344cab2 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/stalled.tsx @@ -0,0 +1,13 @@ +import type { ScriptRouteProps } from 'agent-bundle'; + +/** + * A rendered script whose module never finishes evaluating. The generated + * executable loads it inside its render worker; only an abort — SIGINT or + * SIGTERM failing the parent stream and terminating the worker — ends the + * run, so the harness must honor an abort while the module is still loading. + */ +await new Promise(() => undefined); + +export default function Stalled(_props: ScriptRouteProps) { + return null; +} diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/summary.tsx b/packages/agent-bundle/fixtures/route-harness/src/scripts/summary.tsx new file mode 100644 index 000000000..f0ebf569e --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/summary.tsx @@ -0,0 +1,74 @@ +import { Agent, agent } from '@agent-bundle/runtime'; +import type { ScriptRouteProps } from 'agent-bundle'; + +// Evaluation is observable: harness tests prove the module is not loaded for +// a run the shell rejected, nor once a run has reported its cancellation. +const tally = globalThis as { routeHarnessSummaryLoads?: number }; +tally.routeHarnessSummaryLoads = (tally.routeHarnessSummaryLoads ?? 0) + 1; + +/** + * A rendered script: `argv` is whatever remained after the framework reserved + * `--json` / `--ndjson`. `--fail` renders a represented error document, and + * `--explode` throws, so the harness can prove both exit paths. + */ +export default async function Summary({ argv, signal }: ScriptRouteProps) { + const context = await agent(); + await context.progress.report({ completed: 1, message: 'collecting arguments', total: 2 }); + if (argv.includes('--explode')) { + throw new Error('summary render exploded'); + } + if (argv.includes('--log')) { + // Diagnostic output from a rendered script: the generated executable + // forwards the render worker's stdout and stderr onto its own stderr, so + // machine output on stdout stays clean. + console.log('summary log line'); + process.stdout.write('summary stdout line\n'); + process.stderr.write('summary stderr line\n'); + } + if (argv.includes('--wait-for-abort')) { + await new Promise((_resolve, reject) => { + const rejectAborted = () => reject(new DOMException('Summary render aborted', 'AbortError')); + if (signal.aborted) { + rejectAborted(); + return; + } + signal.addEventListener('abort', rejectAborted, { once: true }); + }); + } + const exitFlag = argv.find((argument) => argument.startsWith('--exit=')); + if (exitFlag !== undefined) { + // `process.exit` from rendered code: in the generated executable this ends + // the render worker, and the shell reports the exit. `--catch-exit` swallows + // the call and carries on, as careless code might. + const code = Number(exitFlag.slice('--exit='.length)); + if (argv.includes('--catch-exit')) { + try { + process.exit(code); + } catch { + console.log('summary carried on after process.exit'); + } + } else { + process.exit(code); + } + } + await context.progress.report({ completed: 2, message: 'summary ready', total: 2 }); + const value = { + arguments: [...argv], + invocation: context.invocation.kind, + stateMounted: context.state !== undefined, + surface: context.invocation.surface ?? null, + }; + if (argv.includes('--fail')) { + return ( + + The summary was asked to fail. + + ); + } + return ( + + {`# Summary\n\n${String(argv.length)} argument(s).`} + {`surface: ${String(context.invocation.surface)}`} + + ); +} diff --git a/packages/agent-bundle/fixtures/route-harness/src/state.ts b/packages/agent-bundle/fixtures/route-harness/src/state.ts index 53aadbb9b..2e09d77e1 100644 --- a/packages/agent-bundle/fixtures/route-harness/src/state.ts +++ b/packages/agent-bundle/fixtures/route-harness/src/state.ts @@ -1,6 +1,11 @@ import { defineState } from '@agent-bundle/runtime/state'; import { z } from 'zod'; +// Evaluation is observable: harness tests prove the state module is not loaded +// before a rendered-script shell has accepted its argv. +const tally = globalThis as { routeHarnessStateLoads?: number }; +tally.routeHarnessStateLoads = (tally.routeHarnessStateLoads ?? 0) + 1; + const journalEntrySchema = z.object({ note: z.string(), }).strict(); diff --git a/packages/agent-bundle/src/index.ts b/packages/agent-bundle/src/index.ts index 71c53605c..683b8fc65 100644 --- a/packages/agent-bundle/src/index.ts +++ b/packages/agent-bundle/src/index.ts @@ -44,6 +44,7 @@ export type { RouteSchema, RouteSchemaOutput, RouteUiMeta, + ScriptRouteProps, ToolConfig, ToolRouteProps, } from './routes/public.ts'; diff --git a/packages/agent-bundle/src/routes/public.ts b/packages/agent-bundle/src/routes/public.ts index d84d4cff7..58f8da9d2 100644 --- a/packages/agent-bundle/src/routes/public.ts +++ b/packages/agent-bundle/src/routes/public.ts @@ -260,3 +260,14 @@ export interface CliRouteProps { readonly input: RouteSchemaOutput; readonly signal: AbortSignal; } + +/** + * Props received by a rendered script's (`src/scripts/.tsx`) async + * default Server Component: the argv left after the framework reserved + * `--json` / `--ndjson`, exactly as the generated executable passes it, and + * the request abort signal. + */ +export interface ScriptRouteProps { + readonly argv: readonly string[]; + readonly signal: AbortSignal; +} diff --git a/packages/agent-bundle/src/test/cli.ts b/packages/agent-bundle/src/test/cli.ts index b7bbe39f7..514c7dcaa 100644 --- a/packages/agent-bundle/src/test/cli.ts +++ b/packages/agent-bundle/src/test/cli.ts @@ -25,6 +25,7 @@ import { createProviderProcessLifetime } from '../routes/provider-execution.ts'; import type { CompiledCliCommand } from '../routes/types.ts'; import { AgentTestError, captured } from './errors.ts'; import { CLI_DISPATCH_PROOF_LEVEL, type AgentBundleTestManifest } from './manifest.ts'; +import { parseCanonicalJsonLine, parseRenderedEventLines } from './output-modes.ts'; import { claimProcessHit, mountProviders } from './providers.ts'; import { registeredRouteLoader, testManifest } from './registry.ts'; import { prepareCliRenderHost, type HarnessOptionsArguments, type RenderRouteContextInit } from './render.ts'; @@ -303,7 +304,7 @@ export const invokeCli = async ( /** The parsed canonical JSON line a successful command wrote to stdout. */ export const cliJson = (invocation: CliInvocation): unknown => { try { - return JSON.parse(invocation.stdout) as unknown; + return parseCanonicalJsonLine(invocation.stdout); } catch (error) { throw new AgentTestError('projection-failed', 'The dispatched command did not write one canonical JSON line to stdout.', { cause: error, @@ -327,33 +328,7 @@ export const cliJson = (invocation: CliInvocation): unknown => { /** The ordered render events a successful `--ndjson` invocation wrote to stdout. */ export const cliNdjson = (invocation: CliInvocation): readonly CliRenderedEvent[] => { try { - const lines = invocation.stdout.endsWith('\n') - ? invocation.stdout.slice(0, -1).split('\n') - : invocation.stdout.split('\n'); - if (lines.length === 0 || lines.some((line) => line.trim() === '')) { - throw new SyntaxError('NDJSON output must contain one non-empty JSON object per line.'); - } - return Object.freeze(lines.map((line) => { - const event = JSON.parse(line) as unknown; - if (typeof event !== 'object' || event === null || Array.isArray(event)) { - throw new SyntaxError('NDJSON output lines must be JSON objects.'); - } - const record = event as Record; - if (!Number.isInteger(record['sequence'])) { - throw new SyntaxError('NDJSON render events must carry an integer sequence.'); - } - switch (record['type']) { - case 'shell': - case 'progress': - case 'replace': - case 'error': - case 'complete': - break; - default: - throw new SyntaxError('NDJSON output contains an unknown render-event type.'); - } - return event as CliRenderedEvent; - })); + return parseRenderedEventLines(invocation.stdout); } catch (error) { throw new AgentTestError('projection-failed', 'The dispatched command did not write one JSON object per line to stdout.', { cause: error, diff --git a/packages/agent-bundle/src/test/errors.ts b/packages/agent-bundle/src/test/errors.ts index d8a6daf5b..0586f8942 100644 --- a/packages/agent-bundle/src/test/errors.ts +++ b/packages/agent-bundle/src/test/errors.ts @@ -14,6 +14,7 @@ export type AgentTestErrorCode = | 'render-failed' | 'result-rejected' | 'route-not-found' + | 'script-not-found' | 'server-not-found' | 'unsupported-rich-content' | 'unsupported-route-kind'; diff --git a/packages/agent-bundle/src/test/index.ts b/packages/agent-bundle/src/test/index.ts index b80c01879..df295a33d 100644 --- a/packages/agent-bundle/src/test/index.ts +++ b/packages/agent-bundle/src/test/index.ts @@ -1,7 +1,7 @@ /** * `agent-bundle/test` — the consumer test harness helpers. * - * Eight Node proof levels ship here, and the browser-safe ninth level ships + * Ten Node proof levels ship here, and the browser-safe eleventh level ships * from `agent-bundle/test/browser`. The repository's real-host install proof * uses the same level convention. Each helper names the level it supplies, * stamps it into its provenance, and prints it in every failure: @@ -12,6 +12,8 @@ * | `mcp-in-memory` | `openInMemoryMcpServer`, `invokeMcpTool`, `readMcpResource`, `getMcpPrompt`, `listMcpSurface`, `runContractMatrix` | the real generated MCP server's protocol contract, over the SDK's in-memory transport; MCP App routes are not registered and report `not-applicable` | * | `dev-epoch` | `runDevEpochContractMatrix` | an epoch-pinned generated stdio process opened through the Workbench session service; MCP App routes are covered (surface + `ui://` sweep) and auto-covered without a fixture | * | `cli-dispatch` | `invokeCli`, `cliJson`, `cliNdjson` | a compiled plain or rendered CLI command dispatched through the routed CLI's own shell, including rendered output modes, in this process | + * | `script-dispatch` | `runScript`, `scriptJson`, `scriptNdjson` | a conventional `src/scripts/*` module run through its generated executable's contract — the rendered-script shell with its four output modes in this process, or the plain `main` envelope as a Node process of its own over the source — without bundling | + * | `workbench-surface` | `inspectWorkbenchSurface` | the compiled route graph projected exactly as the dev server serves it to the Workbench: route catalog, state, lifecycle fixtures, page availability, without a browser or dev server | * | `packed-stdio` | `openPackedMcpServer`, `runPackedContractMatrix` | a built artifact's generated entry running as a real process over stdio; MCP App routes are covered (surface + `ui://` sweep) and auto-covered without a fixture | * | `packed-deleted-source` | `removeProjectSource`, `openPackedMcpServer({ deletedSource })`, `runPackedContractMatrix` | the packed stdio process still runs after project source and configuration are removed and verified absent; MCP App routes are covered as at `packed-stdio` | * | `browser-app` | `mountBrowserApp` (`agent-bundle/test/browser`) | production-compiled MCP App HTML mounted over the product bridge in a real browser page | @@ -36,7 +38,9 @@ export { PACKED_DELETED_SOURCE_PROOF_LEVEL, PACKED_STDIO_PROOF_LEVEL, ROUTE_UNIT_PROOF_LEVEL, + SCRIPT_DISPATCH_PROOF_LEVEL, SIMULATED_PROOF_LEVEL, + WORKBENCH_SURFACE_PROOF_LEVEL, compileTestManifest, proofLevelLabel, testManifestFromRouteGraph, @@ -50,6 +54,7 @@ export type { TestableProviderDescriptor, TestableLayoutDescriptor, TestableRouteDescriptor, + TestableScriptDescriptor, TestableStateDescriptor, } from './manifest.ts'; export { AGENT_TEST_REGISTRY_VERSION, registerTestRoutes, testManifest } from './registry.ts'; @@ -144,6 +149,34 @@ export type { } from './mcp.ts'; export { cliJson, cliNdjson, invokeCli } from './cli.ts'; export type { CliDispatchProvenance, CliInvocation, CliRenderedEvent, InvokeCliOptions, InvokeCliOptionsBase } from './cli.ts'; +export { runScript, scriptJson, scriptNdjson } from './script.ts'; +export type { + RunScriptOptions, + RunScriptOptionsBase, + ScriptDispatchProvenance, + ScriptExecution, + ScriptInvocation, +} from './script.ts'; +export { + inspectWorkbenchSurface, + workbenchCommandUsage, + workbenchPageLabel, + workbenchPagesFor, + workbenchRouteCatalog, + workbenchSurfaceFromRouteGraph, +} from './workbench.ts'; +export type { + InspectWorkbenchSurfaceOptions, + WorkbenchCapabilityCounts, + WorkbenchPageName, + WorkbenchRouteCatalog, + WorkbenchRouteCatalogEntry, + WorkbenchRouteCatalogGroup, + WorkbenchRouteCatalogServer, + WorkbenchSurface, + WorkbenchSurfaceFromGraphInput, + WorkbenchSurfaceProvenance, +} from './workbench.ts'; export { openPackedMcpServer, removeProjectSource } from './packed.ts'; export type { DeletedSourceReceipt, diff --git a/packages/agent-bundle/src/test/manifest.ts b/packages/agent-bundle/src/test/manifest.ts index 2bacfaaf7..17a1f1be1 100644 --- a/packages/agent-bundle/src/test/manifest.ts +++ b/packages/agent-bundle/src/test/manifest.ts @@ -1,9 +1,10 @@ import { relative, resolve, sep } from 'node:path'; +import { isRenderedScriptRoute, judgeScriptRoute, scriptRouteName } from '../config/script-routes.ts'; import type { Diagnostic } from '../core/diagnostics.ts'; import { stableJson } from '../core/digest.ts'; import { deepFreeze } from '../core/freeze.ts'; -import type { NormalizedMcpApp, NormalizedStateDefinition } from '../core/types.ts'; +import type { NormalizedMcpApp, NormalizedScript, NormalizedStateDefinition } from '../core/types.ts'; import { orderedProviders } from '../routes/provider-execution.ts'; import { providerKeyFromName } from '../routes/providers.ts'; import type { @@ -33,6 +34,18 @@ import type { * - `cli-dispatch` runs an argv vector through the routed CLI's own shell over * the compiled command graph, in this process. It proves command * resolution, argv projection, and exit codes, not a spawned binary. + * - `script-dispatch` runs one conventional `src/scripts/*` module through the + * contract its generated `scripts/.mjs` executable carries: a rendered + * `.tsx` script through the rendered-script shell and its four output modes + * in this process, a plain `.ts` script as a Node child process of its own + * over the source module with the `main` envelope. It proves the script's + * behavior and output contract — process exit, streams, and signals + * included for a plain script — not the bundled artifact. + * - `workbench-surface` projects the compiled route graph the way the dev + * server serves it to the Workbench — route catalog, state, lifecycle + * fixtures, page availability — without a browser or a dev server. It + * proves what the Workbench would be given, not that the Workbench rendered + * it. * - `packed-stdio` installs the packed release tarball into a clean consumer, * spawns the generated stdio entry as a real process, and drives it with a * real MCP client. This is the packed process-and-protocol evidence level. @@ -54,6 +67,8 @@ export type AgentTestProofLevel = | 'mcp-in-memory' | 'dev-epoch' | 'cli-dispatch' + | 'script-dispatch' + | 'workbench-surface' | 'packed-stdio' | 'packed-deleted-source' | 'browser-app' @@ -64,6 +79,8 @@ export const ROUTE_UNIT_PROOF_LEVEL = 'route-unit' as const; export const MCP_IN_MEMORY_PROOF_LEVEL = 'mcp-in-memory' as const; export const DEV_EPOCH_PROOF_LEVEL = 'dev-epoch' as const; export const CLI_DISPATCH_PROOF_LEVEL = 'cli-dispatch' as const; +export const SCRIPT_DISPATCH_PROOF_LEVEL = 'script-dispatch' as const; +export const WORKBENCH_SURFACE_PROOF_LEVEL = 'workbench-surface' as const; export const PACKED_STDIO_PROOF_LEVEL = 'packed-stdio' as const; export const PACKED_DELETED_SOURCE_PROOF_LEVEL = 'packed-deleted-source' as const; export const BROWSER_APP_PROOF_LEVEL = 'browser-app' as const; @@ -85,6 +102,10 @@ export const proofLevelLabel = (level: AgentTestProofLevel): string => { return 'dev-epoch (epoch-pinned generated stdio entry spawned as a real process through the Workbench session service; NOT packed or native-host evidence)'; case 'cli-dispatch': return 'cli-dispatch (argv dispatched through the routed CLI shell in-process; NOT a spawned binary)'; + case 'script-dispatch': + return 'script-dispatch (conventional script run through its generated executable contract — rendered-script shell in-process, or plain main envelope as a Node process over the source; NOT the bundled scripts/.mjs artifact)'; + case 'workbench-surface': + return 'workbench-surface (compiled route graph projected exactly as the dev server serves it to the Workbench; NOT a browser, dev-server, or built-artifact receipt)'; case 'packed-stdio': return 'packed-stdio (packed tarball installed into a clean consumer, generated stdio entry spawned as a real process)'; case 'packed-deleted-source': @@ -162,6 +183,24 @@ export interface TestableLayoutDescriptor { readonly source: string; } +/** + * One conventional `src/scripts/` module the script-dispatch level can + * run. `rendered` is the extension contract (#102 stage 3): `.tsx`/`.jsx` + * scripts render through the Agent renderer, everything else is a plain + * executable module. Explicit `scripts:` configuration entries are bundled + * entries rather than routes, so they never appear here. + */ +export interface TestableScriptDescriptor { + /** The path-derived script name (`script:verify` -> `verify`). */ + readonly name: string; + /** Project-relative POSIX path of the script module. */ + readonly relativePath: string; + readonly rendered: boolean; + readonly routeId: string; + /** Absolute script module path. */ + readonly source: string; +} + /** The conventional state module the generated route-unit registry can load. */ export interface TestableStateDescriptor { readonly id: string; @@ -241,6 +280,13 @@ export interface AgentBundleTestManifest { */ readonly providers?: readonly TestableProviderDescriptor[]; readonly routes: Readonly>; + /** + * The conventional `src/scripts/*` modules from the same pass, collision- + * checked by name, so the script-dispatch level runs the product's own + * script contract instead of guessing at file extensions. Empty when the + * project compiles no conventional scripts. + */ + readonly scripts: readonly TestableScriptDescriptor[]; /** Conventional project state mounted automatically for manifest route renders. */ readonly state?: TestableStateDescriptor; /** Host targets the project selected. Route-unit rendering is target-neutral; these name the projection surfaces a later proof level owns. */ @@ -323,6 +369,59 @@ const appDescriptors = ( return descriptors; }; +/** + * The script names explicit `scripts:` configuration claims. Normalization + * carries them as config-provenance scripts; a conventional route that + * collides with one is an `AB4809` error and never ships. + */ +const configuredScriptNames = (scripts: readonly NormalizedScript[]): ReadonlySet => + new Set(scripts.filter((script) => script.provenance.kind === 'config').map((script) => script.name)); + +/** + * Only the conventional script routes normalization ships become + * `scripts/.mjs` executables. The same #102 judgment gates this + * inventory, so `runScript` can never carry a `script-dispatch` proof for a + * nested (`AB4808`) or configuration-conflicting (`AB4809`) route whose + * executable cannot exist. + */ +const scriptDescriptors = ( + graph: CompiledRouteGraph, + configured: ReadonlySet, +): readonly TestableScriptDescriptor[] => { + const seen = new Map(); + return graph.scripts.flatMap((route): TestableScriptDescriptor[] => { + const judgment = judgeScriptRoute(route, configured); + switch (judgment) { + case 'nested': + case 'conflicting': + return []; + case 'rendered': + case 'shippable': + break; + default: { + const exhaustive: never = judgment; + return exhaustive; + } + } + const name = scriptRouteName(route); + const existing = seen.get(name); + if (existing !== undefined) { + throw new Error( + `Duplicate compiled script name ${JSON.stringify(name)}: ${existing} and ${route.provenance.relativePath} ` + + 'would both compile to the same scripts/.mjs executable.', + ); + } + seen.set(name, route.provenance.relativePath); + return [{ + name, + relativePath: route.provenance.relativePath, + rendered: isRenderedScriptRoute(route), + routeId: route.id, + source: route.source, + }]; + }); +}; + /** * Projects the compiled route graph into the manifest the harness addresses. * The graph is an input here, never recompiled: one compiler pass feeds the @@ -335,6 +434,8 @@ export const testManifestFromRouteGraph = (input: { readonly graph: CompiledRouteGraph; readonly plugin?: TestManifestPluginIdentity; readonly projectRoot: string; + /** The normalized script inventory; its config-provenance entries decide which conventional routes conflict. */ + readonly scripts?: readonly NormalizedScript[]; readonly state?: NormalizedStateDefinition; readonly targets?: readonly string[]; }): AgentBundleTestManifest => { @@ -364,6 +465,7 @@ export const testManifestFromRouteGraph = (input: { proofLevel: ROUTE_UNIT_PROOF_LEVEL, ...(input.graph.providers.length === 0 ? {} : { providers: providerDescriptors(input.graph.providers) }), routes, + scripts: scriptDescriptors(input.graph, configuredScriptNames(input.scripts ?? [])), ...(input.state === undefined ? {} : { @@ -416,6 +518,7 @@ export const compileTestManifest = async ( }, }), projectRoot: prepared.root, + scripts: prepared.model?.scripts ?? [], ...(prepared.model?.state === undefined ? {} : { state: prepared.model.state }), targets: prepared.model?.targets.map((target) => target.name) ?? [], }); diff --git a/packages/agent-bundle/src/test/output-modes.ts b/packages/agent-bundle/src/test/output-modes.ts new file mode 100644 index 000000000..b1454870d --- /dev/null +++ b/packages/agent-bundle/src/test/output-modes.ts @@ -0,0 +1,41 @@ +import type { CliRenderedEvent } from '../cli-entry.ts'; + +/** + * Parsers for the two machine output modes the routed CLI shell and the + * rendered-script shell share (`--json`, `--ndjson`). They throw a plain + * `SyntaxError`; the dispatch levels wrap that into an `AgentTestError` + * carrying their own provenance, so one parser serves both without either + * level borrowing the other's failure identity. + */ + +/** The parsed canonical JSON line a successful `--json` invocation wrote to stdout. */ +export const parseCanonicalJsonLine = (stdout: string): unknown => JSON.parse(stdout) as unknown; + +/** The ordered render events a successful `--ndjson` invocation wrote to stdout. */ +export const parseRenderedEventLines = (stdout: string): readonly CliRenderedEvent[] => { + const lines = stdout.endsWith('\n') ? stdout.slice(0, -1).split('\n') : stdout.split('\n'); + if (lines.length === 0 || lines.some((line) => line.trim() === '')) { + throw new SyntaxError('NDJSON output must contain one non-empty JSON object per line.'); + } + return Object.freeze(lines.map((line) => { + const event = JSON.parse(line) as unknown; + if (typeof event !== 'object' || event === null || Array.isArray(event)) { + throw new SyntaxError('NDJSON output lines must be JSON objects.'); + } + const record = event as Record; + if (!Number.isInteger(record['sequence'])) { + throw new SyntaxError('NDJSON render events must carry an integer sequence.'); + } + switch (record['type']) { + case 'shell': + case 'progress': + case 'replace': + case 'error': + case 'complete': + break; + default: + throw new SyntaxError('NDJSON output contains an unknown render-event type.'); + } + return event as CliRenderedEvent; + })); +}; diff --git a/packages/agent-bundle/src/test/registry.ts b/packages/agent-bundle/src/test/registry.ts index 0f7da6ef8..55ed8ea80 100644 --- a/packages/agent-bundle/src/test/registry.ts +++ b/packages/agent-bundle/src/test/registry.ts @@ -21,8 +21,9 @@ const REGISTRY_SYMBOL = Symbol.for(AGENT_TEST_REGISTRY_SYMBOL_KEY); * helpers reading it never silently disagree about what the registry carries. * 4: `providerLoaders` (conventional context providers mounted by the harness). * 5: `layoutLoaders` (conventional layouts composed around manifest renders). + * 6: `manifest.scripts` (the script-dispatch level's inventory). */ -export const AGENT_TEST_REGISTRY_VERSION = 5; +export const AGENT_TEST_REGISTRY_VERSION = 6; export type AgentStateModuleLoader = () => Promise<{ readonly default: AgentStateDefinition; diff --git a/packages/agent-bundle/src/test/render.ts b/packages/agent-bundle/src/test/render.ts index 4eda7ce86..1a9d0b5ae 100644 --- a/packages/agent-bundle/src/test/render.ts +++ b/packages/agent-bundle/src/test/render.ts @@ -37,6 +37,7 @@ import { registeredRouteLoader, registeredStateLoader, testManifest, + type AgentStateModuleLoader, } from './registry.ts'; import type { AgentRouteModule, @@ -522,23 +523,25 @@ const noMountedState: AutoMountedState = Object.freeze({ close: async () => undefined, }); +type StateMount = (renderer: Renderer, signal: AbortSignal) => Promise; + /** - * Mounts one fresh state owner for a manifest render. Durable definitions use - * a disposable sqlite root so repeated route-unit renders are deterministic. + * Resolves how a manifest render mounts its state, without mounting it: the + * loader lookup is harness wiring and fails here, while loading the state + * module and opening its driver — user code and a filesystem root — wait for + * the returned mount to be called. */ -const mountManifestState = async ( +const manifestStateMount = ( manifest: AgentBundleTestManifest | undefined, provenance: RenderedRouteProvenance, context: RenderRouteContext, - renderer: Renderer, - signal: AbortSignal, -): Promise => { +): StateMount => { const descriptor = manifest?.state; if ( manifest === undefined || descriptor === undefined || (context.state !== undefined && context.noticeLedger !== undefined) - ) return noMountedState; + ) return async () => noMountedState; const loader = registeredStateLoader(manifest); if (loader === undefined) { throw new AgentTestError( @@ -550,6 +553,28 @@ const mountManifestState = async ( }, ); } + return (renderer, signal) => mountState(descriptor, loader, context, renderer, signal); +}; + +/** + * Mounts one fresh state owner for a manifest render. Durable definitions use + * a disposable sqlite root so repeated route-unit renders are deterministic. + */ +const mountManifestState = async ( + manifest: AgentBundleTestManifest | undefined, + provenance: RenderedRouteProvenance, + context: RenderRouteContext, + renderer: Renderer, + signal: AbortSignal, +): Promise => manifestStateMount(manifest, provenance, context)(renderer, signal); + +const mountState = async ( + descriptor: NonNullable, + loader: AgentStateModuleLoader, + context: RenderRouteContext, + renderer: Renderer, + signal: AbortSignal, +): Promise => { const definition = (await loader()).default; let root: string | undefined; let driver: AgentState.AgentStateDriver; @@ -797,6 +822,216 @@ export const prepareCliRenderHost = async ( }); }; +export interface PrepareScriptRenderHostOptions { + readonly context?: RenderRouteContext; + /** + * Loads the script module. It is called only when the shell opens a + * session — after its own argv checks — and its failure, like a module + * without a default component, reaches the shell through the event stream, + * exactly as the generated executable's render worker reports it. + */ + readonly loadModule: () => Promise; + readonly manifest: AgentBundleTestManifest; + /** The path-derived script name the generated executable reports as its surface. */ + readonly name: string; + /** Receives the completed document's value, exactly what the rendered-script shell validated. */ + readonly onComplete: (value: unknown) => void; + /** The script executable's process identity; a generated `scripts/.mjs` is a fresh process per run. */ + readonly processLifetime: ProviderProcessLifetime; + readonly provenance: RenderedRouteProvenance; + readonly signal: AbortSignal; +} + +export interface PreparedScriptRenderHost { + readonly close: () => Promise; + readonly createSession: ( + argv: readonly string[], + context: { readonly signal: AbortSignal }, + ) => GeneratedCliRenderSession; + /** + * Ends the render the way the generated executable's render worker ending + * does: every session's event stream fails with `reason`, nothing further + * starts, and the shell reports the failure. The harness calls this when + * rendered user code calls `process.exit`, which in that worker ends the + * worker — never the executable's own shell. + */ + readonly terminate: (reason: Error) => void; +} + +/** + * Settles with `pending`, or rejects with the signal's reason as soon as it + * aborts: the pending work is abandoned, the way the generated executable + * abandons its render worker. + */ +const settledBeforeAbort = (pending: Promise, signal: AbortSignal): Promise => + new Promise((resolve, reject) => { + const onAbort = (): void => { reject(signal.reason); }; + if (signal.aborted) { + onAbort(); + return; + } + signal.addEventListener('abort', onAbort, { once: true }); + pending.then(resolve, reject).finally(() => { signal.removeEventListener('abort', onAbort); }); + }); + +/** + * The render host behind the script-dispatch level for rendered scripts. It + * mirrors the generated `scripts/.mjs` executable exactly: the + * component receives `{ argv, signal }`, the request scope opens with the + * `script` invocation naming the route and the script surface, and — like + * that executable — the completed value passes through unvalidated; rendered + * scripts carry no `resultSchema` contract. + */ +export const prepareScriptRenderHost = async ( + options: PrepareScriptRenderHostOptions, +): Promise => { + const context = options.context ?? {}; + // Resolving the state loader is harness wiring and happens now. Loading the + // renderer (React, the runtime, the Flight server) and mounting the state — + // loading its module, opening a driver root — is the render worker's work: + // the generated executable starts that worker only once the shell has + // accepted argv, so both wait, like the module, for the shell to open a + // session. + const mount = manifestStateMount(options.manifest, options.provenance, context); + // The render worker's lifetime: `terminate` ends it, as `process.exit` in + // worker code ends the generated executable's worker, and every step and + // stream of the host observes that end alongside the caller's own signal. + const termination = new AbortController(); + const terminate = (reason: Error): void => { termination.abort(reason); }; + const ended = (signal: AbortSignal): AbortSignal => AbortSignal.any([signal, termination.signal]); + let rendering: Promise | undefined; + let mounting: Promise | undefined; + let mounted: AutoMountedState | undefined; + const close = async (): Promise => { + if (mounting === undefined) return; + if (mounted === undefined && ended(options.signal).aborted) { + // The executable terminates its render worker on abort; here a mount + // still pending may never settle, so closing waits for nothing and + // whatever the mount does produce later is closed on arrival. + void mounting.then((state) => state.close(), () => undefined); + return; + } + const state = await mounting.catch(() => undefined); + await state?.close(); + }; + return Object.freeze({ + close, + createSession: (argv: readonly string[], execution: { readonly signal: AbortSignal }): GeneratedCliRenderSession => { + const invocation: AgentRenderInvocation = { + kind: 'script', + props: { input: argv as never, name: options.name }, + }; + const collected: AgentProgressUpdate[] = []; + // State and module are user code: they load when the shell asks for + // events, and a load or shape failure is the stream's failure, never + // the harness's. + // Once the signal aborts, the executable's render worker is gone: no + // state is mounted and no module is loaded after it. A load already in + // flight cannot be recalled, but each step that has not begun checks + // the signal before it starts rather than running on behind a run that + // has already reported its cancellation. + const hostSignal = ended(options.signal); + const signal = ended(execution.signal); + rendering ??= loadRenderer(); + mounting ??= rendering + .then((renderer) => { + hostSignal.throwIfAborted(); + return mount(renderer, hostSignal); + }) + .then((state) => { mounted = state; return state; }); + // The generated worker composes the project's root layout around a + // rendered script (a script belongs to no server, so no server layout + // applies); the layout modules are user code and load with the script's. + const layoutRoute: LayoutChainTarget = { id: options.provenance.routeId, kind: 'script' }; + const pending = Promise.all([rendering, mounting]).then(async ([renderer, state]) => { + signal.throwIfAborted(); + const [module, layouts] = await Promise.all([ + options.loadModule(), + loadLayoutChain(options.manifest, layoutRoute, options.provenance), + ]); + signal.throwIfAborted(); + return createFlightDispatcher({ + collected, + component: componentOf(module, options.provenance), + componentProps: (request) => ({ argv, signal: request.signal }), + contextProgress: context.progress, + layoutRoute, + layouts, + renderer, + requestInit: async (request) => { + const root = process.cwd(); + // The generated script's render worker hands its providers the + // `script` invocation with the path-derived name, never the route id. + const providers = await mountProviders({ + explicit: context.providers, + invocation, + manifest: options.manifest, + processHit: claimProcessHit(options.processLifetime), + provenance: options.provenance, + signal: request.signal, + }); + return { + capabilities: { + command: renderer.unavailable(), + filesystem: renderer.unavailable(), + network: renderer.unavailable(), + projectRoot: renderer.available({ root }, 'derived'), + }, + host: renderer.unavailable('unsupported-surface'), + workspace: renderer.available({ root }, 'derived'), + ...context, + ...state.context, + providers, + invocation: { + kind: 'script', + operationId: options.provenance.routeId, + surface: options.name, + ...context.invocation, + }, + signal: request.signal, + }; + }, + }); + }); + void pending.catch(() => undefined); + let inner: ReadableStreamDefaultReader | undefined; + return Object.freeze({ + close, + events: (): ReadableStream => new ReadableStream({ + cancel: async (reason) => { await inner?.cancel(reason); }, + start: async (controller) => { + try { + // The executable fails its parent stream the moment the signal + // aborts and terminates the worker, however far along the + // module or state load is; the stream here fails the same way + // rather than waiting for a load that may never settle. + const dispatcher = await settledBeforeAbort(pending, signal); + inner = dispatcher.stream({ invocation, signal }).getReader(); + for (;;) { + const next = await inner.read(); + if (next.done) break; + termination.signal.throwIfAborted(); + controller.enqueue(next.value); + } + termination.signal.throwIfAborted(); + controller.close(); + } catch (error) { + // A worker that exited fails the executable's pending render + // with the exit, whatever the render itself was doing. + controller.error(termination.signal.aborted ? termination.signal.reason : error); + } + }, + }), + validate: (value: unknown) => { + options.onComplete(value); + return value; + }, + }); + }, + terminate, + }); +}; + interface PreparedRender { readonly close: () => Promise; readonly collected: readonly AgentProgressUpdate[]; diff --git a/packages/agent-bundle/src/test/script.ts b/packages/agent-bundle/src/test/script.ts new file mode 100644 index 000000000..72bb5e383 --- /dev/null +++ b/packages/agent-bundle/src/test/script.ts @@ -0,0 +1,692 @@ +/** + * The script dispatch proof level. + * + * `runScript` runs one conventional `src/scripts/` module through the + * contract its generated `scripts/.mjs` executable carries (#102): + * + * - a rendered `.tsx` script runs through the rendered-script shell + * (`runGeneratedRenderedScript`) — the same `--json` / `--ndjson` + * reservation, TTY progress, piped Markdown, and exit-code mapping the + * executable applies — over an in-process render session that shares the + * route-unit harness's dispatcher and Flight renderer; + * - a plain `.ts` script runs as its own Node process over the source + * itself: the generated envelope evaluates the module (afresh, every run), + * awaits a `main` export with argv, adopts a numeric return as the exit + * code, or simply lets a self-executing module run. `process.argv`, + * `process.exit`, `process.chdir`, the exit code, and the streams are the + * real process's; an escaped rejection takes Node's top-level failure path. + * + * It does **not** bundle the script or touch a host artifact: no + * `scripts/.mjs`, no `-flight.mjs` worker sibling. The packed CLI + * route suite owns that evidence. + */ +import { AsyncLocalStorage } from 'node:async_hooks'; +import { spawn } from 'node:child_process'; +import { access } from 'node:fs/promises'; +import { createRequire } from 'node:module'; +import { constants as osConstants } from 'node:os'; +import { format } from 'node:util'; + +import { scanEntryExports } from '../build/entry-exports.ts'; +import { metaModuleSpecifier } from '../build/meta.ts'; +import { runGeneratedRenderedScript } from '../cli-entry.ts'; +import type { CliRenderedEvent } from '../cli-entry.ts'; +import { testMetaModuleSource } from '../rstest/meta-module.ts'; +import { createProviderProcessLifetime } from '../routes/provider-execution.ts'; +import { AgentTestError, captured } from './errors.ts'; +import { + SCRIPT_DISPATCH_PROOF_LEVEL, + type AgentBundleTestManifest, + type TestableScriptDescriptor, +} from './manifest.ts'; +import { parseCanonicalJsonLine, parseRenderedEventLines } from './output-modes.ts'; +import { registeredRouteLoader, testManifest } from './registry.ts'; +import { + prepareScriptRenderHost, + type HarnessOptionsArguments, + type RenderRouteContextInit, +} from './render.ts'; +import type { AgentRouteModule, RenderedRouteProvenance } from './types.ts'; + +export interface RunScriptOptionsBase { + readonly manifest?: AgentBundleTestManifest; + readonly signal?: AbortSignal; + /** + * Input piped to a plain script's stdin, ended after the last byte. Omitted, + * the script reads end-of-file at once. A generated executable inherits the + * invoking terminal's stdin; a test names its input instead. Plain scripts + * only. + */ + readonly stdin?: string; + /** + * Selects interactive rendered output explicitly. Generated executables use + * `process.stdout.isTTY`; the in-process harness defaults to piped output. + * Rendered scripts only. + */ + readonly tty?: boolean; +} + +/** + * Run options. `context` carries the request-scope overrides for a rendered + * script over the runtime's request contract (see {@link RenderRouteContextInit}): + * omitted, the harness mounts the project's `src/providers/*` with the + * `script` invocation exactly as the generated executable does before the + * component renders, and `context.providers` substitutes a fixture map. A + * plain script has no request scope and accepts no `context`. + */ +export type RunScriptOptions = RunScriptOptionsBase & RenderRouteContextInit; + +/** How one script module executed at this level. */ +export type ScriptExecution = 'main-envelope' | 'rendered-shell' | 'self-executing'; + +export interface ScriptDispatchProvenance extends Pick { + readonly execution: ScriptExecution; + readonly proofLevel: typeof SCRIPT_DISPATCH_PROOF_LEVEL; + /** Every conventional script the graph compiled, so a lookup failure can name the alternatives. */ + readonly scripts: readonly string[]; +} + +export interface ScriptInvocation { + /** The argv vector as dispatched; for rendered scripts this still includes the `--json` / `--ndjson` mode flags. */ + readonly argv: readonly string[]; + /** + * The process status the contract mapped: rendered scripts exit 0 on a + * `success` document and 1 otherwise (2 for conflicting mode flags); plain + * scripts report their process's exit status — a numeric `main` return, an + * assigned `process.exitCode`, a `process.exit` call, 1 after an escaped + * rejection, or 128 + the signal number when a signal ended the process. + */ + readonly exitCode: number; + readonly kind: 'plain' | 'rendered'; + readonly name: string; + readonly provenance: ScriptDispatchProvenance; + readonly routeId: string; + /** Everything the script wrote to its diagnostic stream. */ + readonly stderr: string; + /** Everything the script wrote to stdout, including rendered Markdown, TTY, JSON, or NDJSON output. */ + readonly stdout: string; + /** Rendered scripts: the completed document value the shell passed through. Absent for plain scripts. */ + readonly value?: unknown; +} + +const scriptNames = (manifest: AgentBundleTestManifest): readonly string[] => + Object.freeze(manifest.scripts.map((script) => script.name)); + +const provenanceOf = ( + manifest: AgentBundleTestManifest, + execution: ScriptExecution, +): ScriptDispatchProvenance => Object.freeze({ + execution, + manifestDigest: manifest.digest, + proofLevel: SCRIPT_DISPATCH_PROOF_LEVEL, + projectRoot: manifest.projectRoot, + scripts: scriptNames(manifest), +}); + +const routeProvenance = ( + manifest: AgentBundleTestManifest, + script: TestableScriptDescriptor, +): RenderedRouteProvenance => Object.freeze({ + kind: 'script', + manifestDigest: manifest.digest, + modulePath: script.source, + projectRoot: manifest.projectRoot, + proofLevel: SCRIPT_DISPATCH_PROOF_LEVEL, + relativePath: script.relativePath, + routeId: script.routeId, + source: 'manifest', + targets: manifest.targets, +}); + +const compilerDetail = (manifest: AgentBundleTestManifest): readonly string[] => manifest.diagnostics.length === 0 + ? [] + : [`compiler: ${String(manifest.diagnostics.length)} diagnostic(s), first ${manifest.diagnostics[0]!.code}: ${manifest.diagnostics[0]!.message}`]; + +const resolveScript = (manifest: AgentBundleTestManifest, name: string): TestableScriptDescriptor => { + const script = manifest.scripts.find((candidate) => candidate.name === name || candidate.routeId === name); + if (script !== undefined) return script; + const names = scriptNames(manifest); + throw new AgentTestError( + 'script-not-found', + names.length === 0 + ? 'This project compiled no conventional scripts.' + : `No compiled conventional script is named ${JSON.stringify(name)}.`, + { + details: [ + `project root: ${manifest.projectRoot}`, + `compiled: ${names.length === 0 ? 'no src/scripts/* modules' : names.join(', ')}`, + ...compilerDetail(manifest), + ], + recovery: names.length === 0 + ? 'Add a module under src/scripts/ (a plain .ts exporting main, or a rendered .tsx component). Explicit scripts: configuration entries are bundled entries, not routes; the packed pool proves those.' + : 'Run one of the compiled script names, or its script: route id.', + }, + ); +}; + +/** + * The registered loader for one script, resolved before any user code runs: + * a missing loader is harness wiring, reported as such, never as the + * script's own failure. + */ +const loaderFor = ( + manifest: AgentBundleTestManifest, + script: TestableScriptDescriptor, +): (() => Promise) => { + const loader = registeredRouteLoader(manifest, script.routeId); + if (loader === undefined) { + throw new AgentTestError( + 'manifest-unavailable', + `Script ${script.name} is compiled but no test-time module loader is registered for it.`, + { + provenance: routeProvenance(manifest, script), + recovery: 'Build the Rstest configuration with agentBundleRstest() so the generated setup registers script loaders.', + }, + ); + } + return loader; +}; + +/** + * The compiled source of a plain script, confirmed present before a process + * is started: a manifest that names a module the tree no longer holds is + * harness wiring (a stale graph), reported as such rather than as the + * script's own module-not-found failure. + */ +const plainSourceFor = async ( + manifest: AgentBundleTestManifest, + script: TestableScriptDescriptor, +): Promise => { + try { + await access(script.source); + return script.source; + } catch (cause) { + throw new AgentTestError( + 'manifest-unavailable', + `Script ${script.name} is compiled but its source ${script.source} is not on disk.`, + { + cause, + provenance: routeProvenance(manifest, script), + recovery: 'Recompile the test manifest against the current tree, or restore the module the manifest names.', + }, + ); + } +}; + +/** The status a shell reports for a process: its exit code, or 128 + the number of the signal that ended it. */ +const processStatus = (code: number | null, signal: NodeJS.Signals | null): number => { + if (signal !== null) return 128 + (osConstants.signals[signal] ?? 0); + return code ?? 1; +}; + +/** + * `@rsbuild/core` as the child resolves it: its bundled SWC lowers a `.tsx` + * or `.jsx` module the way the production Rslib profile does (automatic + * React runtime), so a plain script may import the same helpers its bundle + * would. + */ +const rsbuildCorePath = ((): string | undefined => { + try { + return createRequire(import.meta.url).resolve('@rsbuild/core'); + } catch { + return undefined; + } +})(); + +/** + * The child process's module hooks, preloaded through `--import` as source: + * `agent-bundle/meta` resolves to the identity module the Rstest pool serves + * (the build's generated module over the manifest's plugin identity, or the + * AB4760 module when the compiler pass produced no model), relative `.js` specifiers that name TypeScript sources resolve to + * them (as the bundler resolves them for the generated executable), Node's + * own type transform handles `.ts`, and `.tsx` / `.jsx` lower through the + * bundler's SWC. + */ +const hooksSource = (manifest: AgentBundleTestManifest): string => ` +import { createRequire, registerHooks } from 'node:module'; +import { existsSync, readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; + +const metaSpecifier = ${JSON.stringify(metaModuleSpecifier)}; +const metaUrl = 'data:text/javascript,' + encodeURIComponent(${JSON.stringify(testMetaModuleSource(manifest))}); +const rsbuildCore = ${JSON.stringify(rsbuildCorePath ?? null)}; +// JSX in a dependency, in TypeScript (.tsx) or JavaScript (.jsx): the build +// lowers both through the React plugin, Node on its own loads neither. +const lowerJsx = (filename, source) => { + if (rsbuildCore === null) { + throw new Error('Cannot load ' + filename + ': the harness could not resolve @rsbuild/core to lower JSX.'); + } + return createRequire(rsbuildCore)(rsbuildCore).rspack.experiments.swc.transformSync(source, { + filename, + isModule: true, + jsc: { + parser: filename.endsWith('.tsx') ? { syntax: 'typescript', tsx: true } : { syntax: 'ecmascript', jsx: true }, + target: 'es2022', + transform: { react: { runtime: 'automatic' } }, + }, + module: { type: 'es6' }, + sourceMaps: false, + }).code; +}; + +registerHooks({ + resolve(specifier, context, nextResolve) { + if (specifier === metaSpecifier) return { shortCircuit: true, url: metaUrl }; + try { + return nextResolve(specifier, context); + } catch (error) { + const relativeJs = /^\\.\\.?\\/.*\\.[cm]?js$/u.test(specifier); + if (error !== null && typeof error === 'object' && error.code === 'ERR_MODULE_NOT_FOUND' && relativeJs && context.parentURL !== undefined) { + for (const extension of ['.ts', '.tsx', '.mts', '.cts']) { + const candidate = specifier.replace(/\\.[cm]?js$/u, extension); + const url = new URL(candidate, context.parentURL); + if (url.protocol === 'file:' && existsSync(fileURLToPath(url))) return nextResolve(candidate, context); + } + } + throw error; + } + }, + load(url, context, nextLoad) { + if (!url.startsWith('file:') || !(url.endsWith('.tsx') || url.endsWith('.jsx'))) return nextLoad(url, context); + const filename = fileURLToPath(url); + return { format: 'module', shortCircuit: true, source: lowerJsx(filename, readFileSync(filename, 'utf8')) }; + }, +}); +`; + +/** + * The generated process envelope, as the child's entry: the same statements + * `generatedExecutableEntrySource` emits into `scripts/.mjs`, over the + * source module named by `process.argv[1]`. + */ +const envelopeSource = (execution: ScriptExecution): string => [ + "import { pathToFileURL } from 'node:url';", + '', + // A generated `scripts/.mjs` runs under plain `node `: the + // loader flags this launch needs are the harness's, not the script's. + 'process.execArgv = [];', + 'const source = process.argv[1];', + 'const entry = await import(pathToFileURL(source).href);', + ...(execution === 'main-envelope' + ? [ + 'const main = entry.main;', + "if (typeof main !== 'function') {", + " throw new TypeError('Executable entry must export a main function: ' + source);", + '}', + 'const code = await main(process.argv.slice(2));', + "if (typeof code === 'number') process.exitCode = code;", + ] + : []), + '', +].join('\n'); + +interface PlainRunResult { + readonly execution: ScriptExecution; + readonly exitCode: number; + readonly stderr: string; + readonly stdout: string; +} + +/** + * How long an aborted script may take to leave after SIGTERM before it is + * killed: a script that traps the signal must still not outlive its run. + */ +const TERMINATION_GRACE_MS = 1000; + +const runPlainScript = async ( + manifest: AgentBundleTestManifest, + script: TestableScriptDescriptor, + argv: readonly string[], + stdin: string | undefined, + signal: AbortSignal, +): Promise => { + signal.throwIfAborted(); + const source = await plainSourceFor(manifest, script); + // The builder decides the envelope statically, before the module ever + // runs; the same scan decides here so a non-callable `main` export fails + // the way the generated executable fails instead of passing as a + // self-executing module. + const execution: ScriptExecution = (await scanEntryExports(source)).hasMainExport + ? 'main-envelope' + : 'self-executing'; + // An abort that landed while the source was being resolved and scanned is + // not replayed to a listener added afterwards; checked here, with nothing + // asynchronous between the check, the spawn, and the listener, no process + // can start without the abort reaching it. + signal.throwIfAborted(); + // A generated executable runs under plain `node`; the test runner's own + // flags are not inherited, only the type transform the source needs. + const child = spawn(process.execPath, [ + '--experimental-transform-types', + '--disable-warning=ExperimentalWarning', + '--import', `data:text/javascript,${encodeURIComponent(hooksSource(manifest))}`, + '--input-type=module', + '--eval', envelopeSource(execution), + '--', + source, + ...argv, + ], { stdio: [stdin === undefined ? 'ignore' : 'pipe', 'pipe', 'pipe'] }); + if (child.stdin !== null) { + // The script may exit without reading; a closed pipe is not the run's failure. + child.stdin.on('error', () => undefined); + child.stdin.end(stdin); + } + let out = ''; + let err = ''; + child.stdout?.setEncoding('utf8').on('data', (chunk: string) => { out += chunk; }); + child.stderr?.setEncoding('utf8').on('data', (chunk: string) => { err += chunk; }); + // Abort asks the process to leave as an operator would, then makes sure it + // has: the run settles only once the process is gone, so a script that + // traps SIGTERM cannot outlive its test. + let escalation: NodeJS.Timeout | undefined; + const terminate = (): void => { + child.kill('SIGTERM'); + escalation = setTimeout(() => { child.kill('SIGKILL'); }, TERMINATION_GRACE_MS); + escalation.unref(); + }; + signal.addEventListener('abort', terminate, { once: true }); + try { + const status = await new Promise((resolve, reject) => { + child.once('error', reject); + child.once('close', (code, ended) => { resolve(processStatus(code, ended)); }); + }); + signal.throwIfAborted(); + return Object.freeze({ execution, exitCode: status, stderr: err, stdout: out }); + } finally { + signal.removeEventListener('abort', terminate); + if (escalation !== undefined) clearTimeout(escalation); + } +}; + +const rejectRenderedOnlyOptions = ( + manifest: AgentBundleTestManifest, + script: TestableScriptDescriptor, + options: RunScriptOptions, +): void => { + // A plain script opens no request scope: no providers are mounted for it, + // so there is nothing for `context` — fixtures included — to override. + const offending = [ + ...(options.context === undefined ? [] : ['context']), + ...(options.tty === undefined ? [] : ['tty']), + ]; + if (offending.length === 0) return; + throw new AgentTestError( + 'invalid-input', + `Plain script ${script.name} has no render session; ${offending.join(' and ')} apply to rendered (.tsx) scripts only.`, + { + provenance: routeProvenance(manifest, script), + recovery: 'Drop the rendered-only options, or rename the script to .tsx to render it through the Agent renderer.', + }, + ); +}; + +type StreamWrite = typeof process.stdout.write; + +/** + * What a rendered run stands in for: the generated executable's render + * worker. Its stdout and stderr are both forwarded onto the executable's + * stderr, so machine output owns stdout and a `console.log` in a route never + * reaches it; and `process.exit` in worker code ends the worker — the shell + * sees its pending render fail with the exit — never the executable itself. + * In this process the render happens on the test worker's own streams and + * `process`; while a rendered run is under way, writes and exits made in its + * async context go to the run, and those from anywhere else pass through + * untouched, so rendered runs may overlap other tests. + */ +interface RenderedWorker { + /** Set once the worker "exited": nothing it writes afterwards exists. */ + exited: boolean; + readonly onExit: (code: number) => void; + readonly sink: (text: string) => void; +} + +const renderedWorker = new AsyncLocalStorage(); +let capturingRenders = 0; + +/** The failure the generated executable's shell reports when its render worker exits mid-render. */ +export const renderWorkerExited = (code: number): string => `Generated render worker exited with code ${String(code)}.`; + +const routedExit = (original: typeof process.exit): typeof process.exit => function exit( + this: unknown, + code?: number | string | null, +): never { + const worker = renderedWorker.getStore(); + if (worker === undefined) return Reflect.apply(original, process, [code]) as never; + const numeric = code === undefined || code === null ? 0 : Number(code); + if (!worker.exited) { + worker.exited = true; + worker.onExit(numeric); + } + // Worker code never runs past its `process.exit`; here the call unwinds the + // caller instead. Should the caller catch that and carry on, the worker has + // already gone: the run has failed and its further output is discarded. + throw new Error(renderWorkerExited(numeric)); +}; + +const routedWrite = (original: StreamWrite, stream: NodeJS.WriteStream): StreamWrite => function write( + this: unknown, + chunk: Uint8Array | string, + encodingOrCallback?: BufferEncoding | ((error?: Error | null) => void), + callback?: (error?: Error | null) => void, +): boolean { + const worker = renderedWorker.getStore(); + if (worker === undefined) { + return Reflect.apply(original, stream, [chunk, encodingOrCallback, callback]) as boolean; + } + const encoding = typeof encodingOrCallback === 'string' ? encodingOrCallback : undefined; + if (!worker.exited) worker.sink(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString(encoding)); + const done = typeof encodingOrCallback === 'function' ? encodingOrCallback : callback; + done?.(null); + return true; +}; + +const consoleMethods = ['debug', 'error', 'info', 'log', 'trace', 'warn'] as const; +type ConsoleMethod = (typeof consoleMethods)[number]; +type ConsoleWrite = (...data: unknown[]) => void; + +/** + * A console method that, inside a rendered run, formats onto the run's sink + * the way Node's console formats onto its stream, and otherwise defers to + * whatever the method was — a test runner commonly installs its own. + */ +const routedConsole = (method: ConsoleMethod, original: ConsoleWrite): ConsoleWrite => (...data) => { + const worker = renderedWorker.getStore(); + if (worker === undefined) { + original(...data); + return; + } + if (!worker.exited) worker.sink(`${format(...data)}\n`); +}; + +interface UnpatchedProcess { + readonly console: Readonly>; + readonly err: StreamWrite; + readonly exit: typeof process.exit; + readonly out: StreamWrite; +} + +let unpatchedProcess: UnpatchedProcess | undefined; + +const withRenderedWorker = async ( + worker: Readonly>, + run: () => Promise, +): Promise => { + if (capturingRenders === 0) { + const out = process.stdout.write; + const err = process.stderr.write; + const exit = process.exit; + const methods = Object.fromEntries(consoleMethods.map((method) => [method, console[method] as ConsoleWrite])); + unpatchedProcess = { console: methods as Record, err, exit, out }; + process.stdout.write = routedWrite(out, process.stdout); + process.stderr.write = routedWrite(err, process.stderr); + process.exit = routedExit(exit); + for (const method of consoleMethods) console[method] = routedConsole(method, methods[method]!); + } + capturingRenders += 1; + try { + return await renderedWorker.run({ exited: false, onExit: worker.onExit, sink: worker.sink }, run); + } finally { + capturingRenders -= 1; + if (capturingRenders === 0 && unpatchedProcess !== undefined) { + process.stdout.write = unpatchedProcess.out; + process.stderr.write = unpatchedProcess.err; + process.exit = unpatchedProcess.exit; + for (const method of consoleMethods) console[method] = unpatchedProcess.console[method]; + unpatchedProcess = undefined; + } + } +}; + +const rejectPlainOnlyOptions = ( + manifest: AgentBundleTestManifest, + script: TestableScriptDescriptor, + options: RunScriptOptions, +): void => { + if (options.stdin === undefined) return; + throw new AgentTestError( + 'invalid-input', + `Rendered script ${script.name} renders in this process and reads no stdin; stdin applies to plain (.ts) scripts only.`, + { + provenance: routeProvenance(manifest, script), + recovery: 'Drop stdin, or pass the input as argv; a rendered script receives its input as { argv }.', + }, + ); +}; + +/** + * Runs one conventional script through its generated-executable contract — + * a rendered script in this process, a plain script as a Node process of its + * own over the source — and returns its exit code, streams, and (for + * rendered scripts) the completed document value. + * + * This is the `script-dispatch` proof level. Nothing is bundled. + */ +export const runScript = async ( + name: string, + argv: readonly string[] = [], + ...[options = {}]: HarnessOptionsArguments +): Promise => { + const manifest = options.manifest ?? testManifest(); + const script = resolveScript(manifest, name); + const signal = options.signal ?? new AbortController().signal; + const frozenArgv = Object.freeze([...argv]); + + if (!script.rendered) { + rejectRenderedOnlyOptions(manifest, script, options); + const plain = await runPlainScript(manifest, script, frozenArgv, options.stdin, signal); + return Object.freeze({ + argv: frozenArgv, + exitCode: plain.exitCode, + kind: 'plain', + name: script.name, + provenance: provenanceOf(manifest, plain.execution), + routeId: script.routeId, + stderr: plain.stderr, + stdout: plain.stdout, + }); + } + + const provenance = routeProvenance(manifest, script); + rejectPlainOnlyOptions(manifest, script, options); + // Resolving the loader is harness wiring; loading the module is user code + // and happens only when the shell opens a session, after its argv checks. + const loadModule = loaderFor(manifest, script); + let value: unknown; + let out = ''; + let err = ''; + const host = await prepareScriptRenderHost({ + ...(options.context === undefined ? {} : { context: options.context }), + loadModule, + manifest, + name: script.name, + onComplete: (completed) => { value = completed; }, + // Each generated `scripts/.mjs` is a process of its own: its + // providers see hit 1 of a fresh identity on every run. + processLifetime: createProviderProcessLifetime(), + provenance, + signal, + }); + let exitCode: number; + try { + exitCode = await withRenderedWorker({ + // `process.exit` in the render worker ends it; the shell's pending + // render fails with the exit and the shell reports that failure. + onExit: (code) => { host.terminate(new Error(renderWorkerExited(code))); }, + sink: (text) => { err += text; }, + }, () => runGeneratedRenderedScript({ + argv: frozenArgv, + createSession: host.createSession, + isTty: () => options.tty === true, + name: script.name, + signal, + writeErr: (text) => { err += text; }, + writeOut: (text) => { out += text; }, + })); + } finally { + await host.close(); + } + return Object.freeze({ + argv: frozenArgv, + exitCode, + kind: 'rendered', + name: script.name, + provenance: provenanceOf(manifest, 'rendered-shell'), + routeId: script.routeId, + stderr: err, + stdout: out, + ...(value === undefined ? {} : { value }), + }); +}; + +const outputFailure = ( + invocation: ScriptInvocation, + message: string, + recovery: string, + cause: unknown, +): AgentTestError => new AgentTestError('projection-failed', message, { + cause, + details: [ + `exit code: ${String(invocation.exitCode)}`, + `execution: ${invocation.provenance.execution}`, + `stdout: ${captured(invocation.stdout)}`, + ...(invocation.stderr === '' ? [] : [`stderr: ${captured(invocation.stderr)}`]), + ], + provenance: { + kind: 'script', + manifestDigest: invocation.provenance.manifestDigest, + projectRoot: invocation.provenance.projectRoot, + proofLevel: SCRIPT_DISPATCH_PROOF_LEVEL, + routeId: invocation.routeId, + source: 'manifest', + targets: [], + }, + recovery, +}); + +/** The parsed canonical JSON line a successful `--json` rendered-script run wrote to stdout. */ +export const scriptJson = (invocation: ScriptInvocation): unknown => { + try { + return parseCanonicalJsonLine(invocation.stdout); + } catch (error) { + throw outputFailure( + invocation, + 'The dispatched script did not write one canonical JSON line to stdout.', + 'Call scriptJson() only for a rendered script run that passed --json and completed; plain scripts write whatever their main function wrote.', + error, + ); + } +}; + +/** The ordered render events a successful `--ndjson` rendered-script run wrote to stdout. */ +export const scriptNdjson = (invocation: ScriptInvocation): readonly CliRenderedEvent[] => { + try { + return parseRenderedEventLines(invocation.stdout); + } catch (error) { + throw outputFailure( + invocation, + 'The dispatched script did not write one JSON object per line to stdout.', + 'Call scriptNdjson() only for a rendered script run that passed --ndjson and wrote a complete event stream.', + error, + ); + } +}; diff --git a/packages/agent-bundle/src/test/workbench.ts b/packages/agent-bundle/src/test/workbench.ts new file mode 100644 index 000000000..693831841 --- /dev/null +++ b/packages/agent-bundle/src/test/workbench.ts @@ -0,0 +1,467 @@ +/** + * The Workbench-surface proof level. + * + * The developer Workbench never discovers a project itself: the dev server + * runs one compiler pass and serves projections of it — the route manifest + * (`GET /api/routes/manifest`), the state declaration inside it, the + * lifecycle-replay inventory (`GET /api/lifecycles`), and the capability + * counts navigation derives its pages from. `inspectWorkbenchSurface` runs + * that same compiler pass and the same projection functions in this process, + * so a consumer can assert what the Workbench would be given for their + * project without a browser or a dev server. + * + * It does **not** start the dev server, build an artifact, or render the + * Workbench: page-availability and catalog grouping are re-derived here by the + * Workbench's own rules over the same wire shapes, and the repository proves + * that derivation against the real-Chrome Workbench acceptance. Artifact-only + * facts — per-target executables, published epochs, host discovery, live MCP + * probes — stay with the dev-server and browser levels. + */ +import { resolve } from 'node:path'; + +import type { Lifecycle, LifecycleListResponse } from '../contracts/lifecycles.ts'; +import type { Diagnostic } from '../core/diagnostics.ts'; +import { deepFreeze } from '../core/freeze.ts'; +import type { NormalizedStateDefinition } from '../core/types.ts'; +import { routeManifestFor } from '../dev/routes/route-manifest.ts'; +import type { + RouteManifest, + RouteManifestCliCommand, + RouteManifestKind, + RouteManifestRoute, + RouteManifestServerMode, + RouteManifestState, +} from '../dev/routes/route-manifest.ts'; +import type { CompiledRouteGraph } from '../routes/types.ts'; +import { AgentTestError } from './errors.ts'; +import { WORKBENCH_SURFACE_PROOF_LEVEL } from './manifest.ts'; + +/** Every Workbench page the navigation can show, in the Workbench's own order. */ +export type WorkbenchPageName = + | 'overview' + | 'routes' + | 'skills' + | 'hooks' + | 'lifecycles' + | 'hosts' + | 'mcp' + | 'artifacts' + | 'playground' + | 'logs' + | 'evals' + | 'comparisons'; + +/** + * The rail order `packages/workbench/src/main.tsx` renders its navigation + * items in, minus Runtime (a dev-server runtime capability, not a compile-time + * fact). The Workbench e2e pins this list against the real rail. + */ +const workbenchPageOrder: readonly WorkbenchPageName[] = Object.freeze([ + 'overview', + 'routes', + 'skills', + 'hooks', + 'lifecycles', + 'hosts', + 'mcp', + 'artifacts', + 'playground', + 'logs', + 'evals', + 'comparisons', +]); + +/** The Workbench's navigation labels, so an assertion can name the link a browser would show. */ +export const workbenchPageLabel = (page: WorkbenchPageName): string => { + switch (page) { + case 'overview': + return 'Overview'; + case 'routes': + return 'Routes'; + case 'skills': + return 'Skills'; + case 'hooks': + return 'Hooks'; + case 'lifecycles': + return 'Lifecycles'; + case 'hosts': + return 'Hosts'; + case 'playground': + return 'Playground'; + case 'mcp': + return 'MCP playground'; + case 'evals': + return 'Evals'; + case 'comparisons': + return 'Comparisons'; + case 'artifacts': + return 'Artifacts'; + case 'logs': + return 'Logs'; + default: { + const exhaustive: never = page; + throw new TypeError(`Unknown Workbench page ${String(exhaustive)}.`); + } + } +}; + +/** + * The capability counts the Workbench derives its navigation from, as the + * built artifact's inventory would list them: one instance per hook, MCP + * server, or script declaration per selected target it names (a hook shipped + * to two hosts counts twice; a declaration whose targets select none of the + * project's targets emits nothing and counts nothing), plus the declared + * Skills, eval suites, and targets. + */ +export interface WorkbenchCapabilityCounts { + readonly evalSuites: number; + readonly hooks: number; + readonly mcpServers: number; + readonly scripts: number; + readonly skills: number; + readonly targets: number; +} + +/** One route as the Workbench catalog lists it: the manifest route plus, for CLI routes, its compiled command. */ +export interface WorkbenchRouteCatalogEntry { + readonly command?: RouteManifestCliCommand; + /** The ` …` usage line the Routes page renders for a CLI command. */ + readonly commandUsage?: string; + readonly route: RouteManifestRoute; +} + +/** + * One catalog section, exactly as the Routes page groups them: per server and + * kind for MCP routes (`curator · Tools`), project-level for event routes, + * CLI commands, and scripts. + */ +export interface WorkbenchRouteCatalogGroup { + readonly entries: readonly WorkbenchRouteCatalogEntry[]; + readonly kind: RouteManifestKind; + /** The heading text the Routes page renders for this group. */ + readonly label: string; + readonly mode?: string; + readonly server?: string; + readonly serverId?: string; +} + +export interface WorkbenchRouteCatalogServer { + readonly id: string; + readonly mode: RouteManifestServerMode; + readonly name: string; + readonly routeCount: number; +} + +export interface WorkbenchRouteCatalog { + readonly diagnostics: readonly Diagnostic[]; + readonly digest: string; + readonly groups: readonly WorkbenchRouteCatalogGroup[]; + readonly providers: readonly RouteManifest['providers'][number][]; + /** The number the Routes page shows under "Route graph identity". */ + readonly routeCount: number; + readonly servers: readonly WorkbenchRouteCatalogServer[]; + /** The state declaration the Routes page's State region renders; absent when the project declares none. */ + readonly stateDefinition?: RouteManifestState; +} + +export interface WorkbenchSurfaceProvenance { + readonly configPath?: string; + readonly manifestDigest: string; + readonly projectRoot: string; + readonly proofLevel: typeof WORKBENCH_SURFACE_PROOF_LEVEL; + /** The compiler pass revision the dev server would stamp on the manifest. */ + readonly sourceRevision: string; + readonly targets: readonly string[]; +} + +export interface WorkbenchSurface { + readonly catalog: WorkbenchRouteCatalog; + readonly counts: WorkbenchCapabilityCounts; + /** Every event route with the concrete hosts and starter fixtures the Lifecycles page offers for replay. */ + readonly lifecycles: readonly Lifecycle[]; + /** Exactly the wire body of `GET /api/routes/manifest`. */ + readonly manifest: RouteManifest; + /** The navigation pages the Workbench would show, in navigation order. */ + readonly pages: readonly WorkbenchPageName[]; + readonly provenance: WorkbenchSurfaceProvenance; + /** The navigation pages the Workbench would hide for this project. */ + readonly unavailablePages: readonly WorkbenchPageName[]; +} + +const kindLabels: Readonly> = Object.freeze({ + app: 'MCP Apps', + cli: 'CLI commands', + 'event-route': 'Event routes', + prompt: 'Prompts', + resource: 'Resources', + script: 'Scripts', + tool: 'Tools', +}); + +/** The Routes page's group order for one server: MCP kinds first, then project surfaces. */ +const catalogKinds: readonly RouteManifestKind[] = Object.freeze([ + 'tool', + 'resource', + 'prompt', + 'app', + 'event-route', + 'cli', + 'script', +]); + +const byRouteId = (left: WorkbenchRouteCatalogEntry, right: WorkbenchRouteCatalogEntry): number => + left.route.id.localeCompare(right.route.id); + +const cliOperand = (option: RouteManifestCliCommand['options'][number]): string => { + const kind = option.kind === 'enum' ? option.choices?.join('|') ?? 'string' : option.kind; + return `<${kind}>`; +}; + +/** The usage line the Routes page renders: positionals in order, then flags, required ones unbracketed. */ +export const workbenchCommandUsage = (command: RouteManifestCliCommand): string => { + const positionals = command.options.filter((option) => option.positional !== undefined) + .toSorted((left, right) => left.positional! - right.positional!) + .map((option) => option.required + ? `<${option.option}${option.repeated ? '...' : ''}>` + : `[${option.option}${option.repeated ? '...' : ''}]`); + const flags = command.options.filter((option) => option.positional === undefined) + .map((option) => { + const value = option.kind === 'boolean' + ? `--${option.option}` + : `--${option.option} ${cliOperand(option)}${option.repeated ? ' ...' : ''}`; + return option.required ? value : `[${value}]`; + }); + return [...command.path, ...positionals, ...flags].join(' '); +}; + +const entryFor = (route: RouteManifestRoute, command?: RouteManifestCliCommand): WorkbenchRouteCatalogEntry => ({ + ...(command === undefined ? {} : { command, commandUsage: workbenchCommandUsage(command) }), + route, +}); + +const groupFor = ( + kind: RouteManifestKind, + entries: readonly WorkbenchRouteCatalogEntry[], + server?: Readonly<{ id: string; mode: string; name: string }>, +): WorkbenchRouteCatalogGroup => ({ + entries: [...entries].sort(byRouteId), + kind, + label: server === undefined ? kindLabels[kind] : `${server.name} · ${kindLabels[kind]}`, + ...(server === undefined ? {} : { mode: server.mode, server: server.name, serverId: server.id }), +}); + +const serverGroups = (manifest: RouteManifest): readonly WorkbenchRouteCatalogGroup[] => + [...manifest.servers] + .sort((left, right) => left.name.localeCompare(right.name)) + .flatMap((server) => catalogKinds + .map((kind) => ({ entries: server.routes.filter((route) => route.kind === kind).map((route) => entryFor(route)), kind })) + .filter((group) => group.entries.length > 0) + .map((group) => groupFor(group.kind, group.entries, { id: server.id, mode: server.mode, name: server.name }))); + +const cliGroups = (manifest: RouteManifest): readonly WorkbenchRouteCatalogGroup[] => { + const cli = manifest.cli; + if (cli === undefined || cli.routes.length === 0) return []; + const commands = new Map((cli.commands ?? []).map((command) => [command.routeId, command])); + return [{ + entries: cli.routes.map((route) => entryFor(route, commands.get(route.id))).sort(byRouteId), + kind: 'cli', + label: kindLabels.cli, + mode: cli.mode, + }]; +}; + +const projectGroups = (manifest: RouteManifest): readonly WorkbenchRouteCatalogGroup[] => [ + ...(manifest.events.length === 0 ? [] : [groupFor('event-route', manifest.events.map((route) => entryFor(route)))]), + ...cliGroups(manifest), + ...(manifest.scripts.length === 0 ? [] : [groupFor('script', manifest.scripts.map((route) => entryFor(route)))]), +]; + +/** The Routes page catalog derived from one route manifest, by the Workbench's grouping rules. */ +export const workbenchRouteCatalog = (manifest: RouteManifest): WorkbenchRouteCatalog => { + const groups = [...serverGroups(manifest), ...projectGroups(manifest)]; + return { + diagnostics: manifest.diagnostics, + digest: manifest.digest, + groups, + providers: [...manifest.providers].sort((left, right) => left.name.localeCompare(right.name)), + routeCount: groups.reduce((total, group) => total + group.entries.length, 0), + servers: [...manifest.servers] + .map((server) => ({ id: server.id, mode: server.mode, name: server.name, routeCount: server.routes.length })) + .sort((left, right) => left.name.localeCompare(right.name)), + ...(manifest.state === undefined ? {} : { stateDefinition: manifest.state }), + }; +}; + +const catalogHasKind = (catalog: WorkbenchRouteCatalog, kind: RouteManifestKind): boolean => + catalog.groups.some((group) => group.kind === kind && group.entries.length > 0); + +/** + * The Workbench navigation rule: a page appears when either the compiled + * graph declares its surface or configuration declares it without a route + * module. `hosts` is unconditional; the RSC runtime page depends on a live + * runtime provider and is not projected here. + */ +export const workbenchPagesFor = ( + counts: WorkbenchCapabilityCounts, + catalog: WorkbenchRouteCatalog, +): readonly WorkbenchPageName[] => { + const compiledEvents = catalogHasKind(catalog, 'event-route'); + const compiledScripts = catalogHasKind(catalog, 'script'); + const pages = new Set(['overview', 'routes', 'hosts', 'artifacts', 'logs']); + if (counts.skills > 0) pages.add('skills'); + if (counts.hooks > 0 || compiledEvents) pages.add('hooks'); + if (compiledEvents) pages.add('lifecycles'); + if (counts.mcpServers > 0 || catalog.servers.length > 0) pages.add('mcp'); + if (counts.hooks + counts.scripts > 0 || compiledEvents || compiledScripts) pages.add('playground'); + if (counts.evalSuites > 0) { + pages.add('evals'); + pages.add('comparisons'); + } + return workbenchPageOrder.filter((page) => pages.has(page)); +}; + +export interface WorkbenchSurfaceFromGraphInput { + readonly configPath?: string; + readonly counts: WorkbenchCapabilityCounts; + readonly graph: CompiledRouteGraph; + readonly lifecycles: LifecycleListResponse; + readonly projectRoot: string; + readonly sourceRevision: string; + readonly state?: NormalizedStateDefinition; + readonly targets: readonly string[]; +} + +/** + * The pure projection behind {@link inspectWorkbenchSurface}: the same + * `routeManifestFor` the dev server serves, grouped by the Routes page's + * rules, with the navigation rule applied over the declared counts. + */ +export const workbenchSurfaceFromRouteGraph = (input: WorkbenchSurfaceFromGraphInput): WorkbenchSurface => { + const manifest = routeManifestFor(input.graph, input.sourceRevision, input.state); + const catalog = workbenchRouteCatalog(manifest); + const pages = workbenchPagesFor(input.counts, catalog); + return deepFreeze({ + catalog, + counts: input.counts, + lifecycles: input.lifecycles.lifecycles, + manifest, + pages, + provenance: { + ...(input.configPath === undefined ? {} : { configPath: input.configPath }), + manifestDigest: manifest.digest, + projectRoot: input.projectRoot, + proofLevel: WORKBENCH_SURFACE_PROOF_LEVEL, + sourceRevision: input.sourceRevision, + targets: input.targets, + }, + unavailablePages: workbenchPageOrder.filter((page) => !pages.includes(page)), + }); +}; + +/** + * The artifact instances a set of declarations produces: one per declaration + * per selected target it names. A declaration with `targets: []`, or with + * targets the project does not select, is emitted nowhere. + */ +const targetInstances = ( + declarations: readonly { readonly targets: readonly string[] }[], + selected: readonly string[], +): number => declarations.reduce( + (total, declaration) => total + declaration.targets.filter((target) => selected.includes(target)).length, + 0, +); + +export interface InspectWorkbenchSurfaceOptions { + /** Explicit Agent Bundle configuration path; discovered from `root` when omitted. */ + readonly configPath?: string; + /** Project root; defaults to the working directory. */ + readonly root?: string; +} + +/** + * Runs the dev server's own preparation for one project and projects it the + * way the Workbench receives it. No artifact is built and no server starts. + * + * This is the `workbench-surface` proof level. + */ +export const inspectWorkbenchSurface = async ( + options: InspectWorkbenchSurfaceOptions = {}, +): Promise => { + const root = resolve(options.root ?? process.cwd()); + const [{ ProjectService }, { emptyCompiledRouteGraph }, { LifecycleReplayService }, { EvalService }] = await Promise.all([ + import('../dev/project-service.ts'), + import('../routes/graph.ts'), + import('../dev/playground/lifecycle-replay-service.ts'), + import('../dev/eval/eval-service.ts'), + ]); + // Constructed as the Workbench server constructs its own: the + // configuration factory sees `development`, a `dev` runtime declaration is + // honored, and the dev server's output roots stay out of the source + // snapshot — so a configuration that branches on the mode compiles here to + // exactly what the Workbench shows. + const prepared = await new ProjectService({ + ...(options.configPath === undefined ? {} : { configPath: options.configPath }), + includeDevRuntime: true, + mode: 'development', + outputRoots: ['dist', '.agent-bundle/runtime', '.agent-bundle/playground'], + root, + }).prepare('dev'); + const graph = prepared.routeGraph ?? emptyCompiledRouteGraph; + const model = prepared.model; + const sourceRevision = prepared.source.revision; + // The dev server serves the route manifest only for a `ready` preparation — + // a model, a revision, and no error diagnostic from normalization, + // validation, or an adapter — and otherwise reports it unavailable (or + // keeps serving the previous valid graph) rather than an empty or + // invalid catalog; so does this. + if (model === undefined || sourceRevision === undefined || prepared.source.state !== 'ready') { + const errors = prepared.diagnostics.filter((diagnostic) => diagnostic.severity === 'error'); + throw new AgentTestError( + 'manifest-unavailable', + `The compiler pass produced no valid project (source state ${prepared.source.state}), so the dev server would report the route manifest unavailable.`, + { + details: [ + `project root: ${prepared.root}`, + `config: ${prepared.configPath}`, + ...(errors.length === 0 + ? [] + : [`compiler: ${String(errors.length)} error(s), first ${errors[0]!.code}: ${errors[0]!.message}`]), + ], + recovery: 'Fix the reported configuration or source diagnostics; the Workbench catalog exists only for a project the compiler accepted.', + }, + ); + } + const targets = Object.freeze(model.targets.map((target) => target.name)); + const lifecycles = new LifecycleReplayService({ + prepared: () => Object.freeze({ graph, sourceRevision, targets }), + registry: prepared.registry, + }).list(); + // The same configuration the preparation selected, never a second + // discovery from the root; the eval service otherwise loads as the + // Workbench server's does. + const evalSuites = (await new EvalService({ + configPath: prepared.configPath, + projectRoot: prepared.root, + registry: prepared.registry, + }).suites()).suites.length; + return workbenchSurfaceFromRouteGraph({ + configPath: prepared.configPath, + counts: Object.freeze({ + evalSuites, + // The Workbench counts the artifact's hook index, which holds one + // compiled wrapper per hook per target; a prebuilt hook points the + // host at its payload and is never indexed, so it never counts. + hooks: targetInstances(model.hooks.filter((hook) => hook.prebuiltPath === undefined), targets), + mcpServers: targetInstances(model.mcpServers, targets), + scripts: targetInstances(model.scripts, targets), + skills: model.skills.length, + targets: targets.length, + }), + graph, + lifecycles, + projectRoot: prepared.root, + sourceRevision, + ...(model.state === undefined ? {} : { state: model.state }), + targets, + }); +}; diff --git a/packages/agent-bundle/tests/projection/script-dispatch-abort.test.ts b/packages/agent-bundle/tests/projection/script-dispatch-abort.test.ts new file mode 100644 index 000000000..68605025b --- /dev/null +++ b/packages/agent-bundle/tests/projection/script-dispatch-abort.test.ts @@ -0,0 +1,86 @@ +import { setTimeout as delay } from 'node:timers/promises'; + +import { expect, it } from '@rstest/core'; + +import { runScript, scriptJson } from '../../src/test/index.ts'; +import { + AGENT_TEST_REGISTRY_SYMBOL_KEY, + registerTestRoutes, + type AgentStateModuleLoader, + type AgentTestRouteRegistry, +} from '../../src/test/registry.ts'; + +/** + * The generated rendered-script executable terminates its render worker the + * moment its signal aborts: whatever the worker had not yet done — loading + * the script module after the state mount, opening the render — never + * happens. The harness renders in this process, where work already in flight + * cannot be recalled, so it must refuse to start each following step once + * the run has reported its cancellation. + * + * This file runs in a worker of its own: the fixture's evaluation counters + * are zero when it starts, so they can tell "never loaded" from "loaded + * late". The registered state loader is held at a gate the test controls, + * which pins the abort to a known point — the state mount in flight — and + * makes the run's continuation observable after the fact. + */ +const stateLoads = (): number => (globalThis as { routeHarnessStateLoads?: number }).routeHarnessStateLoads ?? 0; +const summaryLoads = (): number => (globalThis as { routeHarnessSummaryLoads?: number }).routeHarnessSummaryLoads ?? 0; + +const registrySymbol = Symbol.for(AGENT_TEST_REGISTRY_SYMBOL_KEY); +const registeredRegistry = (): AgentTestRouteRegistry => { + const registry = (globalThis as { [registrySymbol]?: AgentTestRouteRegistry })[registrySymbol]; + if (registry === undefined) throw new Error('The projection pool registered no test routes.'); + return registry; +}; + +it('loads no script module once a rendered run is aborted during its state mount, then prepares fully for the next accepted run', async () => { + expect(stateLoads()).toBe(0); + expect(summaryLoads()).toBe(0); + + const registry = registeredRegistry(); + const realStateLoader = registry.stateLoader; + if (realStateLoader === undefined) throw new Error('The route-harness fixture registers a state loader.'); + let release!: () => void; + const gate = new Promise((resolve) => { release = resolve; }); + let entered!: () => void; + const mountStarted = new Promise((resolve) => { entered = resolve; }); + const gatedStateLoader: AgentStateModuleLoader = async () => { + entered(); + await gate; + return realStateLoader(); + }; + registerTestRoutes({ ...registry, stateLoader: gatedStateLoader }); + try { + const controller = new AbortController(); + const pending = runScript('summary', ['--json', 'aborted'], { signal: controller.signal }); + // The shell accepted argv, opened its session, loaded the renderer, and + // is now inside the state mount, waiting at the gate. + await mountStarted; + controller.abort(); + const aborted = await pending; + + expect(aborted.exitCode).toBe(1); + expect(aborted.stdout).toBe(''); + expect(aborted.stderr).toBe('Aborted.\n'); + expect(aborted.value).toBeUndefined(); + expect(summaryLoads()).toBe(0); + + // The mount that was in flight completes when released — the state + // module evaluates, as the executable's worker could not have been kept + // from finishing a load already begun — but the module load that would + // have followed it never starts. + release(); + await delay(250); + expect(stateLoads()).toBe(1); + expect(summaryLoads()).toBe(0); + } finally { + registerTestRoutes(registry); + } + + const accepted = await runScript('summary', ['--json', 'after-abort']); + expect(accepted.exitCode).toBe(0); + expect(scriptJson(accepted)).toMatchObject({ arguments: ['after-abort'], stateMounted: true }); + expect(stateLoads()).toBe(1); + expect(summaryLoads()).toBe(1); +}, 20_000); diff --git a/packages/agent-bundle/tests/projection/script-dispatch-state.test.ts b/packages/agent-bundle/tests/projection/script-dispatch-state.test.ts new file mode 100644 index 000000000..ddf02370f --- /dev/null +++ b/packages/agent-bundle/tests/projection/script-dispatch-state.test.ts @@ -0,0 +1,36 @@ +import { createRequire } from 'node:module'; + +import { expect, it } from '@rstest/core'; + +import { runScript, scriptJson } from '../../src/test/index.ts'; + +/** + * The generated rendered-script executable validates argv in its shell before + * it starts the render worker; only the worker loads the renderer (React, the + * runtime, the Flight server) and the project's state module, and opens its + * driver. The harness keeps that order: this file runs in a worker of its own + * so neither React nor the fixture's state module has been evaluated when the + * first run rejects, and the fixture's evaluation counter plus the CommonJS + * module cache prove both stay that way until a run the shell accepts. + */ +const stateLoads = (): number => (globalThis as { routeHarnessStateLoads?: number }).routeHarnessStateLoads ?? 0; + +const reactLoaded = (): boolean => Object.keys(createRequire(import.meta.url).cache) + .some((filename) => /[\\/]node_modules[\\/]react[\\/]/u.test(filename)); + +it('loads no renderer and mounts no state for a rendered run the shell rejects, then both once the shell accepts', async () => { + expect(stateLoads()).toBe(0); + expect(reactLoaded()).toBe(false); + + const rejected = await runScript('summary', ['--json', '--ndjson']); + expect(rejected.exitCode).toBe(2); + expect(rejected.stderr).toBe('Use either --json or --ndjson, not both.\n'); + expect(stateLoads()).toBe(0); + expect(reactLoaded()).toBe(false); + + const accepted = await runScript('summary', ['--json', 'after-reject']); + expect(accepted.exitCode).toBe(0); + expect(scriptJson(accepted)).toMatchObject({ arguments: ['after-reject'], stateMounted: true }); + expect(stateLoads()).toBe(1); + expect(reactLoaded()).toBe(true); +}); diff --git a/packages/agent-bundle/tests/projection/script-dispatch.test.ts b/packages/agent-bundle/tests/projection/script-dispatch.test.ts new file mode 100644 index 000000000..1d82b667d --- /dev/null +++ b/packages/agent-bundle/tests/projection/script-dispatch.test.ts @@ -0,0 +1,644 @@ +import { dirname } from 'node:path'; + +import { describe, expect, it } from '@rstest/core'; + +import { + AgentTestError, + runScript, + scriptJson, + scriptNdjson, + testManifest, +} from '../../src/test/index.ts'; +import { FALLBACK_PLUGIN_IDENTITY } from '../../src/test/manifest.ts'; + +const summaryValue = (...argv: string[]) => ({ + arguments: argv, + invocation: 'script', + stateMounted: true, + surface: 'summary', +}); + +describe('the compiled script inventory', () => { + it('lists every conventional script with its extension contract', () => { + expect(testManifest().scripts.map((script) => [script.name, script.rendered])).toEqual([ + ['badge', false], + ['banner', false], + ['blank', true], + ['broken', true], + ['checksum', false], + ['constant', false], + ['identity', false], + ['stalled', true], + ['summary', true], + ['tooling-summary', true], + ]); + }); + + it('names the compiled alternatives when a script is unknown', async () => { + const error = await runScript('missing').catch((thrown: unknown) => thrown); + + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).code).toBe('script-not-found'); + expect((error as AgentTestError).message).toContain('compiled: badge, banner, blank, broken, checksum, constant, identity, stalled, summary, tooling-summary'); + expect((error as AgentTestError).message).toContain(`project root: ${testManifest().projectRoot}`); + }); +}); + +describe('rendered scripts at the script dispatch level', () => { + it('routes what the component logs onto the invocation\'s stderr, as the executable forwards its render worker', async () => { + // Anything escaping the run would land on this process's streams. + const escaped: string[] = []; + const outBefore = process.stdout.write; + const errBefore = process.stderr.write; + const record = (chunk: unknown): boolean => { escaped.push(String(chunk)); return true; }; + process.stdout.write = record as typeof process.stdout.write; + process.stderr.write = record as typeof process.stderr.write; + try { + const run = await runScript('summary', ['--json', '--log']); + + expect(run.exitCode).toBe(0); + expect(JSON.parse(run.stdout)).toEqual(summaryValue('--log')); + expect(run.stderr).toBe('summary log line\nsummary stdout line\nsummary stderr line\n'); + expect(escaped).toEqual([]); + // The run restored this process's streams. + expect(process.stdout.write).toBe(record); + expect(process.stderr.write).toBe(record); + // And a write outside any run still passes through untouched. + process.stdout.write('outside any run\n'); + expect(escaped).toEqual(['outside any run\n']); + } finally { + process.stdout.write = outBefore; + process.stderr.write = errBefore; + } + }); + + it('reports process.exit from rendered code as the worker exit the executable\'s shell reports, and keeps this process alive', async () => { + // The generated executable renders in a worker: `process.exit` there ends + // the worker, the shell's pending render fails with the exit, and the + // shell reports it with exit code 1 — whatever the code was, 0 included. + const exitBefore = process.exit; + const three = await runScript('summary', ['--json', '--exit=3']); + expect(three.exitCode).toBe(1); + expect(three.stdout).toBe(''); + expect(three.stderr).toBe('Generated render worker exited with code 3.\n'); + expect(three.value).toBeUndefined(); + + const zero = await runScript('summary', ['--exit=0']); + expect(zero.exitCode).toBe(1); + expect(zero.stderr).toBe('Generated render worker exited with code 0.\n'); + + // The run restored this process's exit, and the next run is unaffected. + expect(process.exit).toBe(exitBefore); + const after = await runScript('summary', ['--json', 'after-exit']); + expect(after.exitCode).toBe(0); + expect(scriptJson(after)).toEqual(summaryValue('after-exit')); + }); + + it('holds a process.exit the rendered code catches: the worker is gone, so the run fails and later output is discarded', async () => { + const run = await runScript('summary', ['--json', '--exit=5', '--catch-exit']); + + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe(''); + expect(run.stderr).toBe('Generated render worker exited with code 5.\n'); + expect(run.value).toBeUndefined(); + }); + + it('projects a rendered script to final Markdown when stdout is piped', async () => { + const run = await runScript('summary', ['alpha', 'beta']); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + expect(run.stdout).toBe('# Summary\n\n2 argument(s).\n\nsurface: summary\n'); + expect(run.stdout).not.toContain('collecting arguments'); + expect(run.value).toEqual(summaryValue('alpha', 'beta')); + expect(run).toMatchObject({ + kind: 'rendered', + name: 'summary', + provenance: { execution: 'rendered-shell', proofLevel: 'script-dispatch', scripts: ['badge', 'banner', 'blank', 'broken', 'checksum', 'constant', 'identity', 'stalled', 'summary', 'tooling-summary'] }, + routeId: 'script:summary', + }); + }); + + it('updates rendered progress in place for an explicit TTY', async () => { + const run = await runScript('summary', ['alpha'], { tty: true }); + + expect(run.exitCode).toBe(0); + expect(run.stdout).toContain('\r\u001B[2Kcollecting arguments (1/2)'); + expect(run.stdout).toContain('\r\u001B[2Ksummary ready (2/2)'); + expect(run.stdout.endsWith('# Summary\n\n1 argument(s).\n\nsurface: summary\n')).toBe(true); + }); + + it('reserves --json for the canonical value and passes every other argument through', async () => { + const run = await runScript('summary', ['alpha', '--json', '--verbose']); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + expect(run.stdout).toBe(`${JSON.stringify(summaryValue('alpha', '--verbose'))}\n`); + expect(scriptJson(run)).toEqual(summaryValue('alpha', '--verbose')); + // The dispatched argv is reported as given; the component saw the reserved flag removed. + expect(run.argv).toEqual(['alpha', '--json', '--verbose']); + }); + + it('passes reserved flags through untouched after a -- terminator', async () => { + const run = await runScript('summary', ['--json', '--', '--ndjson']); + + expect(run.exitCode).toBe(0); + expect(scriptJson(run)).toEqual(summaryValue('--', '--ndjson')); + }); + + it('returns the pure sequence-numbered render-event stream as NDJSON', async () => { + const run = await runScript('summary', ['alpha', '--ndjson']); + const events = scriptNdjson(run); + const sequences = events.map((event) => event.sequence); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + expect(sequences.every((sequence, index) => index === 0 || sequence > sequences[index - 1]!)).toBe(true); + expect(events.some((event) => event.type === 'progress')).toBe(true); + expect(events.at(-1)).toMatchObject({ + document: { status: 'success', value: summaryValue('alpha') }, + type: 'complete', + }); + expect(JSON.stringify(events)).not.toContain('"jsonrpc"'); + expect(run.stdout.trim().split('\n')).toHaveLength(events.length); + }); + + it('composes only the project\'s root layout around a rendered script, inside the script request scope', async () => { + // A script belongs to no server, so the server layout never applies; the + // root layout observes the `script` invocation the worker opened. + const run = await runScript('summary', ['alpha', '--ndjson']); + const complete = scriptNdjson(run).at(-1); + + expect(complete?.type).toBe('complete'); + const root = complete?.type === 'complete' ? complete.document.root : undefined; + expect(root?.kind === 'result' ? root.metadata : undefined).toEqual({ + invocation: 'script', + shell: 'route-harness', + wrapped: 'script', + }); + }); + + it('exits 1 for a represented error document and projects the error node to Markdown', async () => { + const run = await runScript('summary', ['--fail']); + + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe('**[summary.failed]** The summary was asked to fail.\n'); + expect(run.value).toEqual(summaryValue('--fail')); + }); + + it('reports a component render error on stderr', async () => { + const run = await runScript('summary', ['--explode']); + + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe(''); + // The renderer logs the thrown error with its stack (the render worker's + // console, forwarded onto stderr in the generated executable), then the + // shell reports the failure message. + expect(run.stderr).toMatch(/^Error: summary render exploded\n {4}at Summary \(/u); + expect(run.stderr).toMatch(/\nsummary render exploded\n$/u); + }); + + it('reports cancellation through the shell after rendered progress begins', async () => { + const controller = new AbortController(); + const run = await runScript('summary', ['--wait-for-abort'], { + context: { progress: { report: async () => controller.abort() } }, + signal: controller.signal, + }); + + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe(''); + expect(run.stderr).toContain('Aborted.'); + }); + + it('reports cancellation while the module is still loading, as the executable fails its stream and drops the worker', async () => { + // `stalled` never finishes evaluating; without the abort the run could not end. + const controller = new AbortController(); + const pending = runScript('stalled', ['--json'], { signal: controller.signal }); + await new Promise((resolve) => setTimeout(resolve, 50)); + controller.abort(); + const run = await pending; + + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe(''); + expect(run.stderr).toBe('Aborted.\n'); + expect(run.value).toBeUndefined(); + }, 5000); + + it('mounts the conventional providers for a rendered script with the script invocation, as the generated executable does', async () => { + type Summary = { arguments: number; keys: string[]; libraryTooling: unknown }; + const first = await runScript('tooling-summary', ['--json', 'a.mp4', '--fast']); + const second = await runScript('tooling-summary', ['--json', 'b.mp4']); + + expect(first.exitCode).toBe(0); + // The generated script passes `name: 'tooling-summary'`, never the route id. + expect(scriptJson(first)).toEqual({ + arguments: 2, + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { kind: 'script', surface: 'tooling-summary', tool: 'ffprobe 6.1' }, + }); + expect((scriptJson(second) as Summary).arguments).toBe(1); + + // An explicit fixture map is mounted verbatim: nothing under + // src/providers/ runs for this invocation. + const stubbed = await runScript('tooling-summary', ['--json', 'c.mp4'], { + context: { providers: { libraryTooling: { tool: 'stub' }, processLifetime: { hits: 7, instanceId: 'fixture', pid: 0 } } }, + }); + expect(scriptJson(stubbed)).toEqual({ + arguments: 1, + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { tool: 'stub' }, + }); + }); + + it('rejects conflicting output flags at the shell boundary', async () => { + const run = await runScript('summary', ['--json', '--ndjson']); + + expect(run.exitCode).toBe(2); + expect(run.stdout).toBe(''); + expect(run.stderr).toBe('Use either --json or --ndjson, not both.\n'); + }); + + it('rejects Markdown as canonical JSON or NDJSON with diagnostics naming the level and route', async () => { + const run = await runScript('summary', ['alpha']); + + for (const [accessor, fragment] of [ + [scriptJson, 'did not write one canonical JSON line'], + [scriptNdjson, 'did not write one JSON object per line'], + ] as const) { + const error = ((): unknown => { + try { + return accessor(run); + } catch (thrown) { + return thrown; + } + })(); + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).message).toContain(fragment); + expect((error as AgentTestError).message).toContain('proof level: script-dispatch'); + expect((error as AgentTestError).message).toContain('route: script:summary (script)'); + expect((error as AgentTestError).message).toContain('execution: rendered-shell'); + } + }); +}); + +describe('plain scripts at the script dispatch level', () => { + it('runs a main-exporting script through the process envelope contract', async () => { + const stdoutWrite = process.stdout.write; + const run = await runScript('checksum', ['ab', 'cde']); + + expect(run.exitCode).toBe(0); + expect(run.stdout).toBe('Fixture checksum: 5\n'); + expect(run.stderr).toBe(''); + expect(run.value).toBeUndefined(); + expect(run).toMatchObject({ + kind: 'plain', + name: 'checksum', + provenance: { execution: 'main-envelope', proofLevel: 'script-dispatch' }, + routeId: 'script:checksum', + }); + // The script had its own process view; this one was never patched. + expect(process.stdout.write).toBe(stdoutWrite); + }); + + it('adopts a numeric return as the exit code and keeps stderr separate', async () => { + const run = await runScript('checksum'); + + expect(run.exitCode).toBe(2); + expect(run.stdout).toBe('Fixture checksum: 0\n'); + expect(run.stderr).toBe('No arguments to checksum.\n'); + }); + + it('adopts an assigned process.exitCode when main returns nothing', async () => { + const previous = process.exitCode; + const run = await runScript('checksum', ['--exit-code-property']); + + expect(run.exitCode).toBe(4); + expect(run.stdout).toBe('checksum set process.exitCode\n'); + expect(process.exitCode).toBe(previous); + }); + + it('turns a process.exit call into the exit code instead of ending the test process', async () => { + const run = await runScript('checksum', ['--process-exit']); + + expect(run.exitCode).toBe(5); + expect(run.stdout).toBe('checksum called process.exit\n'); + }); + + it('exits 1 with the stack on stderr when main rejects, like the generated process', async () => { + const run = await runScript('checksum', ['--explode']); + + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe(''); + expect(run.stderr).toContain('Error: checksum exploded'); + }); + + it('reports a stale manifest as harness wiring before any user code runs', async () => { + const compiled = testManifest(); + const checksum = compiled.scripts.find((script) => script.name === 'checksum')!; + const ghost = { + ...checksum, + name: 'ghost', + relativePath: 'src/scripts/ghost.ts', + routeId: 'script:ghost', + source: checksum.source.replace(/checksum\.ts$/u, 'ghost.ts'), + }; + + const error = await runScript('ghost', ['--explode'], { manifest: { ...compiled, scripts: [...compiled.scripts, ghost] } }) + .catch((thrown: unknown) => thrown); + + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).code).toBe('manifest-unavailable'); + expect((error as AgentTestError).message).toContain('is not on disk'); + expect((error as AgentTestError).message).toContain('route: script:ghost (script)'); + + // The harness failure left nothing behind: the next plain run still + // reports its own streams and exit code. + const run = await runScript('checksum', ['--explode']); + expect(run.exitCode).toBe(1); + expect(run.stderr).toContain('Error: checksum exploded'); + }); + + it('refuses rendered-only options for a plain script instead of ignoring them', async () => { + const error = await runScript('checksum', ['ab'], { tty: true }).catch((thrown: unknown) => thrown); + + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).code).toBe('invalid-input'); + expect((error as AgentTestError).message).toContain('tty apply to rendered (.tsx) scripts only'); + expect((error as AgentTestError).message).toContain('module: src/scripts/checksum.ts'); + }); + + it('evaluates a self-executing module afresh on every run, as every process would', async () => { + const first = await runScript('banner', ['x', 'y']); + + expect(first.exitCode).toBe(0); + expect(first.stdout).toBe('banner: x y\n'); + expect(first.provenance.execution).toBe('self-executing'); + + const again = await runScript('banner', ['again']); + expect(again.exitCode).toBe(0); + expect(again.stdout).toBe('banner: again\n'); + + // A manifest for another tree names modules that are not there: that is + // harness wiring, reported before anything runs. + const compiled = testManifest(); + const elsewhere = `${compiled.projectRoot}-sibling`; + const sibling = { + ...compiled, + projectRoot: elsewhere, + scripts: compiled.scripts.map((script) => ({ ...script, source: script.source.replace(compiled.projectRoot, elsewhere) })), + }; + const other = await runScript('banner', ['again'], { manifest: sibling }).catch((thrown: unknown) => thrown); + expect(other).toBeInstanceOf(AgentTestError); + expect((other as AgentTestError).code).toBe('manifest-unavailable'); + expect((other as AgentTestError).message).toContain(`${elsewhere}/src/scripts/banner.ts is not on disk`); + }); + + it('starts a main-exporting script from fresh module state on every run', async () => { + const source = testManifest().scripts.find((script) => script.name === 'checksum')!.source; + const first = await runScript('checksum', ['--calls']); + const second = await runScript('checksum', ['--calls']); + + // Module-level state does not survive between runs, and argv[1] is the + // executable's own path, exactly as the generated process reports it. + expect(first.stdout).toBe(`checksum call 1 in ${source}\n`); + expect(second.stdout).toBe(`checksum call 1 in ${source}\n`); + }); + + it('lowers a .tsx helper a plain script imports, as the bundler does for the generated executable', async () => { + const run = await runScript('badge', ['needs', 'review']); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + expect(run.stdout).toBe('needs review\n'); + expect(run.provenance.execution).toBe('main-envelope'); + }); + + it('lowers a JavaScript .jsx helper imported by extension, as the bundler does', async () => { + const run = await runScript('badge', ['--ribbon', 'needs', 'review']); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + expect(run.stdout).toBe('needs review\n'); + }); + + it('serves agent-bundle/meta as the identity the build stamps, not the published entry that throws', async () => { + const run = await runScript('identity'); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + // The route-harness fixture declares plugin name/version and has no package.json. + expect(run.stdout).toBe('route-harness@1.0.0 - -\n'); + + const compiled = testManifest(); + const packaged = await runScript('identity', [], { + manifest: { ...compiled, plugin: { ...compiled.plugin, packageName: '@fixture/route-harness', packageVersion: '2.3.4' } }, + }); + expect(packaged.stdout).toBe('route-harness@1.0.0 @fixture/route-harness 2.3.4\n'); + + // A manifest without a plugin model carries the frozen sentinel; the + // script is served the same AB4760 module the Rstest pool serves, never + // a fabricated identity. + const modelless = await runScript('identity', [], { manifest: { ...compiled, plugin: FALLBACK_PLUGIN_IDENTITY } }); + expect(modelless.exitCode).toBe(1); + expect(modelless.stdout).toBe(''); + expect(modelless.stderr).toContain('[AB4760]'); + expect(modelless.stderr).toContain('produced no plugin model'); + }); + + it('rejects context on a plain script: it opens no request scope, so no providers are mounted for it', async () => { + const error = await runScript('checksum', ['ab'], { context: { providers: {} } }) + .catch((thrown: unknown) => thrown); + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).code).toBe('invalid-input'); + expect((error as AgentTestError).message).toContain('context apply to rendered (.tsx) scripts only'); + }); + + it('gives the script the process APIs a worker thread refuses, such as process.chdir, without moving this process', async () => { + const cwd = process.cwd(); + const run = await runScript('checksum', ['--chdir']); + + expect(run.exitCode).toBe(0); + expect(run.stdout).toBe(`checksum cwd ${dirname(cwd)}\n`); + expect(process.cwd()).toBe(cwd); + }); + + it('fails a non-callable main export the way the generated envelope does, after evaluating the module', async () => { + const run = await runScript('constant', ['ignored']); + + expect(run.exitCode).toBe(1); + expect(run.provenance.execution).toBe('main-envelope'); + expect(run.stdout).toBe('constant evaluated\n'); + expect(run.stderr).toContain(`TypeError: Executable entry must export a main function: ${testManifest().scripts.find((script) => script.name === 'constant')!.source}`); + }); +}); + +describe('a rendered script whose module fails to evaluate', () => { + const loads = (): number => (globalThis as { routeHarnessBrokenLoads?: number }).routeHarnessBrokenLoads ?? 0; + + it('never evaluates the module when the shell rejects the argv first, exactly like the generated executable', async () => { + const before = loads(); + const run = await runScript('broken', ['--json', '--ndjson']); + + expect(run.exitCode).toBe(2); + expect(run.stderr).toBe('Use either --json or --ndjson, not both.\n'); + expect(loads()).toBe(before); + }); + + it('reports the load failure through the shell as stderr and exit code 1', async () => { + const before = loads(); + const run = await runScript('broken', ['--json']); + + expect(loads()).toBe(before + 1); + expect(run.kind).toBe('rendered'); + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe(''); + expect(run.stderr).toBe('broken script failed to load\n'); + expect(run.value).toBeUndefined(); + expect(run.provenance.execution).toBe('rendered-shell'); + }); + + it('reports a module without a default component the same way, and still closes the mounted state', async () => { + const run = await runScript('blank'); + + expect(run.kind).toBe('rendered'); + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe(''); + expect(run.stderr).toContain('A route module default-exports its route component.'); + expect(run.stderr).toContain('received: default export of type undefined'); + expect(run.value).toBeUndefined(); + + // The host closed cleanly: the next rendered run mounts state again and completes. + const next = await runScript('summary', ['--json', 'after-blank']); + expect(next.exitCode).toBe(0); + expect(scriptJson(next)).toEqual(summaryValue('after-blank')); + }); +}); + +describe('the plain-script process contract', () => { + it('leaves this process untouched while the script runs: its argv, exit code, and streams are its own', async () => { + const argvBefore = process.argv; + const exitCodeBefore = process.exitCode; + const writeBefore = process.stdout.write; + const pending = runScript('checksum', ['abc', '--delay']); + await new Promise((resolve) => setTimeout(resolve, 5)); + expect(process.argv).toBe(argvBefore); + expect(process.exitCode).toBe(exitCodeBefore); + expect(process.stdout.write).toBe(writeBefore); + process.stdout.write('unrelated concurrent stdout\n'); + process.stderr.write('unrelated concurrent stderr\n'); + process.exitCode = 9; + const run = await pending; + process.exitCode = exitCodeBefore; + + expect(run.exitCode).toBe(0); + expect(run.stdout).toBe('Fixture checksum: 3\n'); + expect(run.stderr).toBe(''); + }); + + it('runs two plain scripts at once without either seeing the other\'s output', async () => { + const [banner, checksum] = await Promise.all([ + runScript('banner', ['side', 'by', 'side']), + runScript('checksum', ['abcd', '--delay']), + ]); + + expect(banner.stdout).toBe('banner: side by side\n'); + expect(checksum.stdout).toBe('Fixture checksum: 4\n'); + }); + + it('ends the script at process.exit even when the script catches the call and carries on', async () => { + const run = await runScript('checksum', ['--swallow-exit']); + + expect(run.exitCode).toBe(3); + expect(run.stdout).toBe(''); + }); + + it('never hangs on work the script queued after process.exit', async () => { + const run = await runScript('checksum', ['--exit-then-hang']); + + expect(run.exitCode).toBe(6); + expect(run.stdout).toBe(''); + }); + + it('terminates the script when the signal aborts, and reports the abort', async () => { + const controller = new AbortController(); + const pending = runScript('checksum', ['abc', '--delay'], { signal: controller.signal }); + controller.abort(new Error('stop the script')); + + await expect(pending).rejects.toThrow('stop the script'); + }); + + it('ends a script that never finishes whenever the abort lands: before, while, or after its process is started', async () => { + // Resolving and scanning the source are asynchronous, so an abort can + // land before the process exists, while it is being prepared, or once it + // runs; `--hang` never exits on its own, so a missed abort would hang + // the run. Each timing must reject with the abort reason. + const timings: readonly ((abort: () => void) => void)[] = [ + (abort) => { abort(); }, + (abort) => { queueMicrotask(abort); }, + (abort) => { setImmediate(abort); }, + (abort) => { setTimeout(abort, 1); }, + (abort) => { setTimeout(abort, 5); }, + (abort) => { setTimeout(abort, 25); }, + (abort) => { setTimeout(abort, 150); }, + ]; + for (const [index, schedule] of timings.entries()) { + const controller = new AbortController(); + const pending = runScript('checksum', ['--hang'], { signal: controller.signal }); + schedule(() => { controller.abort(new Error(`stop the hanging script (${String(index)})`)); }); + + await expect(pending).rejects.toThrow(`stop the hanging script (${String(index)})`); + } + }, 15_000); + + it('reaps a script that traps SIGTERM before reporting the abort, so it cannot outlive its run', async () => { + const controller = new AbortController(); + const pending = runScript('checksum', ['--ignore-sigterm'], { signal: controller.signal }); + // Let the script install its handler before asking it to leave. + await new Promise((resolve) => setTimeout(resolve, 400)); + const started = Date.now(); + controller.abort(new Error('stop the trapping script')); + + await expect(pending).rejects.toThrow('stop the trapping script'); + // SIGTERM was trapped; the run settled only after the escalation killed + // the process, within the grace period plus scheduling slack. + expect(Date.now() - started).toBeGreaterThanOrEqual(900); + expect(Date.now() - started).toBeLessThan(4000); + }, 10_000); + + it('hides the harness loader flags from the script: process.execArgv is empty, as under plain node', async () => { + const run = await runScript('checksum', ['--exec-argv']); + + expect(run.exitCode).toBe(0); + expect(run.stdout).toBe('checksum execArgv []\n'); + }); + + it('pipes stdin to a plain script when given, and ends it at once otherwise', async () => { + const fed = await runScript('checksum', ['--stdin'], { stdin: 'piped input\n' }); + expect(fed.exitCode).toBe(0); + expect(fed.stdout).toBe('checksum read 12 byte(s): piped input\n'); + + const empty = await runScript('checksum', ['--stdin']); + expect(empty.exitCode).toBe(0); + expect(empty.stdout).toBe('checksum read 0 byte(s): \n'); + + const error = await runScript('summary', ['x'], { stdin: 'ignored' }).catch((thrown: unknown) => thrown); + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).code).toBe('invalid-input'); + expect((error as AgentTestError).message).toContain('stdin applies to plain (.ts) scripts only'); + }); + + it('reports the status the operating system would for an out-of-range numeric return', async () => { + const run = await runScript('checksum', ['--return=300']); + + expect(run.exitCode).toBe(44); + expect(run.stderr).toBe(''); + }); + + it('exits 1 with the setter\'s own error when main returns a non-integer, as the envelope does', async () => { + const run = await runScript('checksum', ['--return=1.5']); + + expect(run.exitCode).toBe(1); + expect(run.stderr).toContain('RangeError'); + expect(run.stderr).toContain('"code"'); + }); +}); diff --git a/packages/agent-bundle/tests/test-harness-manifest.test.ts b/packages/agent-bundle/tests/test-harness-manifest.test.ts index 5ddcf2f6f..9bbdf07e1 100644 --- a/packages/agent-bundle/tests/test-harness-manifest.test.ts +++ b/packages/agent-bundle/tests/test-harness-manifest.test.ts @@ -72,6 +72,15 @@ describe('the compiled test manifest', () => { 'event:tool/after', 'prompt:harness/summarize', 'resource:harness/notes', + 'script:badge', + 'script:banner', + 'script:blank', + 'script:broken', + 'script:checksum', + 'script:constant', + 'script:identity', + 'script:stalled', + 'script:summary', 'script:tooling-summary', 'tool:harness/catalog', 'tool:harness/context', @@ -132,6 +141,80 @@ describe('the compiled test manifest', () => { source: resolve(fixtureRoot, 'src/state.ts'), }); expect(manifest.targets).toEqual(['claude']); + // Script descriptors carry the extension contract: `.tsx` renders, `.ts` + // is a plain executable module; the name is the path-derived identity. + expect(manifest.scripts).toEqual([ + { + name: 'badge', + relativePath: 'src/scripts/badge.ts', + rendered: false, + routeId: 'script:badge', + source: resolve(fixtureRoot, 'src/scripts/badge.ts'), + }, + { + name: 'banner', + relativePath: 'src/scripts/banner.ts', + rendered: false, + routeId: 'script:banner', + source: resolve(fixtureRoot, 'src/scripts/banner.ts'), + }, + { + name: 'blank', + relativePath: 'src/scripts/blank.tsx', + rendered: true, + routeId: 'script:blank', + source: resolve(fixtureRoot, 'src/scripts/blank.tsx'), + }, + { + name: 'broken', + relativePath: 'src/scripts/broken.tsx', + rendered: true, + routeId: 'script:broken', + source: resolve(fixtureRoot, 'src/scripts/broken.tsx'), + }, + { + name: 'checksum', + relativePath: 'src/scripts/checksum.ts', + rendered: false, + routeId: 'script:checksum', + source: resolve(fixtureRoot, 'src/scripts/checksum.ts'), + }, + { + name: 'constant', + relativePath: 'src/scripts/constant.ts', + rendered: false, + routeId: 'script:constant', + source: resolve(fixtureRoot, 'src/scripts/constant.ts'), + }, + { + name: 'identity', + relativePath: 'src/scripts/identity.ts', + rendered: false, + routeId: 'script:identity', + source: resolve(fixtureRoot, 'src/scripts/identity.ts'), + }, + { + name: 'stalled', + relativePath: 'src/scripts/stalled.tsx', + rendered: true, + routeId: 'script:stalled', + source: resolve(fixtureRoot, 'src/scripts/stalled.tsx'), + }, + { + name: 'summary', + relativePath: 'src/scripts/summary.tsx', + rendered: true, + routeId: 'script:summary', + source: resolve(fixtureRoot, 'src/scripts/summary.tsx'), + }, + { + name: 'tooling-summary', + relativePath: 'src/scripts/tooling-summary.tsx', + rendered: true, + routeId: 'script:tooling-summary', + source: resolve(fixtureRoot, 'src/scripts/tooling-summary.tsx'), + }, + ]); expect(manifest.apps).toEqual({ panel: { id: 'mcp-app:harness:panel', @@ -306,6 +389,36 @@ describe('the compiled test manifest', () => { expect(Object.isFrozen(projected.routes)).toBe(true); }); + it('lists only the conventional scripts normalization ships, never a nested or configuration-conflicting route', async () => { + const graph = await compileRouteGraph(fixtureRoot, { targets: ['claude'] } as never); + const checksum = graph.scripts.find((route) => route.id === 'script:checksum'); + if (checksum === undefined) throw new Error('fixture must compile script:checksum'); + const nested = { + ...checksum, + id: 'script:release/verify', + provenance: { ...checksum.provenance, relativePath: 'src/scripts/release/verify.ts' }, + source: resolve(fixtureRoot, 'src/scripts/release/verify.ts'), + }; + const projected = testManifestFromRouteGraph({ + graph: { ...graph, scripts: [...graph.scripts, nested] }, + projectRoot: fixtureRoot, + scripts: [{ + id: 'script:banner', + mode: 'bundle', + name: 'banner', + provenance: { kind: 'config', sourcePath: resolve(fixtureRoot, 'agent-bundle.config.ts') }, + source: resolve(fixtureRoot, 'tools/banner.ts'), + targets: ['claude'], + }], + }); + + // `banner` is claimed by configuration (AB4809) and `release/verify` is + // nested (AB4808): neither becomes a scripts/.mjs executable, so + // neither is a script-dispatch target. + expect(projected.scripts.map((script) => script.name)).toEqual(['badge', 'blank', 'broken', 'checksum', 'constant', 'identity', 'stalled', 'summary', 'tooling-summary']); + expect(manifest.scripts.map((script) => script.name)).toEqual(['badge', 'banner', 'blank', 'broken', 'checksum', 'constant', 'identity', 'stalled', 'summary', 'tooling-summary']); + }); + it('rejects a shared app name whose compile-relevant declaration differs', async () => { const graph = await compileRouteGraph(fixtureRoot, { targets: ['claude'] } as never); const app = { diff --git a/packages/agent-bundle/tests/workbench-surface-dev-server.test.ts b/packages/agent-bundle/tests/workbench-surface-dev-server.test.ts new file mode 100644 index 000000000..3d387c05f --- /dev/null +++ b/packages/agent-bundle/tests/workbench-surface-dev-server.test.ts @@ -0,0 +1,135 @@ +import { mkdir, rm, symlink, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; + +import { expect, it } from '@rstest/core'; + +import type { LifecycleListResponse } from '../src/contracts/lifecycles.ts'; +import type { RouteManifestResponse } from '../src/dev/routes/route-manifest.ts'; +import { createWorkbenchAssetSource } from '../src/dev/workbench-assets.ts'; +import { startDevServer } from '../src/dev/workbench-server.ts'; +import { inspectWorkbenchSurface } from '../src/test/index.ts'; +import { createProjectFixture } from './helpers/project-fixture.ts'; +import { agentBundleNodeModules } from './helpers/workspace-paths.ts'; + +/** + * The workbench-surface level claims to hand a consumer exactly what the dev + * server serves the Workbench. This proves the claim against a real dev + * server: the route manifest body and the lifecycle inventory the browser + * would fetch are byte-equivalent to the helper's in-process projection. + */ +it('matches the route manifest and lifecycle inventory a real dev server serves', { timeout: 60_000 }, async () => { + const project = await createProjectFixture({ + config: [ + 'export default {', + " plugin: { name: 'workbench-surface-dev-server', version: '1.0.0' },", + " targets: ['claude'],", + '};', + '', + ].join('\n'), + files: { + 'package.json': '{"type":"module"}\n', + 'src/cli/greet.ts': [ + "import { z } from 'zod';", + '', + "export const config = { description: 'Greets one name.', positionals: ['name'] };", + "export const inputSchema = z.object({ loud: z.boolean().optional(), name: z.string().min(1) }).strict();", + 'export const resultSchema = z.object({ message: z.string() }).strict();', + '', + 'export default async function greet({ input }) {', + ' return { message: `Hello, ${input.name}${input.loud ? \'!\' : \'.\'}` };', + '}', + '', + ].join('\n'), + 'src/events/tool/after.tsx': [ + "import { Agent } from '@agent-bundle/runtime';", + "import { createElement } from 'react';", + '', + "export const config = { runtime: 'standalone' };", + '', + 'export default async function AfterTool({ canonical }) {', + " return createElement(Agent.Result, null, createElement(Agent.Context, null, `Recorded ${canonical.provenance.host}.`));", + '}', + '', + ].join('\n'), + 'src/mcp/status/tools/report.tsx': [ + "import { Agent } from '@agent-bundle/runtime';", + "import { createElement } from 'react';", + "import { z } from 'zod';", + '', + "export const config = { annotations: { readOnlyHint: true }, description: 'Reports one service.' };", + "export const inputSchema = z.object({ service: z.string().min(1) }).strict();", + 'export const resultSchema = z.object({ service: z.string() }).strict();', + '', + 'export default async function Report({ input }) {', + " return createElement(Agent.Result, { value: { service: input.service } }, createElement(Agent.Text, null, input.service));", + '}', + '', + ].join('\n'), + 'src/providers/clock.ts': [ + 'export default () => ({ now: 0 });', + '', + ].join('\n'), + }, + prefix: 'agent-bundle-workbench-surface-dev-server-', + }); + const assetsRoot = join(project.root, 'workbench'); + let server: Awaited> | undefined; + await mkdir(assetsRoot, { recursive: true }); + await Promise.all([ + symlink(agentBundleNodeModules, join(project.root, 'node_modules'), 'dir'), + writeFile(join(assetsRoot, 'index.html'), 'Workbench surface'), + ]); + try { + const surface = await inspectWorkbenchSurface({ root: project.root }); + server = await startDevServer({ + assets: createWorkbenchAssetSource({ root: assetsRoot }), + open: false, + port: 0, + root: project.root, + }); + const bootstrap = await fetch(`${server.url}/api/project/session`, { + headers: { 'sec-fetch-site': 'same-origin' }, + }); + expect(bootstrap.status).toBe(200); + const { token } = await bootstrap.json() as { readonly token: string }; + const headers = { origin: server.url, 'x-agent-bundle-session': token }; + try { + await expect.poll( + async () => fetch(`${server!.url}/api/routes/manifest`, { headers }).then((response) => response.status), + { timeout: 10_000 }, + ).toBe(200); + } catch (error) { + throw new Error(`Route manifest did not become ready: ${JSON.stringify(server.status())}`, { cause: error }); + } + const [manifestResponse, lifecyclesResponse] = await Promise.all([ + fetch(`${server.url}/api/routes/manifest`, { headers }), + fetch(`${server.url}/api/lifecycles`, { headers }), + ]); + expect(manifestResponse.status).toBe(200); + expect(lifecyclesResponse.status).toBe(200); + const served = await manifestResponse.json() as RouteManifestResponse; + const lifecycles = await lifecyclesResponse.json() as LifecycleListResponse; + + // JSON round-trip on both sides: the wire drops `undefined` members, the + // helper's frozen objects never carry them, and structural equality is + // the claim. + expect(JSON.parse(JSON.stringify(surface.manifest))).toEqual(served.manifest); + expect(JSON.parse(JSON.stringify(surface.lifecycles))).toEqual(lifecycles.lifecycles); + expect(surface.provenance.sourceRevision).toBe(lifecycles.manifestDigest); + expect(surface.catalog.groups.map((group) => group.label)).toEqual([ + 'status · Tools', + 'Event routes', + 'CLI commands', + ]); + expect(surface.catalog.groups[2]?.entries[0]?.commandUsage).toBe('greet [--loud]'); + expect(surface.lifecycles).toMatchObject([{ + event: 'tool/after', + routeId: 'event:tool/after', + targets: [{ nativeEvent: 'PostToolUse', target: 'claude' }], + }]); + expect(surface.pages).toEqual(['overview', 'routes', 'hooks', 'lifecycles', 'hosts', 'mcp', 'artifacts', 'playground', 'logs']); + } finally { + await server?.close().catch(() => undefined); + await rm(project.root, { force: true, maxRetries: 5, recursive: true, retryDelay: 50 }); + } +}); diff --git a/packages/agent-bundle/tests/workbench-surface.test.ts b/packages/agent-bundle/tests/workbench-surface.test.ts new file mode 100644 index 000000000..6bd8e7fde --- /dev/null +++ b/packages/agent-bundle/tests/workbench-surface.test.ts @@ -0,0 +1,432 @@ +import { rm } from 'node:fs/promises'; +import { resolve } from 'node:path'; + +import { describe, expect, it } from '@rstest/core'; + +import { + AgentTestError, + inspectWorkbenchSurface, + workbenchPageLabel, + type WorkbenchRouteCatalogGroup, + type WorkbenchSurface, +} from '../src/test/index.ts'; +import { createProjectFixture } from './helpers/project-fixture.ts'; + +const exampleRoot = (name: string): string => resolve(import.meta.dirname, '../../../examples', name); + +const groupNamed = (surface: WorkbenchSurface, label: string): WorkbenchRouteCatalogGroup => { + const group = surface.catalog.groups.find((candidate) => candidate.label === label); + if (group === undefined) { + throw new Error(`Expected a ${JSON.stringify(label)} group; found ${JSON.stringify(surface.catalog.groups.map((candidate) => candidate.label))}.`); + } + return group; +}; + +const visibleLabels = (surface: WorkbenchSurface): readonly string[] => surface.pages.map(workbenchPageLabel); + +/** + * These assertions are the ones `packages/workbench/tests/examples-real.e2e.test.ts` + * makes against the real Workbench in Chrome ("renders the flagship compiled + * route catalog by server and kind"), restated over the helper's output. The + * two must keep agreeing: the e2e proves the browser shows this, the helper + * proves a consumer can assert it without one. + */ +describe('the Workbench surface of the audiobook curator', () => { + const surfacePromise = inspectWorkbenchSurface({ root: exampleRoot('audiobook-curator') }); + + it('stamps the workbench-surface level and the compiler pass identity', async () => { + const surface = await surfacePromise; + + expect(surface.provenance).toMatchObject({ + manifestDigest: surface.manifest.digest, + projectRoot: exampleRoot('audiobook-curator'), + proofLevel: 'workbench-surface', + sourceRevision: surface.manifest.sourceRevision, + targets: ['claude', 'codex'], + }); + expect(surface.catalog.diagnostics).toEqual([]); + }); + + it('projects the State region the Routes page renders', async () => { + const { catalog } = await surfacePromise; + + expect(catalog.stateDefinition).toMatchObject({ + driver: 'sqlite', + id: 'audiobook-curator/shelf', + lifetime: 'workspace-durable', + source: 'src/state.ts', + }); + }); + + it('groups every MCP kind under the one generated curator server', async () => { + const surface = await surfacePromise; + + expect(surface.catalog.servers).toEqual([{ id: 'mcp:curator', mode: 'generated', name: 'curator', routeCount: 18 }]); + for (const label of ['curator · Tools', 'curator · Resources', 'curator · Prompts']) { + expect(groupNamed(surface, label)).toMatchObject({ mode: 'generated', server: 'curator', serverId: 'mcp:curator' }); + } + const tools = groupNamed(surface, 'curator · Tools'); + expect(tools.entries).toHaveLength(16); + expect(tools.entries.map((entry) => entry.route.id)).toEqual(expect.arrayContaining([ + 'tool:curator/convert_audiobook', + 'tool:curator/inventory_sources', + 'tool:curator/review_curation_shelf', + ])); + const convert = tools.entries.find((entry) => entry.route.id === 'tool:curator/convert_audiobook'); + expect(convert?.route.source).toBe('src/mcp/curator/tools/convert_audiobook.tsx'); + expect(convert?.route.provenance).toEqual({ kind: 'conventional' }); + // The extracted config is summarized, never inlined as nested JSON. + expect(convert?.route.config).toEqual(expect.arrayContaining([{ key: 'annotations', kind: 'object', value: '2 keys' }])); + const inventory = tools.entries.find((entry) => entry.route.id === 'tool:curator/inventory_sources'); + expect(inventory?.route.inputSchema).toMatchObject({ + properties: { report: expect.anything(), source: expect.anything(), strict: { type: 'boolean' } }, + required: ['source'], + }); + + expect(groupNamed(surface, 'curator · Resources').entries.map((entry) => entry.route.id)).toContain('resource:curator/catalog'); + expect(groupNamed(surface, 'curator · Resources').entries.find((entry) => entry.route.id === 'resource:curator/catalog')?.route.config) + .toEqual(expect.arrayContaining([{ key: 'uri', kind: 'string', value: 'audiobook-curator://catalog' }])); + expect(groupNamed(surface, 'curator · Prompts').entries.map((entry) => entry.route.id)).toContain('prompt:curator/curate'); + }); + + it('lists the 16 authored commands beside one projected command per tool', async () => { + const surface = await surfacePromise; + const cli = groupNamed(surface, 'CLI commands'); + const routeIds = cli.entries.map((entry) => entry.route.id); + const authored = routeIds.filter((routeId) => routeId.startsWith('cli:')); + const projected = routeIds.filter((routeId) => routeId.startsWith('tool:')); + + expect(cli.mode).toBe('generated'); + expect(authored).toEqual([ + 'cli:acoustic-identify', + 'cli:acoustic-verify', + 'cli:apply-chapters', + 'cli:apply-metadata', + 'cli:audible-cache', + 'cli:audible-search', + 'cli:audible-select', + 'cli:audit', + 'cli:convert', + 'cli:inspect', + 'cli:inventory', + 'cli:library-audit', + 'cli:prepare', + 'cli:select', + 'cli:shelf', + 'cli:whisper-verify', + ]); + expect(projected).toEqual(groupNamed(surface, 'curator · Tools').entries.map((entry) => entry.route.id)); + expect(new Set(routeIds).size).toBe(routeIds.length); + expect(routeIds).toHaveLength(authored.length + projected.length); + const byId = new Map(cli.entries.map((entry) => [entry.route.id, entry])); + expect(byId.get('cli:library-audit')?.route.source).toBe('src/cli/library-audit.tsx'); + expect(byId.get('cli:shelf')?.route.source).toBe('src/cli/shelf.tsx'); + expect(byId.get('cli:library-audit')?.commandUsage) + .toBe('library-audit [--concurrency ] --report [--strict]'); + expect(byId.get('cli:inspect')?.commandUsage).toBe('inspect [--max-files ]'); + // Projected commands carry their MCP provenance and the annotation-derived + // confirmation policy: read-only tools run without --yes, mutation-capable + // tools fail closed without it. + expect(byId.get('tool:curator/inspect_sources')?.command?.mcp).toEqual({ + confirm: false, + server: 'curator', + tool: 'inspect_sources', + }); + expect(byId.get('tool:curator/convert_audiobook')?.command?.mcp).toEqual({ + confirm: true, + server: 'curator', + tool: 'convert_audiobook', + }); + }); + + it('reports the route graph identity and invents nothing', async () => { + const surface = await surfacePromise; + + // 18 MCP routes plus 16 authored and 16 projected CLI routes. + expect(surface.catalog.routeCount).toBe(50); + expect(surface.catalog.groups.map((group) => group.kind)).not.toContain('event-route'); + expect(surface.catalog.groups.map((group) => group.kind)).not.toContain('script'); + expect(surface.catalog.providers).toEqual([{ id: 'provider:library', name: 'library', source: 'src/providers/library.ts' }]); + expect(surface.lifecycles).toEqual([]); + expect(surface.manifest.events).toEqual([]); + expect(surface.manifest.scripts).toEqual([]); + }); + + it('derives the navigation the Workbench shows for this project', async () => { + const surface = await surfacePromise; + + expect(visibleLabels(surface)).toEqual(expect.arrayContaining(['Overview', 'Routes', 'Skills', 'MCP playground', 'Hosts', 'Artifacts', 'Logs'])); + expect(surface.unavailablePages).toEqual(expect.arrayContaining(['hooks', 'lifecycles', 'playground'])); + // One MCP server shipped to two hosts: two instances, as the artifact inventory lists them. + expect(surface.counts).toMatchObject({ hooks: 0, mcpServers: 2, scripts: 0, skills: 1, targets: 2 }); + }); +}); + +/** + * `examples-real.e2e.test.ts` asserts the MCP App example keeps all nine + * configured pages while its compiled catalog is empty, and that the Skills + * Starter shows no Hooks, MCP playground, or Playground link. + */ +describe('the Workbench surface of the configured-only examples', () => { + it('keeps every configured page while reporting an empty compiled graph for the MCP App example', async () => { + const surface = await inspectWorkbenchSurface({ root: exampleRoot('mcp-app') }); + + expect(surface.catalog.routeCount).toBe(0); + expect(surface.catalog.groups).toEqual([]); + expect(surface.catalog.stateDefinition).toBeUndefined(); + // The rail order of packages/workbench/src/main.tsx, minus the hidden Lifecycles link. + expect(visibleLabels(surface)).toEqual([ + 'Overview', 'Routes', 'Skills', 'Hooks', 'Hosts', 'MCP playground', 'Artifacts', 'Playground', 'Logs', 'Evals', 'Comparisons', + ]); + expect(surface.unavailablePages).toEqual(['lifecycles']); + expect(surface.counts).toMatchObject({ evalSuites: 1, skills: 1, targets: 3 }); + expect(surface.counts.hooks).toBeGreaterThan(0); + expect(surface.counts.mcpServers).toBeGreaterThan(0); + expect(surface.counts.scripts).toBeGreaterThan(0); + }); + + it('hides Hooks, MCP playground, and Playground for the Skills Starter', async () => { + const surface = await inspectWorkbenchSurface({ root: exampleRoot('skills-starter') }); + + for (const hidden of ['Hooks', 'MCP playground', 'Playground']) { + expect(visibleLabels(surface)).not.toContain(hidden); + } + expect(visibleLabels(surface)).toEqual(expect.arrayContaining(['Overview', 'Routes', 'Skills', 'Artifacts', 'Logs'])); + expect(surface.counts).toMatchObject({ hooks: 0, mcpServers: 0, scripts: 0, skills: 3, targets: 3 }); + }); +}); + +/** + * The Workbench counts what the built artifact lists — one instance per + * declaration per target — and hides Hooks and Playground when nothing is + * emitted. A declaration whose `targets` select none of the project's targets + * is declared but emitted nowhere, so it must not count. + */ +describe('capability counts', () => { + it('counts declaration instances per selected target, not declarations', async () => { + const project = await createProjectFixture({ + config: [ + 'export default {', + " plugin: { name: 'workbench-surface-counts', version: '1.0.0' },", + " targets: ['claude', 'codex'],", + " hooks: { PostToolUse: [{ handler: 'src/hooks/audit.ts', targets: [] }] },", + ' scripts: {', + " everywhere: 'src/tools/everywhere.ts',", + " 'codex-only': { entry: 'src/tools/codex-only.ts', targets: ['codex'] },", + " nowhere: { entry: 'src/tools/nowhere.ts', targets: [] },", + ' },', + '};', + '', + ].join('\n'), + files: { + 'package.json': '{"type":"module"}\n', + 'src/hooks/audit.ts': 'export const main = async () => 0;\n', + 'src/tools/codex-only.ts': 'export const main = async () => 0;\n', + 'src/tools/everywhere.ts': 'export const main = async () => 0;\n', + 'src/tools/nowhere.ts': 'export const main = async () => 0;\n', + }, + prefix: 'agent-bundle-workbench-surface-counts-', + }); + try { + const surface = await inspectWorkbenchSurface({ root: project.root }); + + // everywhere × 2 targets + codex-only × 1 + nowhere × 0; the hook selects no target. + expect(surface.counts).toMatchObject({ hooks: 0, mcpServers: 0, scripts: 3, targets: 2 }); + expect(surface.pages).toContain('playground'); + expect(surface.pages).not.toContain('hooks'); + } finally { + await rm(project.root, { force: true, recursive: true }); + } + }); + + it('counts only compiled hook wrappers: a prebuilt hook is never indexed, so it never counts', async () => { + // One prebuilt hook per target (never indexed) beside one compiled hook + // selected for claude alone: the artifact index — and the Workbench — + // holds exactly one entry. + const project = await createProjectFixture({ + config: [ + 'export default {', + " plugin: { name: 'workbench-surface-prebuilt-hooks', version: '1.0.0' },", + " targets: ['claude', 'codex'],", + " payload: { runtime: './built/runtime' },", + ' hooks: { afterTool: [', + " { args: ['--host', 'claude'], handler: { prebuilt: './built/runtime/hook.js' }, targets: ['claude'], tools: ['file.write'] },", + " { args: ['--host', 'codex'], handler: { prebuilt: './built/runtime/hook.js' }, targets: ['codex'], tools: ['file.write'] },", + " { handler: 'src/hooks/audit.ts', targets: ['claude'], tools: ['file.write'] },", + ' ] },', + '};', + '', + ].join('\n'), + files: { + 'built/runtime/hook.js': 'process.stdout.write("{}");\n', + 'package.json': '{"type":"module"}\n', + 'src/hooks/audit.ts': 'export const main = async () => 0;\n', + }, + prefix: 'agent-bundle-workbench-surface-prebuilt-hooks-', + }); + try { + const surface = await inspectWorkbenchSurface({ root: project.root }); + + expect(surface.counts).toMatchObject({ hooks: 1, targets: 2 }); + expect(surface.pages).toContain('hooks'); + + const prebuiltOnly = await createProjectFixture({ + config: [ + 'export default {', + " plugin: { name: 'workbench-surface-prebuilt-only', version: '1.0.0' },", + " targets: ['claude'],", + " payload: { runtime: './built/runtime' },", + " hooks: { afterTool: [{ args: ['--host', 'claude'], handler: { prebuilt: './built/runtime/hook.js' }, targets: ['claude'], tools: ['file.write'] }] },", + '};', + '', + ].join('\n'), + files: { + 'built/runtime/hook.js': 'process.stdout.write("{}");\n', + 'package.json': '{"type":"module"}\n', + }, + prefix: 'agent-bundle-workbench-surface-prebuilt-only-', + }); + try { + const hidden = await inspectWorkbenchSurface({ root: prebuiltOnly.root }); + + expect(hidden.counts).toMatchObject({ hooks: 0, targets: 1 }); + expect(hidden.pages).not.toContain('hooks'); + expect(hidden.unavailablePages).toContain('hooks'); + } finally { + await rm(prebuiltOnly.root, { force: true, recursive: true }); + } + } finally { + await rm(project.root, { force: true, recursive: true }); + } + }); + + it('hides Playground and Hooks when every declaration selects no target', async () => { + const project = await createProjectFixture({ + config: [ + 'export default {', + " plugin: { name: 'workbench-surface-nothing-emitted', version: '1.0.0' },", + " targets: ['claude'],", + " hooks: { PostToolUse: [{ handler: 'src/hooks/audit.ts', targets: [] }] },", + " scripts: { nowhere: { entry: 'src/tools/nowhere.ts', targets: [] } },", + '};', + '', + ].join('\n'), + files: { + 'package.json': '{"type":"module"}\n', + 'src/hooks/audit.ts': 'export const main = async () => 0;\n', + 'src/tools/nowhere.ts': 'export const main = async () => 0;\n', + }, + prefix: 'agent-bundle-workbench-surface-nothing-', + }); + try { + const surface = await inspectWorkbenchSurface({ root: project.root }); + + expect(surface.counts).toMatchObject({ hooks: 0, scripts: 0, targets: 1 }); + expect(surface.unavailablePages).toEqual(expect.arrayContaining(['hooks', 'playground'])); + } finally { + await rm(project.root, { force: true, recursive: true }); + } + }); +}); + +/** + * The Workbench server prepares a project in development mode from the + * configuration file it was pointed at; the helper must prepare the same + * way, or a configuration that branches on the mode or lives at a + * non-default path projects a surface the Workbench never shows. + */ +describe('preparation parity with the Workbench server', () => { + it('prepares in development mode and discovers eval suites through the selected configuration', async () => { + const assertionsModule = resolve(import.meta.dirname, '../src/eval/assertions.ts'); + const suiteModule = resolve(import.meta.dirname, '../src/eval/suite.ts'); + const configFactory = (name: string, evalsDir: string): string => [ + 'export default (context) => ({', + ` plugin: { name: ${JSON.stringify(name)}, version: '1.0.0' },`, + // Production selects claude alone; the Workbench (development) sees both. + " targets: context.mode === 'development' ? ['claude', 'codex'] : ['claude'],", + ` evals: { include: [${JSON.stringify(`${evalsDir}/**/*.eval.ts`)}] },`, + '});', + '', + ].join('\n'); + const suiteSource = [ + `import { expectExitCode } from ${JSON.stringify(assertionsModule)};`, + `import { defineEvalSuite } from ${JSON.stringify(suiteModule)};`, + '', + 'export default defineEvalSuite({', + ' cases: [{', + ' assertions: [expectExitCode(0)],', + " fixture: './fixtures/repo',", + " hosts: { claude: { model: 'claude-sonnet-4-5' } },", + " id: 'case-a',", + " invocation: { mode: 'automatic' },", + " prompt: 'Do the task.',", + ' }],', + " name: 'review-change',", + '});', + '', + ].join('\n'); + // The default configuration finds no suite; only the selected one does. + const project = await createProjectFixture({ + config: configFactory('workbench-surface-default-config', 'nowhere'), + files: { + 'checks/review.eval.ts': suiteSource, + 'package.json': '{"type":"module"}\n', + 'workbench.config.ts': configFactory('workbench-surface-selected-config', 'checks'), + }, + prefix: 'agent-bundle-workbench-surface-preparation-', + }); + try { + const surface = await inspectWorkbenchSurface({ configPath: 'workbench.config.ts', root: project.root }); + + expect(surface.provenance.configPath).toBe(resolve(project.root, 'workbench.config.ts')); + expect(surface.provenance.targets).toEqual(['claude', 'codex']); + expect(surface.counts).toMatchObject({ evalSuites: 1, targets: 2 }); + expect(surface.pages).toContain('evals'); + + const byDefault = await inspectWorkbenchSurface({ root: project.root }); + expect(byDefault.provenance.configPath).toBe(project.configPath); + expect(byDefault.counts).toMatchObject({ evalSuites: 0, targets: 2 }); + expect(byDefault.unavailablePages).toContain('evals'); + } finally { + await rm(project.root, { force: true, recursive: true }); + } + }); +}); + +describe('an unusable project', () => { + it('reports the manifest unavailable the way the dev server would, with the compiler cause', async () => { + const error = await inspectWorkbenchSurface({ root: resolve(import.meta.dirname, 'fixtures/target-capabilities') }) + .catch((thrown: unknown) => thrown); + + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).code).toBe('manifest-unavailable'); + expect((error as AgentTestError).message).toContain('dev server would report the route manifest unavailable'); + }); + + it('rejects a project that normalizes but fails validation, as the dev server never serves an invalid preparation', async () => { + // An unknown target normalizes into a model and a revision, then + // `validateModel` reports AB4100: source state `invalid`, which the dev + // server never assigns to its served preparation. + const project = await createProjectFixture({ + config: [ + 'export default {', + " plugin: { name: 'workbench-surface-invalid', version: '1.0.0' },", + " targets: ['claude', 'no-such-host'],", + '};', + '', + ].join('\n'), + files: { 'package.json': '{"type":"module"}\n' }, + prefix: 'agent-bundle-workbench-surface-invalid-', + }); + try { + const error = await inspectWorkbenchSurface({ root: project.root }).catch((thrown: unknown) => thrown); + + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).code).toBe('manifest-unavailable'); + expect((error as AgentTestError).message).toContain('source state invalid'); + expect((error as AgentTestError).message).toContain('AB4100'); + } finally { + await rm(project.root, { force: true, recursive: true }); + } + }); +}); diff --git a/packages/create-agent-bundle/README.md b/packages/create-agent-bundle/README.md index 6854077a2..33b90a8cc 100644 --- a/packages/create-agent-bundle/README.md +++ b/packages/create-agent-bundle/README.md @@ -40,20 +40,22 @@ scripted and asks nothing — the remaining values fall back to their defaults. | --- | --- | | `minimal` | A skills-only plugin: one `src/skills//SKILL.md` directory and nothing else. | | `mcp-server` | A stdio MCP server from one `src/mcp//tools/.tsx` route module plus one artifact script, with the framework test harness wired up. | -| `cli-tool` | An installable CLI through the `src/cli.ts` bin convention plus a `src/index.ts` library export with declarations. | +| `cli-tool` | An installable routed CLI from one `src/cli/.ts` route module (generated executable, help, argv grammar, validation), a conventional `src/scripts/.ts` artifact script, and a `src/index.ts` library export with declarations, with the framework test harness wired up. | Every template ships a `check` script (validate + build + typecheck + tests) and validates with zero diagnostics — including the `AB473x` migration nudges, because the templates are written against the entry conventions from the start. -The `mcp-server` template also starts with the consumer test harness: a -route-unit pool (`agentBundleRstest()` from `agent-bundle/rstest`, `renderRoute` -and `expectDocument` from `agent-bundle/test`) and a separate in-memory MCP -projection pool, each labeled with the proof level it carries and run by -`check`. The `minimal` and `cli-tool` templates compile no route modules, so -neither ships a harness pool that would pass without addressing anything; their -READMEs document the wiring to add with the first route. +The `mcp-server` and `cli-tool` templates also start with the consumer test +harness. `mcp-server` ships a route-unit pool (`agentBundleRstest()` from +`agent-bundle/rstest`, `renderRoute` and `expectDocument` from +`agent-bundle/test`) and a separate in-memory MCP projection pool; `cli-tool` +ships one projection pool at the `cli-dispatch` (`invokeCli`, `cliJson`) and +`script-dispatch` (`runScript`) levels. Each pool is labeled with the proof +level it carries and run by `check`. The `minimal` template compiles no route +modules, so it ships no harness pool that would pass without addressing +anything; its README documents the wiring to add with the first route. ## The framework dependency diff --git a/packages/create-agent-bundle/src/options.ts b/packages/create-agent-bundle/src/options.ts index 876f5c1ed..8d36db8a4 100644 --- a/packages/create-agent-bundle/src/options.ts +++ b/packages/create-agent-bundle/src/options.ts @@ -27,7 +27,7 @@ export interface ParsedFlags { } export const templateSummaries: Readonly> = { - 'cli-tool': 'an installable CLI: src/cli.ts bin convention plus a src/index.ts library export', + 'cli-tool': 'an installable routed CLI: src/cli/.ts routes, a src/scripts/.ts script, and a src/index.ts library export', 'mcp-server': 'a stdio MCP server: one conventional src/mcp/.ts entry plus a script', minimal: 'a skills-only plugin: one Skill and nothing else', }; @@ -144,7 +144,7 @@ export interface ProjectName { * (`/^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?$/`, at most 64 characters), which the * unified `plugin` target enforces too and which is also a valid safe * package-output name — so every selectable target validates and the - * `src/cli.ts` bin convention always applies. + * `plugin.name`-derived package executable convention always applies. */ export const formatProjectName = (input: string): ProjectName => { const formatted = input.trim().replace(/\/+$/u, ''); diff --git a/packages/create-agent-bundle/templates/cli-tool/README.md b/packages/create-agent-bundle/templates/cli-tool/README.md index 67c4263b6..34f72e523 100644 --- a/packages/create-agent-bundle/templates/cli-tool/README.md +++ b/packages/create-agent-bundle/templates/cli-tool/README.md @@ -1,21 +1,29 @@ # my-agent-plugin -A command-line tool and library built with +A routed command-line tool and library built with [agent-bundle](https://github.com/ScriptedAlchemy/agent-bundle). There is no -second bundler config and no hand-written bin shim: the `src/cli.ts` -convention makes the executable `dist/bin/my-agent-plugin.js`, the -`src/index.ts` convention makes the library export with declarations, and one -`agent-bundle build` produces both alongside the host artifacts. +second bundler config, no hand-written bin shim, and no argv parser: the +`src/cli/**` convention compiles each command module into one generated +executable, `dist/bin/my-agent-plugin.js`, with help, argv grammar, input +validation, and exit codes derived from the module's own `config` and zod +schemas. `src/index.ts` is the library export with declarations, and +`src/scripts/hello.ts` ships as a plain script inside every host artifact. One +`agent-bundle build` produces all of it alongside the host artifacts. ## Commands ```sh -npm run dev # local workbench with live rebuilds -npm run build # dist/ package build + host artifacts in artifact/ -npm run check # validate + build + typecheck + test +npm run dev # local workbench with live rebuilds +npm run build # dist/ package build + host artifacts in artifact/ +npm run check # validate + build + typecheck + both test pools +npm run test # plain module tests +npm run test:projection # cli-dispatch + script-dispatch pool # after a build -node dist/bin/my-agent-plugin.js World +node dist/bin/my-agent-plugin.js greet World +node dist/bin/my-agent-plugin.js greet World --shout +node dist/bin/my-agent-plugin.js greet --help +node artifact/portable/scripts/hello.mjs World # after publishing/installing the package npx my-agent-plugin-install install claude @@ -26,74 +34,41 @@ Installing the npm package does not mutate any host; run the generated ## Layout -- `agent-bundle.config.ts` — the one typed config; the CLI is also declared - as a script so it ships inside every host artifact. -- `src/cli.ts` — the whole CLI entry: export `main`, the framework generates - the process envelope and the executable bundle. +- `agent-bundle.config.ts` — the one typed config: plugin identity and + targets. Commands, the script, and the library are discovered by convention. +- `src/cli/greet.ts` — the `greet` command: static `config`, `inputSchema`, + `resultSchema`, and an async default function. The file path is the command + name; nesting (`src/cli/library/audit.ts`) becomes `library audit`. A `.tsx` + command renders through the Agent renderer with Markdown, TTY, `--json`, and + `--ndjson` output modes. +- `src/scripts/hello.ts` — a conventional plain script exporting `main(argv)`; + the framework generates the process envelope and `scripts/hello.mjs`. - `src/index.ts` — the library export (`dist/index.js` + `dist/index.d.ts`). -- `tests/` — run with `npm run test`. +- `rstest.projection.config.ts` — the framework-generated projection pool. +- `tests/` — see below. ## Tests -`npm run test` runs ordinary module tests against `src/cli.ts` and -`src/index.ts`, and `npm run check` runs them after validate, build, and -typecheck. - -This template deliberately ships **no** framework test pool. The framework's -consumer harness (`agent-bundle/rstest` + `agent-bundle/test`) addresses -*compiled routes*, and a CLI declared in `agent-bundle.config.ts` under -`scripts:` is a bundled entry, not a route: the compiler hands that module to -the script bundler, so this project compiles zero routes and there would be -nothing for `renderRoute` to render. A pool asserting that is vacuous, and a -vacuous pass is worse than no pool. - -Adopt the harness when the project grows a routed surface: - -- Plain `src/cli/**/*.ts` and rendered `src/cli/**/*.tsx` command routes make - `invokeCli` / `cliJson` (the `cli-dispatch` level) meaningful — argv resolves - and runs through the routed CLI's own shell. Rendered routes can additionally - assert Markdown, explicit TTY, JSON, and NDJSON output; use `cliNdjson` for - the ordered render-event stream. This level remains in-process, so use the - packed CLI route suite for worker-thread, process-framing, executable, and - chunk-by-chunk Flight streaming evidence. - This template ships the conventional `src/cli.ts` entry (and a matching - `scripts` entry in `agent-bundle.config.ts`). Adding command routes while - that file remains triggers `AB4801`. Before creating `src/cli/**` modules, - remove `src/cli.ts`, drop the `./src/cli.ts` script entry, and port any - behavior into route modules — routed commands compile into - `dist/bin/.js` on their own. To keep the single-file CLI - instead, set `routes: { cli: 'conventional' }` and do not add `src/cli/**`. -- `src/mcp//**` route modules make `renderRoute` (`route-unit`) and - `invokeMcpTool` (`mcp-in-memory`) meaningful. - -Then add a pool with the generated configuration and keep it out of the plain -run: - -```ts -// rstest.route-unit.config.ts -import { defineConfig } from '@rstest/core'; -import { agentBundleRstest } from 'agent-bundle/rstest'; - -export default defineConfig(await agentBundleRstest()); -``` - -```json -"test": "rstest tests --exclude \"tests/route-unit/**\"", -"test:routes": "rstest --config rstest.route-unit.config.ts" -``` - -Route rendering needs `react`, `zod`, and `@agent-bundle/runtime` (the same -packages the generated entries import) plus `@rstest/core`; install them -alongside the first route module. Routed commands export zod-based -`inputSchema` and `resultSchema`, so the scaffold cannot typecheck without -`zod` once `src/cli/**` modules exist. The `mcp-server` template ships this -wiring already. - -## The agent-bundle dependency - -agent-bundle has no npm release yet; this project pins a -[pkg.pr.new](https://pkg.pr.new) preview tarball of it. To move to a newer -preview (or a real release once one exists), change the `agent-bundle` entry -in `devDependencies` — see -[Preview packages](https://github.com/ScriptedAlchemy/agent-bundle/blob/main/docs/preview-packages.md) -for the URL forms. +Two pools ship, and each one names the proof level it carries. A pass at one +level is never a receipt for another, so they run — and are reported — +separately. `npm run check` runs both. + +| pool | command | files | what a pass proves | +| --- | --- | --- | --- | +| plain | `npm run test` | `tests/*.test.ts` | ordinary module tests over `src/index.ts`; no framework involved | +| projection (`cli-dispatch`) | `npm run test:projection` | `tests/projection/cli-dispatch.test.ts` | argv resolved and executed through the routed CLI's own shell over the compiled command graph — help, grammar, validation, exit codes — in-process; not the spawned executable | +| projection (`script-dispatch`) | `npm run test:projection` | `tests/projection/script-dispatch.test.ts` | `src/scripts/hello.ts` run through its generated executable's `main` envelope contract with captured stdout/stderr and exit code — as a Node process of its own over the source, not the bundled `scripts/hello.mjs` | + +The projection pool is generated by `agentBundleRstest()` from +`agent-bundle/rstest`: one compiler pass — the same route compilation the build +performs, with no artifact built — supplies the command graph, the script +inventory, and the route loaders. The helpers (`invokeCli`, `cliJson`, +`runScript`, `testManifest`, …) come from `agent-bundle/test`, and every one of +them stamps its proof level into the result and into any failure message. + +Process-level evidence — the executable and the artifact script as real +processes — comes from `npm run build`; the scaffolder's own release matrix +runs both. + +The scaffold pins matching `agent-bundle` and `@agent-bundle/runtime` builds. +For preview URL forms, see [Preview packages](https://github.com/ScriptedAlchemy/agent-bundle/blob/main/docs/preview-packages.md). diff --git a/packages/create-agent-bundle/templates/cli-tool/agent-bundle.config.ts b/packages/create-agent-bundle/templates/cli-tool/agent-bundle.config.ts index d59b34bcb..81eab410f 100644 --- a/packages/create-agent-bundle/templates/cli-tool/agent-bundle.config.ts +++ b/packages/create-agent-bundle/templates/cli-tool/agent-bundle.config.ts @@ -9,12 +9,10 @@ export default defineConfig({ // Optional: move the build artifact root (default `dist`); the CLI // `--output` flag still wins. // output: { distPath: 'artifact' }, - // One CLI bundle, two destinations: `src/cli.ts` is the package bin by - // convention, and declaring it as a script also ships it inside every - // host artifact. `src/index.ts` becomes the library export with - // declarations, also by convention. - scripts: { - 'my-agent-plugin': './src/cli.ts', - }, + // No `bin` or `scripts` fields needed: the routed `src/cli/**` commands + // compile into the package executable (dist/bin/my-agent-plugin.js), the + // conventional `src/scripts/hello.ts` ships as `scripts/hello.mjs` inside + // every host artifact, and `src/index.ts` becomes the library export with + // declarations — all by convention. targets: ['portable', 'codex', 'claude'], }); diff --git a/packages/create-agent-bundle/templates/cli-tool/package_json b/packages/create-agent-bundle/templates/cli-tool/package_json index ecfffcf92..a30e97961 100644 --- a/packages/create-agent-bundle/templates/cli-tool/package_json +++ b/packages/create-agent-bundle/templates/cli-tool/package_json @@ -1,7 +1,7 @@ { "name": "my-agent-plugin", "version": "0.1.0", - "description": "A command-line tool and library built with agent-bundle.", + "description": "A routed command-line tool and library built with agent-bundle.", "type": "module", "engines": { "node": ">=22.19.0" @@ -23,10 +23,11 @@ }, "scripts": { "build": "agent-bundle build --json --output artifact", - "check": "npm run validate && npm run build && npm run typecheck && npm run test", + "check": "npm run validate && npm run build && npm run typecheck && npm run test && npm run test:projection", "dev": "agent-bundle dev", "prepack": "agent-bundle prepack --json --output artifact", - "test": "rstest tests", + "test": "rstest tests --exclude \"tests/projection/**\"", + "test:projection": "rstest --config rstest.projection.config.ts", "typecheck": "tsc -p tsconfig.json --noEmit", "validate": "agent-bundle validate --json" }, @@ -35,5 +36,10 @@ "@types/node": "26.4.0", "agent-bundle": "workspace:*", "typescript": "7.0.2" + }, + "dependencies": { + "@agent-bundle/runtime": "workspace:*", + "react": "19.2.8", + "zod": "4.4.3" } } diff --git a/packages/create-agent-bundle/templates/cli-tool/rstest.projection.config.ts b/packages/create-agent-bundle/templates/cli-tool/rstest.projection.config.ts new file mode 100644 index 000000000..d5084ab4d --- /dev/null +++ b/packages/create-agent-bundle/templates/cli-tool/rstest.projection.config.ts @@ -0,0 +1,14 @@ +import { defineConfig } from '@rstest/core'; +import { agentBundleRstest } from 'agent-bundle/rstest'; + +/** + * The framework-generated projection pool (`tests/projection/**`): the + * `cli-dispatch` and `script-dispatch` proof levels. One Agent Bundle compiler + * pass runs here — the same route compilation the build performs, with no + * artifact built — and supplies the command graph, the script inventory, and + * the route loaders. It is a separate run from the plain `rstest tests` pool + * so the two claims stay separately reported. + */ +export default defineConfig(await agentBundleRstest({ + include: ['tests/projection/**/*.test.ts'], +})); diff --git a/packages/create-agent-bundle/templates/cli-tool/src/cli.ts b/packages/create-agent-bundle/templates/cli-tool/src/cli.ts deleted file mode 100644 index 4a97fb07a..000000000 --- a/packages/create-agent-bundle/templates/cli-tool/src/cli.ts +++ /dev/null @@ -1,29 +0,0 @@ -import { greet } from './index.js'; - -const usage = 'Usage: my-agent-plugin \n'; - -/** Injectable writer so tests can capture output without a child process. */ -export const runCli = ( - argv: readonly string[], - write: (line: string) => void = (line) => { process.stdout.write(line); }, -): 0 | 2 => { - const [name, ...rest] = argv; - if (name === '--help' || name === '-h') { - write(usage); - return 0; - } - if (name === undefined || rest.length > 0) { - write(usage); - return 2; - } - write(`${greet(name).message}\n`); - return 0; -}; - -/** - * `agent-bundle build` detects the `main` export and generates the process - * envelope around it. The same module is the package bin (`src/cli.ts` - * convention → `dist/bin/my-agent-plugin.js`) and, because the config also - * declares it as a script, an executable inside every host artifact. - */ -export const main = async (argv: readonly string[]): Promise => runCli(argv); diff --git a/packages/create-agent-bundle/templates/cli-tool/src/cli/greet.ts b/packages/create-agent-bundle/templates/cli-tool/src/cli/greet.ts new file mode 100644 index 000000000..2934fc6f0 --- /dev/null +++ b/packages/create-agent-bundle/templates/cli-tool/src/cli/greet.ts @@ -0,0 +1,32 @@ +import type { CliRouteConfig, CliRouteProps } from 'agent-bundle'; +import { z } from 'zod'; + +import { greet } from '../index.js'; + +/** + * A routed CLI command: the file path is the command name (`my-agent-plugin + * greet`), the static `config` and `inputSchema` compile into the argv + * grammar and generated help, and `resultSchema` validates what the command + * prints as one canonical JSON line. No argv parsing lives in this file. + */ +export const config = { + description: 'Greet one person by name.', + positionals: ['name'], +} satisfies CliRouteConfig; + +export const inputSchema = z.object({ + name: z.string().min(1).describe('Who to greet.'), + shout: z.boolean().optional().describe('Upper-case the greeting.'), +}).strict(); + +export const resultSchema = z.object({ + message: z.string(), + name: z.string(), +}).strict(); + +export default async function greetCommand({ input }: CliRouteProps) { + const greeting = greet(input.name); + return input.shout === true + ? { ...greeting, message: greeting.message.toUpperCase() } + : greeting; +} diff --git a/packages/create-agent-bundle/templates/cli-tool/src/scripts/hello.ts b/packages/create-agent-bundle/templates/cli-tool/src/scripts/hello.ts new file mode 100644 index 000000000..d74c28ecc --- /dev/null +++ b/packages/create-agent-bundle/templates/cli-tool/src/scripts/hello.ts @@ -0,0 +1,17 @@ +import { greet } from '../index.js'; + +/** + * A conventional plain script: `agent-bundle build` compiles it to + * `scripts/hello.mjs` inside every host artifact and, because the module + * exports `main`, wraps it in the framework process envelope — argv in, a + * numeric return adopted as the exit code, ordinary stdout/stderr semantics. + */ +export const main = async (argv: readonly string[]): Promise => { + const [name, ...rest] = argv; + if (name === undefined || rest.length > 0) { + process.stderr.write('Usage: hello \n'); + return 2; + } + process.stdout.write(`${greet(name).message}\n`); + return 0; +}; diff --git a/packages/create-agent-bundle/templates/cli-tool/tests/cli.test.ts b/packages/create-agent-bundle/templates/cli-tool/tests/cli.test.ts deleted file mode 100644 index 7c0629973..000000000 --- a/packages/create-agent-bundle/templates/cli-tool/tests/cli.test.ts +++ /dev/null @@ -1,22 +0,0 @@ -import { describe, expect, it } from '@rstest/core'; - -import { runCli } from '../src/cli.js'; -import { greet } from '../src/index.js'; - -describe('my-agent-plugin', () => { - it('greets a name and exits zero', () => { - const lines: string[] = []; - expect(runCli(['World'], (line) => lines.push(line))).toBe(0); - expect(lines).toEqual(['Hello, World!\n']); - }); - - it('prints usage and exits 2 without arguments', () => { - const lines: string[] = []; - expect(runCli([], (line) => lines.push(line))).toBe(2); - expect(lines[0]).toContain('Usage:'); - }); - - it('rejects blank names in the library export', () => { - expect(() => greet(' ')).toThrow('A name is required.'); - }); -}); diff --git a/packages/create-agent-bundle/templates/cli-tool/tests/greet.test.ts b/packages/create-agent-bundle/templates/cli-tool/tests/greet.test.ts new file mode 100644 index 000000000..b20451170 --- /dev/null +++ b/packages/create-agent-bundle/templates/cli-tool/tests/greet.test.ts @@ -0,0 +1,13 @@ +import { describe, expect, it } from '@rstest/core'; + +import { greet } from '../src/index.js'; + +describe('my-agent-plugin library', () => { + it('greets a trimmed name', () => { + expect(greet(' World ')).toEqual({ message: 'Hello, World!', name: 'World' }); + }); + + it('rejects blank names', () => { + expect(() => greet(' ')).toThrow('A name is required.'); + }); +}); diff --git a/packages/create-agent-bundle/templates/cli-tool/tests/projection/cli-dispatch.test.ts b/packages/create-agent-bundle/templates/cli-tool/tests/projection/cli-dispatch.test.ts new file mode 100644 index 000000000..a5a5ff573 --- /dev/null +++ b/packages/create-agent-bundle/templates/cli-tool/tests/projection/cli-dispatch.test.ts @@ -0,0 +1,50 @@ +import { expect, it } from '@rstest/core'; +import { cliJson, invokeCli, testManifest } from 'agent-bundle/test'; + +/** + * The cli-dispatch proof level: argv resolved and executed through the routed + * CLI's own shell — command resolution, the compiled argv grammar, generated + * help, input validation, and exit-code mapping are the product's — in this + * process. Nothing is bundled or spawned; the generated + * `dist/bin/my-agent-plugin.js` executable is proven by `npm run build`. + */ +it('compiles the greet command without a build', () => { + const manifest = testManifest(); + + expect(manifest.diagnostics.filter((diagnostic) => diagnostic.severity === 'error')).toEqual([]); + expect(manifest.cliCommands.map((command) => command.path.join(' '))).toEqual(['greet']); + expect(manifest.cliCommands[0]?.options.map((option) => option.option)).toEqual(['name', 'shout']); +}); + +it('greets through the routed CLI shell and prints one canonical JSON line', async () => { + const run = await invokeCli(['greet', 'World']); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + expect(cliJson(run)).toEqual({ message: 'Hello, World!', name: 'World' }); + expect(run.provenance.proofLevel).toBe('cli-dispatch'); +}); + +it('applies boolean flags from the compiled argv grammar', async () => { + const run = await invokeCli(['greet', 'World', '--shout']); + + expect(run.exitCode).toBe(0); + expect(cliJson(run)).toEqual({ message: 'HELLO, WORLD!', name: 'World' }); +}); + +it('maps a missing name to a usage failure with generated help', async () => { + const run = await invokeCli(['greet']); + + expect(run.exitCode).toBe(2); + expect(run.stdout).toBe(''); + expect(run.stderr).toContain('Missing required argument: .'); + expect(run.stderr).toContain("Run 'my-agent-plugin greet --help' for usage."); +}); + +it('documents the command from its schema', async () => { + const run = await invokeCli(['greet', '--help']); + + expect(run.exitCode).toBe(0); + expect(run.stdout).toContain('Usage: my-agent-plugin greet [options] '); + expect(run.stdout).toContain('Upper-case the greeting.'); +}); diff --git a/packages/create-agent-bundle/templates/cli-tool/tests/projection/script-dispatch.test.ts b/packages/create-agent-bundle/templates/cli-tool/tests/projection/script-dispatch.test.ts new file mode 100644 index 000000000..cc2372a89 --- /dev/null +++ b/packages/create-agent-bundle/templates/cli-tool/tests/projection/script-dispatch.test.ts @@ -0,0 +1,31 @@ +import { expect, it } from '@rstest/core'; +import { runScript, testManifest } from 'agent-bundle/test'; + +/** + * The script-dispatch proof level: the conventional `src/scripts/hello.ts` + * module run through the contract its generated `scripts/hello.mjs` carries — + * `main(argv)` awaited, a numeric return adopted as the exit code, stdout and + * stderr captured — as a Node process of its own over the source, with fresh + * module state. The bundled artifact script itself is proven by + * `npm run build`. + */ +it('compiles the hello script as a plain executable module', () => { + expect(testManifest().scripts).toMatchObject([{ name: 'hello', rendered: false }]); +}); + +it('greets through the main process envelope', async () => { + const run = await runScript('hello', ['World']); + + expect(run.exitCode).toBe(0); + expect(run.stdout).toBe('Hello, World!\n'); + expect(run.stderr).toBe(''); + expect(run.provenance).toMatchObject({ execution: 'main-envelope', proofLevel: 'script-dispatch' }); +}); + +it('exits 2 with usage on stderr without a name', async () => { + const run = await runScript('hello'); + + expect(run.exitCode).toBe(2); + expect(run.stdout).toBe(''); + expect(run.stderr).toBe('Usage: hello \n'); +}); diff --git a/packages/create-agent-bundle/templates/cli-tool/tsconfig.json b/packages/create-agent-bundle/templates/cli-tool/tsconfig.json index d94375c56..6c032361a 100644 --- a/packages/create-agent-bundle/templates/cli-tool/tsconfig.json +++ b/packages/create-agent-bundle/templates/cli-tool/tsconfig.json @@ -15,6 +15,7 @@ }, "include": [ "agent-bundle.config.ts", + "rstest.projection.config.ts", "src/**/*.ts", "tests/**/*.ts" ] diff --git a/packages/create-agent-bundle/tests/scaffold-packed-matrix.e2e.test.ts b/packages/create-agent-bundle/tests/scaffold-packed-matrix.e2e.test.ts index 3ccf68225..d1a9f3130 100644 --- a/packages/create-agent-bundle/tests/scaffold-packed-matrix.e2e.test.ts +++ b/packages/create-agent-bundle/tests/scaffold-packed-matrix.e2e.test.ts @@ -74,28 +74,44 @@ it.concurrent('scaffolds the mcp-server template and serves the conventional ent }); }, 600_000); -it.concurrent('scaffolds the cli-tool template with a framework-built bin, lib, and artifact script', async () => { +it.concurrent('scaffolds the cli-tool template with a routed bin, lib, and artifact script', async () => { const projectRoot = await scaffoldProject('cli-tool', 'greeter', ['--no-install']); await execFile('npm', ['install', ...npmInstallArguments], { cwd: projectRoot, env: installedEnvironment(), }); - // This template ships no framework test pool on purpose: its CLI is a - // config-declared script bundle rather than a compiled route, so there is - // nothing for the route-unit or cli-dispatch levels to address (README: - // "Tests"). What `check` does run is asserted rather than assumed. + // The template's own harness pool ran inside `check`, and it is asserted + // positively — a silent `check` would also pass if the pool were dropped or + // matched no files. Each pool names the proof level it carries. const checked = await npmRun(projectRoot, 'check'); - expect(checked).toContain('tests/cli.test.ts'); + expect(checked).toContain('tests/greet.test.ts'); + expect(checked).toContain('tests/projection/cli-dispatch.test.ts'); + expect(checked).toContain('tests/projection/script-dispatch.test.ts'); await expectCleanValidate(projectRoot); await npmRun(projectRoot, 'prepack'); + // The projection pool dispatches through the framework's own generated + // setup, resolved from the packed tarball's `agent-bundle/rstest` export. + const projection = await npmRun(projectRoot, 'test:projection'); + expect(projection).toContain('greets through the routed CLI shell and prints one canonical JSON line'); + expect(projection).toContain('greets through the main process envelope'); + expect(projection).toContain('"failedTests": 0'); - // The src/cli.ts convention produced the executable package bin. + // The src/cli/** convention produced the routed executable package bin: + // generated help, the compiled argv grammar, and one canonical JSON line. const bin = join(projectRoot, 'dist', 'bin', 'greeter.js'); expect((await stat(bin)).mode & 0o111).not.toBe(0); expect((await readFile(bin, 'utf8')).startsWith('#!/usr/bin/env node\n')).toBe(true); - await expect(execFile(bin, ['World'], { cwd: projectRoot, env: installedEnvironment() })) - .resolves.toMatchObject({ stdout: 'Hello, World!\n' }); + const environment = installedEnvironment(); + const help = await execFile(bin, ['--help'], { cwd: projectRoot, env: environment }); + expect(help.stdout).toContain('greeter 0.1.0'); + expect(help.stdout).toContain('greet'); + await expect(execFile(bin, ['greet', 'World'], { cwd: projectRoot, env: environment })) + .resolves.toMatchObject({ stdout: '{"message":"Hello, World!","name":"World"}\n' }); + await expect(execFile(bin, ['greet', 'World', '--shout'], { cwd: projectRoot, env: environment })) + .resolves.toMatchObject({ stdout: '{"message":"HELLO, WORLD!","name":"World"}\n' }); + await expect(execFile(bin, ['greet'], { cwd: projectRoot, env: environment })) + .rejects.toMatchObject({ code: 2, stderr: expect.stringContaining('Missing required argument: .') }); // The src/index.ts convention produced the library export with declarations. const library = await import(pathToFileURL(join(projectRoot, 'dist', 'index.js')).href) as { @@ -104,10 +120,13 @@ it.concurrent('scaffolds the cli-tool template with a framework-built bin, lib, expect(library.greet('World').message).toBe('Hello, World!'); await expect(readFile(join(projectRoot, 'dist', 'index.d.ts'), 'utf8')).resolves.toContain('Greeting'); - // The same CLI also shipped inside the host artifact as a script. - await expect(execFile(process.execPath, [ - join(projectRoot, 'artifact', 'portable', 'scripts', 'greeter.mjs'), 'World', - ], { cwd: projectRoot, env: installedEnvironment() })).resolves.toMatchObject({ stdout: 'Hello, World!\n' }); + // The conventional plain script shipped inside the host artifact with the + // framework process envelope around its `main` export. + const script = join(projectRoot, 'artifact', 'portable', 'scripts', 'hello.mjs'); + await expect(execFile(process.execPath, [script, 'World'], { cwd: projectRoot, env: environment })) + .resolves.toMatchObject({ stdout: 'Hello, World!\n' }); + await expect(execFile(process.execPath, [script], { cwd: projectRoot, env: environment })) + .rejects.toMatchObject({ code: 2, stderr: 'Usage: hello \n' }); const packDestination = await mkdtemp(join(tmpdir(), 'create-agent-bundle-cli-pack-')); try { diff --git a/packages/create-agent-bundle/tests/scaffold.test.ts b/packages/create-agent-bundle/tests/scaffold.test.ts index fbb3a5a33..34838b442 100644 --- a/packages/create-agent-bundle/tests/scaffold.test.ts +++ b/packages/create-agent-bundle/tests/scaffold.test.ts @@ -89,9 +89,13 @@ describe('scaffold', () => { 'README.md', 'agent-bundle.config.ts', 'package.json', - 'src/cli.ts', + 'rstest.projection.config.ts', + 'src/cli/greet.ts', 'src/index.ts', - 'tests/cli.test.ts', + 'src/scripts/hello.ts', + 'tests/greet.test.ts', + 'tests/projection/cli-dispatch.test.ts', + 'tests/projection/script-dispatch.test.ts', 'tsconfig.json', ]); } finally { @@ -248,19 +252,32 @@ describe('scaffold', () => { } }); - for (const template of ['minimal', 'cli-tool'] as const) { - it(`scaffolds the ${template} template without a runtime tarball`, async () => { - const { root } = await scaffoldTemplate(template, { withRuntimeTarball: false }); - try { - const manifest = JSON.parse(await readFile(join(root, 'package.json'), 'utf8')) as { - readonly devDependencies: Record; - }; - expect(manifest.devDependencies['agent-bundle']).toMatch(/^file:/u); - } finally { - await rm(root, { force: true, recursive: true }); - } - }); - } + it('scaffolds the minimal template without a runtime tarball', async () => { + const { root } = await scaffoldTemplate('minimal', { withRuntimeTarball: false }); + try { + const manifest = JSON.parse(await readFile(join(root, 'package.json'), 'utf8')) as { + readonly devDependencies: Record; + }; + expect(manifest.devDependencies['agent-bundle']).toMatch(/^file:/u); + } finally { + await rm(root, { force: true, recursive: true }); + } + }); + + // Routed commands execute inside the typed Agent request context, so the + // cli-tool template now pairs the runtime package like mcp-server does. + it('pins the paired runtime for the routed cli-tool template', async () => { + const { frameworkSpec, root } = await scaffoldTemplate('cli-tool', { pluginName: 'greeter' }); + try { + const manifest = JSON.parse(await readFile(join(root, 'package.json'), 'utf8')) as { + readonly dependencies: Record; + }; + expect(manifest.dependencies['@agent-bundle/runtime']).toBe(runtimeSpecForFramework(frameworkSpec)); + expect(manifest.dependencies['zod']).toBeDefined(); + } finally { + await rm(root, { force: true, recursive: true }); + } + }); it('replaces every placeholder and pins the framework spec', async () => { const { files, frameworkSpec, root } = await scaffoldTemplate('cli-tool', { @@ -281,16 +298,22 @@ describe('scaffold', () => { }; expect(manifest.name).toBe('@scope/status-plugin'); expect(manifest.devDependencies['agent-bundle']).toBe(frameworkSpec); - if (files.includes('src/mcp/status/tools/report-status.tsx')) { - expect(manifest.dependencies?.['@agent-bundle/runtime']).toBe(runtimeSpecForFramework(frameworkSpec)); - } + expect(files).toContain('src/cli/greet.ts'); + expect(manifest.dependencies?.['@agent-bundle/runtime']).toBe(runtimeSpecForFramework(frameworkSpec)); expect(manifest.bin).toEqual({ 'status-plugin': './dist/bin/status-plugin.js', 'status-plugin-install': './dist/bin/status-plugin-install.js', }); const config = await readFile(join(root, 'agent-bundle.config.ts'), 'utf8'); expect(config).toContain("name: 'status-plugin'"); - expect(config).toContain("'status-plugin': './src/cli.ts'"); + // Routed CLI: no `scripts` or `bin` entry names the executable; the + // command graph compiles from src/cli/** by convention. + expect(config).not.toMatch(/\bscripts:/u); + expect(config).not.toMatch(/\bbin:/u); + // The generated help names the scaffolded plugin, and the template's own + // proof asserts that exact text, so the rename must reach the test. + expect(await readFile(join(root, 'tests/projection/cli-dispatch.test.ts'), 'utf8')) + .toContain("Run 'status-plugin greet --help' for usage."); } finally { await rm(root, { force: true, recursive: true }); } diff --git a/packages/workbench/tests/examples-real.e2e.test.ts b/packages/workbench/tests/examples-real.e2e.test.ts index 866bacfef..89b6ae355 100644 --- a/packages/workbench/tests/examples-real.e2e.test.ts +++ b/packages/workbench/tests/examples-real.e2e.test.ts @@ -5,6 +5,7 @@ import { expect } from '@rstest/playwright'; import { createWorkbenchAssetSource } from '../../agent-bundle/src/dev/workbench-assets.ts'; import { startDevServer } from '../../agent-bundle/src/dev/workbench-server.ts'; +import { inspectWorkbenchSurface, workbenchPageLabel } from '../../agent-bundle/src/test/index.ts'; import { captureExampleState, copyExample, @@ -668,6 +669,39 @@ e2e('renders the flagship compiled route catalog by server and kind in real Chro expect(projectedMcpRouteIds).toEqual(await tools.locator('.route-id').allTextContents()); expect(new Set(cliRouteIds).size).toBe(cliRouteIds.length); expect(cliRouteIds).toHaveLength(authoredCliRouteIds.length + projectedMcpRouteIds.length); + + // The consumer harness's workbench-surface level claims to hand a test + // exactly what this page shows, without a browser. Pin that claim to the + // real page: same CLI catalog order, same tool inventory, same headings, + // same navigation. + const surface = await inspectWorkbenchSurface({ root: project.root }); + const surfaceCli = surface.catalog.groups.find((group) => group.label === 'CLI commands'); + expect(surfaceCli?.entries.map((entry) => entry.route.id)).toEqual(cliRouteIds); + expect(surface.catalog.groups.find((group) => group.label === 'curator · Tools')?.entries.map((entry) => entry.route.id)) + .toEqual(await tools.locator('.route-id').allTextContents()); + for (const group of surface.catalog.groups) { + await expect(page.getByRole('heading', { name: group.label, exact: true })).toBeVisible({ timeout: browserTimeout }); + } + expect(surfaceCli?.entries.find((entry) => entry.route.id === 'cli:library-audit')?.commandUsage) + .toBe(await cli.locator('.route-command').filter({ hasText: 'library-audit' }).textContent()); + for (const pageName of surface.pages) { + await expect(page.getByRole('link', { name: workbenchPageLabel(pageName), exact: true })).toBeVisible({ timeout: browserTimeout }); + } + for (const pageName of surface.unavailablePages) { + await expect(page.getByRole('link', { name: workbenchPageLabel(pageName), exact: true })).toHaveCount(0); + } + // Same rail order, too: `surface.pages` claims the Workbench's own + // navigation order, so it must equal the rendered rail (Runtime is a + // dev-server runtime capability the surface does not model). + // Each rail link is an aria-hidden glyph span followed by the label text node. + const railLabels = (await page.getByLabel('Workbench navigation').getByRole('link').evaluateAll((links) => + links.map((link) => Array.from(link.childNodes) + .filter((node) => node.nodeType === Node.TEXT_NODE) + .map((node) => node.textContent ?? '') + .join('') + .trim()))) + .filter((label) => label !== 'Runtime'); + expect(railLabels).toEqual(surface.pages.map(workbenchPageLabel)); await expect(cli).toContainText('cli:library-audit', { timeout: browserTimeout }); await expect(cli).toContainText('src/cli/library-audit.tsx', { timeout: browserTimeout }); await expect(cli).toContainText('src/cli/shelf.tsx', { timeout: browserTimeout }); @@ -689,7 +723,8 @@ e2e('renders the flagship compiled route catalog by server and kind in real Chro // invented: the curator declares no conventional event routes or scripts // and discovers one conventional context provider. await expect(page.getByRole('region', { name: 'Route graph identity' }).locator('dd').first()) - .toHaveText('50', { timeout: browserTimeout }); + .toHaveText(String(surface.catalog.routeCount), { timeout: browserTimeout }); + expect(surface.catalog.routeCount).toBe(50); await expect(page.getByRole('heading', { name: 'Event routes', exact: true })).toHaveCount(0); await expect(page.getByRole('heading', { name: 'Scripts', exact: true })).toHaveCount(0); await expect(page.getByRole('heading', { name: 'Context providers', exact: true })).toBeVisible({ timeout: browserTimeout }); diff --git a/rstest.integration-tests.ts b/rstest.integration-tests.ts index 9ac020e65..858a59293 100644 --- a/rstest.integration-tests.ts +++ b/rstest.integration-tests.ts @@ -61,6 +61,7 @@ export const integrationTestFiles: readonly string[] = [ 'packages/agent-bundle/tests/script-playground-service.test.ts', 'packages/agent-bundle/tests/target-hook-contract.test.ts', 'packages/agent-bundle/tests/target-mcp-runtime.test.ts', + 'packages/agent-bundle/tests/workbench-surface-dev-server.test.ts', 'packages/agent-bundle/tests/worktree-proximity-journeys.test.ts', 'packages/rsc-runtime/tests/notices-sqlite-cross-process.test.ts', 'packages/rsc-runtime/tests/state-packaging.test.ts',