diff --git a/.changeset/test-harness-conventional-providers.md b/.changeset/test-harness-conventional-providers.md new file mode 100644 index 000000000..956027f31 --- /dev/null +++ b/.changeset/test-harness-conventional-providers.md @@ -0,0 +1,5 @@ +--- +"agent-bundle": patch +--- + +Mount conventional request context providers (`src/providers/*`) in the `agent-bundle/test` harness for every manifest-backed call — `renderRoute`, `renderRouteEvents`, `invokeCli` (plain, rendered, and projected MCP commands), `openInMemoryMcpServer`, and `invokeMcpTool` — exactly as the generated request scopes do: same deterministic key order, same surface-specific `invocation`, same fail-closed factory errors, and a `providers.processLifetime` scoped like the artifact's (fresh per `invokeCli` call and per `renderRoute` render, shared across one open in-memory MCP session). Pass `context.providers` to mount an explicit fixture map instead; `context` and its `providers` stay optional even once the generated `.agent-bundle/routes.d.ts` augmentation declares provider keys (`HarnessOptionsArguments`, `RenderRouteContextInit`), while an explicit map must carry every declared key and a direct `runAgentRequest` still requires `providers`. Provider modules are evaluated once per test worker, so module-level provider state is shared across simulated executables; prove cold state through the proof levels that spawn the artifact. Hand `renderRoute` providers and the request scope the executable surface the artifact records (a routed CLI command's space-joined path, a script's path-derived name) instead of the route id, and mount an event route's compiled id (`event:tool/after`) as `invocation.operationId` in the generated Flight worker, matching the hook shell, lifecycle replay, and harness. The test manifest gains `providers`, the generated Rstest setup registers provider loaders (test registry version 4), and a project whose setup predates that registration fails with the `manifest-unavailable` harness error naming the provider. (#399) diff --git a/.gitignore b/.gitignore index 7a5cb2ac5..7d6d8ee5a 100644 --- a/.gitignore +++ b/.gitignore @@ -5,8 +5,8 @@ dist/ coverage/ artifacts/ *.log -examples/audiobook-curator/artifact/ -examples/worktree-proximity/artifact/ +# Every example's `pnpm build` (and `pnpm examples:check`) writes here. +examples/*/artifact/ # Build-time copies of the root LICENSE and NOTICE (scripts/sync-license-files.mjs) packages/*/LICENSE diff --git a/docs/entry-conventions.md b/docs/entry-conventions.md index 006154c55..d74623e92 100644 --- a/docs/entry-conventions.md +++ b/docs/entry-conventions.md @@ -174,14 +174,25 @@ of throwing. `invocation.kind` stays surface-specific (`tool`, `event`, `cli`, `processLifetime` is reserved for the framework-owned process identity and hit counter, so provider filenames must not derive that key. -Route-unit and CLI-dispatch tests inject provider values through the same -`context` seam as identity axes (`renderRoute(id, { context: { providers: -{ library: fixture } } })`); the harness never executes conventional provider -modules, so a test chooses exactly the values a component observes. Once the -generated `.agent-bundle/routes.d.ts` augmentation declares provider keys, the -harness `options` and its `context.providers` become required (as does -`providers` on a direct `runAgentRequest`), so omitting a fixture the route's -types promise is a compile error rather than a runtime `undefined`. +The `agent-bundle/test` harness mounts the same providers, in the same order +and with the same fail-closed semantics, for every manifest-backed helper +(`renderRoute`, `renderRouteEvents`, `invokeCli`, and the in-memory MCP +helpers), so a route test observes what the artifact would mount. A test that +wants to choose the values instead injects them through the same `context` +seam as identity axes (`renderRoute(id, { context: { providers: { library: +fixture } } })`): an explicit map is mounted verbatim and no conventional +provider module executes. A module rendered directly (no compiled manifest) +has no project to discover, so it observes only `processLifetime`. Once the +generated `.agent-bundle/routes.d.ts` augmentation declares provider keys, an +explicit `context.providers` map must carry every declared key (as must +`providers` on a direct `runAgentRequest`), so a fixture that omits a value the +route's types promise is a compile error rather than a runtime `undefined`; +omitting `context.providers` altogether stays legal and mounts the real +providers. The harness reproduces the per-executable process identity, not +per-executable module evaluation: provider modules are evaluated once per test +worker, so module-level provider state is shared across the simulated +executables of that worker and is only proven cold by the proof levels that +spawn the artifact. ### Handler request context diff --git a/docs/framework-mode.md b/docs/framework-mode.md index caa0de9de..adcbf68d9 100644 --- a/docs/framework-mode.md +++ b/docs/framework-mode.md @@ -136,13 +136,19 @@ and augments `@agent-bundle/runtime`'s `AgentProviderValues`, so `(await agent()).providers.library` is a `LibraryContext` with no cast once the file is part of the project's TypeScript program (add `".agent-bundle/routes.d.ts"` to `tsconfig.json` `include`). Undeclared keys -stay `unknown`. Route-unit and CLI-dispatch tests inject fixture values through -`renderRoute(id, { context: { providers: { library } } })`; the harness never -executes provider modules on a test's behalf. Because the augmentation makes -declared keys required, the same program also requires `context.providers` -(and the harness `options` argument) on every `renderRoute`, `invokeCli`, and -in-memory MCP call, and `providers` on a direct `runAgentRequest`: a handler -typed against `providers.library` can never observe an unchecked `undefined`. +stay `unknown`. The `agent-bundle/test` harness (`renderRoute`, `invokeCli`, +the in-memory MCP helpers) mounts the project's providers automatically, in the +same order and with the same fail-closed semantics as the generated request +scopes, so a route-unit test observes what the artifact would mount — including +a provider that reaches the network or the file system. To stub one, inject +fixture values through `renderRoute(id, { context: { providers: { library } } })`: +an explicit map is mounted verbatim and no provider module executes. Because +the augmentation makes declared keys required, an explicit `context.providers` +must carry every declared key, and a direct `runAgentRequest` (where nothing +else supplies providers) requires `providers` outright: a handler typed against +`providers.library` can never observe an unchecked `undefined`. See the +[harness section](../packages/agent-bundle/README.md#testing-routes) for the +module-evaluation caveat that applies to provider-level state. ### What reaches the MCP wire diff --git a/docs/local-ci.md b/docs/local-ci.md index 26246af4f..dd73a4cbe 100644 --- a/docs/local-ci.md +++ b/docs/local-ci.md @@ -175,6 +175,18 @@ machine is exactly the contention that scale exists for. Exporting running other heavy work) overrides the default; the integration config never lets it drop below what its own pool shape requires. +Load-sensitive failures are fixed at their cause, never absorbed with a +per-test `retry`. The recurring shape is a test that acts before the product +has published the state it is about to assert on; the fix is to wait on the +product's own readiness signal. Precedents: the dev watcher's stat-signature +dedupe (#122/#329), content-identity reload announcements (#200/#332), and +the `examples-real.e2e` source edits, which used to pair a file write with an +immediate manual rebuild and so raced the watcher's own rebuild of the same +write for a second epoch. Those edits now go through +`replaceWatchedSourceAndAwaitRebuild` (`packages/agent-bundle/tests/support/watched-files.ts`): +one atomic replacement, then a wait on the coordinator's published build +attempt, so one edit is exactly one build. + ## What is deliberately not covered - **dependency-review** runs as a GitHub-side action against the GitHub diff --git a/examples/audiobook-curator/tests/route-unit/context.test.ts b/examples/audiobook-curator/tests/route-unit/context.test.ts index af9abfeaf..87ff71b0b 100644 --- a/examples/audiobook-curator/tests/route-unit/context.test.ts +++ b/examples/audiobook-curator/tests/route-unit/context.test.ts @@ -38,7 +38,10 @@ it('renders the catalog from injected library context with its contents envelope }); it('renders an honest degraded catalog when library context is absent', async () => { + // The harness mounts `src/providers/library.ts` automatically, so the + // degraded path needs an explicit empty provider map to keep it absent. const rendered = await renderRoute('resource:curator/catalog', { + context: { providers: {} }, input: { uri: 'audiobook-curator://catalog' }, }); const value = rendered.document.value as { diff --git a/packages/agent-bundle/README.md b/packages/agent-bundle/README.md index b46f56646..a55377789 100644 --- a/packages/agent-bundle/README.md +++ b/packages/agent-bundle/README.md @@ -411,6 +411,48 @@ than paying for a build per route. Every failure — an unknown route, a refused route kind, a rejected input, a render error — names the route id, the target kind, and the module provenance. +Conventional request context providers (`src/providers/*`, see +[entry conventions](../../docs/entry-conventions.md#request-context-providers-power-tier)) +are mounted automatically for every manifest-backed helper — `renderRoute`, +`renderRouteEvents`, `invokeCli`, and the in-memory MCP helpers — exactly as the +generated request scopes mount them: discovered from the compiled manifest, +executed once per request in the same deterministic key order, handed the same +surface-specific `invocation` (`tool`, `event`, `cli`, `script`), and failing the +request closed when a factory throws. `providers.processLifetime` is scoped the +way the artifact scopes it: each `invokeCli` call and each `renderRoute` render +is a fresh simulated executable (hit 1, new `instanceId`), while one open +`openInMemoryMcpServer` session shares a single identity across every request +it handles, like the artifact's warm Flight worker. Pass `context.providers` to opt out: an explicit map is mounted +verbatim and no conventional provider runs, which is how a test stubs a provider +that would otherwise reach the network or the file system. + +```ts +// Real providers, as the artifact would mount them. +const real = await renderRoute('tool:library/summarize', { input: { title: 'Dune' } }); + +// Stubbed providers: nothing under src/providers/ executes. +const stubbed = await invokeCli(['library', 'audit', './books'], { + context: { providers: { libraryTooling: { tool: 'ffprobe 6.1' } } }, +}); +``` + +`context` (and `context.providers`) stays optional even once the generated +`.agent-bundle/routes.d.ts` augmentation declares provider keys: omitting it +runs the real providers, which is what the artifact does, while an explicit map +must carry every declared key, so a fixture cannot leave a promised value +`undefined`. Only a direct `runAgentRequest` requires `providers` in that case, +because nothing else would supply them. + +The harness simulates the process identity per executable, not module +evaluation: one Rstest worker evaluates each provider module once, so +module-level state in a provider is shared across every simulated CLI +invocation, render, and in-memory server in that worker (as it is for the route +modules themselves), whereas a real artifact evaluates the module afresh in +every CLI process and Flight worker. A provider's module-level cache, counter, +or singleton is therefore only proven by the packed and projected proof levels +that spawn the artifact; a route-unit test that needs cold state should stub +the provider through `context.providers` or reset that state between calls. + Matchers over the Agent Document contracts: `toHaveStatus`, `toContainMarkdown`, `toContainText`, `toHaveValue`, `toHaveError`, and `toHaveNodeKinds`. diff --git a/packages/agent-bundle/fixtures/route-harness/src/cli/tooling/inspect.ts b/packages/agent-bundle/fixtures/route-harness/src/cli/tooling/inspect.ts new file mode 100644 index 000000000..7d509a2e1 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/cli/tooling/inspect.ts @@ -0,0 +1,24 @@ +import { agent } from '@agent-bundle/runtime'; +import type { CliRouteConfig, CliRouteProps } from 'agent-bundle'; +import { z } from 'zod'; + +export const config = { + description: 'Reports the request providers a plain command observes.', +} satisfies CliRouteConfig; + +export const inputSchema = z.object({}).strict(); + +export const resultSchema = z.object({ + keys: z.array(z.string()), + libraryTooling: z.unknown().optional(), + processLifetime: z.object({ hits: z.number().int().min(1), instanceId: z.string(), pid: z.number().int() }).strict(), +}).strict(); + +export default async function inspect(_props: CliRouteProps) { + const { providers } = await agent(); + return { + keys: Object.keys(providers).sort(), + libraryTooling: providers['libraryTooling'], + processLifetime: providers['processLifetime'], + }; +} diff --git a/packages/agent-bundle/fixtures/route-harness/src/cli/tooling/report.tsx b/packages/agent-bundle/fixtures/route-harness/src/cli/tooling/report.tsx new file mode 100644 index 000000000..13b43e542 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/cli/tooling/report.tsx @@ -0,0 +1,24 @@ +import { Agent, agent, type JsonValue } from '@agent-bundle/runtime'; +import type { CliRouteConfig, CliRouteProps } from 'agent-bundle'; +import { z } from 'zod'; + +export const config = { + description: 'Renders the request providers a rendered command observes.', +} satisfies CliRouteConfig; + +export const inputSchema = z.object({}).strict(); + +export const resultSchema = z.object({ + keys: z.array(z.string()), + libraryTooling: z.unknown().optional(), +}).strict(); + +export default async function ToolingReport(_props: CliRouteProps) { + const { providers } = await agent(); + const value = { keys: Object.keys(providers).sort(), libraryTooling: providers['libraryTooling'] as JsonValue }; + return ( + + {`tooling: ${JSON.stringify(providers['libraryTooling'])}`} + + ); +} diff --git a/packages/agent-bundle/fixtures/route-harness/src/mcp/harness/tools/tooling.tsx b/packages/agent-bundle/fixtures/route-harness/src/mcp/harness/tools/tooling.tsx new file mode 100644 index 000000000..95b1d5a2c --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/mcp/harness/tools/tooling.tsx @@ -0,0 +1,34 @@ +import { Agent, agent, type JsonValue } from '@agent-bundle/runtime'; +import { z } from 'zod'; + +export const config = { + annotations: { readOnlyHint: true }, + description: 'Reports the request providers an MCP tool observes.', + title: 'Tooling', +}; + +export const inputSchema = z.object({ + /** Makes the `library-tooling` provider throw, to prove the request fails closed. */ + failProvider: z.boolean().optional(), +}).strict(); + +export const resultSchema = z.object({ + keys: z.array(z.string()), + libraryTooling: z.unknown().optional(), + processLifetime: z.object({ hits: z.number(), instanceId: z.string(), pid: z.number() }).optional(), +}).strict(); + +export default async function Tooling() { + const { providers } = await agent(); + const { processLifetime } = providers; + const value = { + keys: Object.keys(providers).sort(), + libraryTooling: providers['libraryTooling'] as JsonValue, + ...(processLifetime === undefined ? {} : { processLifetime: { ...processLifetime } }), + }; + return ( + + {`tooling: ${JSON.stringify(providers['libraryTooling'])}`} + + ); +} diff --git a/packages/agent-bundle/fixtures/route-harness/src/providers/library-tooling.ts b/packages/agent-bundle/fixtures/route-harness/src/providers/library-tooling.ts new file mode 100644 index 000000000..2ea925c5a --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/providers/library-tooling.ts @@ -0,0 +1,25 @@ +import type { AgentProviderContext } from 'agent-bundle'; + +/** + * A conventional request context provider. The harness mounts it for every + * manifest request scope exactly as the generated entries do, so routes on + * every surface observe `providers.libraryTooling` with the surface-specific + * invocation kind the provider saw. + */ +export default async function libraryTooling({ invocation, signal }: AgentProviderContext) { + if (signal.aborted) throw new DOMException('aborted', 'AbortError'); + const input = invocation.kind === 'tool' ? invocation.props.input : undefined; + if (typeof input === 'object' && input !== null && (input as { readonly failProvider?: unknown }).failProvider === true) { + throw new Error('ffprobe is not installed'); + } + const surface = invocation.kind === 'tool' + ? invocation.props.operationId + : invocation.kind === 'cli' + ? invocation.props.command + : invocation.kind === 'script' + ? invocation.props.name + : invocation.kind === 'event' + ? invocation.props.event + : invocation.props.view; + return { kind: invocation.kind, surface, tool: 'ffprobe 6.1' }; +} diff --git a/packages/agent-bundle/fixtures/route-harness/src/scripts/tooling-summary.tsx b/packages/agent-bundle/fixtures/route-harness/src/scripts/tooling-summary.tsx new file mode 100644 index 000000000..d134b2e42 --- /dev/null +++ b/packages/agent-bundle/fixtures/route-harness/src/scripts/tooling-summary.tsx @@ -0,0 +1,26 @@ +import { Agent, agent, type JsonValue } from '@agent-bundle/runtime'; +import { z } from 'zod'; + +export const resultSchema = z.object({ + arguments: z.number().int().nonnegative(), + keys: z.array(z.string()), + libraryTooling: z.unknown().optional(), +}).strict(); + +export default async function ToolingSummary({ argv, signal }: { + readonly argv: readonly string[]; + readonly signal: AbortSignal; +}) { + if (signal.aborted) throw new DOMException('aborted', 'AbortError'); + const { providers } = await agent(); + const value = { + arguments: argv.length, + keys: Object.keys(providers).sort(), + libraryTooling: providers['libraryTooling'] as JsonValue, + }; + return ( + + {`Summarized ${String(argv.length)} arguments.`} + + ); +} diff --git a/packages/agent-bundle/src/build/entry-shell.ts b/packages/agent-bundle/src/build/entry-shell.ts index 410afb912..428c8d5b2 100644 --- a/packages/agent-bundle/src/build/entry-shell.ts +++ b/packages/agent-bundle/src/build/entry-shell.ts @@ -4,6 +4,7 @@ import { fileURLToPath } from 'node:url'; import { eventIpcRuntimeSpecifier, eventProjectRuntimeSpecifier } from '../adapters/hook-contract.ts'; import { stableJson } from '../core/digest.ts'; import type { NormalizedHook, NormalizedStateDefinition } from '../core/types.ts'; +import { orderedProviders } from '../routes/provider-execution.ts'; import { providerKeyFromName } from '../routes/providers.ts'; import type { CompiledAgentRoute, CompiledCliCommand, CompiledProvider } from '../routes/types.ts'; @@ -292,7 +293,7 @@ export const generatedCliBinEntrySource = (options: GeneratedCliBinEntryOptions) " if (route === undefined || typeof route.module.default !== 'function') throw new TypeError('Generated CLI route must default-export an async function.');", ' const parsed = parseInput(route, input);', ' const cwd = process.cwd();', - ' processLifetime.hits += 1;', + ...processHitSource(' '), ...(options.state === undefined ? [] : [' const bindings = await runtimeState.requestBindings({ signal: context.signal });', ' try {']), @@ -411,7 +412,7 @@ export const generatedRenderedRouteWorkerSource = ( " if (route === undefined || typeof route.module.default !== 'function') throw new TypeError('Generated rendered route must default-export an async function component.');", ' const controller = new AbortController();', ' requests.set(message.id, controller);', - ' processLifetime.hits += 1;', + ...processHitSource(' '), ' try {', ' const cwd = process.cwd();', ...(options.state === undefined @@ -554,17 +555,19 @@ const eventRouteImports = ( ): readonly string[] => routes.map((route, index) => `import * as route${String(offset + index)} from ${JSON.stringify(route.source)};`); +/** + * Event route records stay keyed by the hook identity the worker resolves + * from the canonical event (`hook:event-route:tool-after`), but the record's + * `id` is the compiled route id (`event:tool/after`): that is the + * `operationId` the hook shell opens the request scope with, the lifecycle + * replay mounts, the test manifest addresses, and the harness renders, so a + * route reading `invocation.operationId` sees one value everywhere. + */ const eventRouteRecords = ( routes: readonly NormalizedHook[], offset: number, ): readonly string[] => routes.map((route, index) => - ` ${JSON.stringify(route.id)}: Object.freeze({ event: ${JSON.stringify(route.eventRoute!.event)}, id: ${JSON.stringify(route.id)}, kind: 'event-route', module: route${String(offset + index)}, name: ${JSON.stringify(route.eventRoute!.event)} }),`); - -const orderedProviders = (providers: readonly CompiledProvider[]): readonly CompiledProvider[] => - [...providers].sort((left, right) => { - const byKey = providerKeyFromName(left.name).localeCompare(providerKeyFromName(right.name)); - return byKey === 0 ? left.source.localeCompare(right.source) : byKey; - }); + ` ${JSON.stringify(route.id)}: Object.freeze({ event: ${JSON.stringify(route.eventRoute!.event)}, id: ${JSON.stringify(`event:${route.eventRoute!.event}`)}, kind: 'event-route', module: route${String(offset + index)}, name: ${JSON.stringify(route.eventRoute!.event)} }),`); const providerImports = (providers: readonly CompiledProvider[]): readonly string[] => providers.map((provider, index) => @@ -580,15 +583,27 @@ const providerRegistrySource = (providers: readonly CompiledProvider[]): readonl ? [] : ['const providers = Object.freeze([', ...providerRecords(providers), ']);']; -const processLifetimeValueSource = - '{ hits: processLifetime.hits, instanceId: processLifetime.instanceId, pid: processLifetime.pid }'; +/** + * Claims this request's hit on the process identity and snapshots it in the + * same synchronous step, before any state binding or provider `await`, so a + * concurrent request on the same scope cannot move the value this request + * mounts as `providers.processLifetime`. + */ +const processHitSource = (indent: string): readonly string[] => [ + `${indent}processLifetime.hits += 1;`, + `${indent}const processHit = { hits: processLifetime.hits, instanceId: processLifetime.instanceId, pid: processLifetime.pid };`, +]; + +const processLifetimeValueSource = 'processHit'; /** * Per-request provider execution shared by every generated request scope * (shared Flight worker, rendered CLI/script worker, plain routed CLI): once * per request, sequentially in deterministic key order, fail-closed on a * missing factory or a thrown/rejected factory, with the framework-owned - * `processLifetime` value seeded first. + * `processLifetime` value seeded first. The emitted loop mirrors + * `executeProviders` in `../routes/provider-execution.ts`, which the + * in-process test harness runs; `entry-shell.test.ts` pins the two together. */ const providerExecutionSource = ( providers: readonly CompiledProvider[], @@ -656,7 +671,7 @@ export const generatedRouteFlightWorkerSource = (options: GeneratedRouteFlightWo " if (route === undefined || typeof route.module.default !== 'function') throw new TypeError('Generated route must default-export an async Server Component.');", ' const controller = new AbortController();', ' requests.set(message.id, controller);', - ' processLifetime.hits += 1;', + ...processHitSource(' '), ' try {', ...(options.state === undefined ? [] diff --git a/packages/agent-bundle/src/routes/provider-execution.ts b/packages/agent-bundle/src/routes/provider-execution.ts new file mode 100644 index 000000000..fdaa34d74 --- /dev/null +++ b/packages/agent-bundle/src/routes/provider-execution.ts @@ -0,0 +1,101 @@ +import { providerKeyFromName } from './providers.ts'; +import type { CompiledProvider } from './types.ts'; + +/** + * The per-request provider execution contract every generated request scope + * implements (#313, #366): once per request, sequentially in deterministic + * key order, fail-closed on a missing factory or a thrown/rejected factory, + * with the framework-owned `processLifetime` value seeded first. + * + * `entry-shell.ts` emits this loop as generated source so artifacts stay + * self-contained; `agent-bundle/test` runs it in-process through + * {@link executeProviders}. Ordering and the fail-closed messages live here so + * the two cannot drift: the harness must mount exactly what the artifact does. + */ + +/** Deterministic execution order: by mounted key, then by source path for a key collision. */ +export const orderedProviders = >( + providers: readonly T[], +): readonly T[] => [...providers].sort((left, right) => { + const byKey = providerKeyFromName(left.name).localeCompare(providerKeyFromName(right.name)); + return byKey === 0 ? left.source.localeCompare(right.source) : byKey; +}); + +export const providerFactoryMissingMessage = (key: string, source: string): string => + `Context provider "${key}" (${source}) must default-export a factory.`; + +export const providerFailedMessage = (key: string, source: string, cause: unknown): string => + `Context provider "${key}" (${source}) failed: ${cause instanceof Error ? cause.message : String(cause)}`; + +/** The framework-owned process identity a request scope mounts at `providers.processLifetime`. */ +export interface ProviderProcessLifetime { + hits: number; + readonly instanceId: string; + readonly pid: number; +} + +export const createProviderProcessLifetime = (): ProviderProcessLifetime => ({ + hits: 0, + instanceId: crypto.randomUUID(), + pid: process.pid, +}); + +/** The immutable snapshot of one process lifetime a request observes. */ +export interface ProviderProcessLifetimeValue { + readonly hits: number; + readonly instanceId: string; + readonly pid: number; +} + +export const providerProcessLifetimeValue = ( + lifetime: ProviderProcessLifetime, +): ProviderProcessLifetimeValue => ({ + hits: lifetime.hits, + instanceId: lifetime.instanceId, + pid: lifetime.pid, +}); + +/** One loaded provider module in execution order, with the identity its failures name. */ +export interface ExecutableProvider { + readonly key: string; + readonly module: { readonly default?: unknown }; + /** Project-relative path, as the generated scopes report it. */ + readonly source: string; +} + +export interface ExecuteProvidersOptions { + /** The surface-specific provider invocation (`tool`, `event`, `cli`, `script`). */ + readonly invocation: unknown; + readonly processLifetime: ProviderProcessLifetime; + /** Providers already in {@link orderedProviders} order. */ + readonly providers: readonly ExecutableProvider[]; + readonly signal: AbortSignal; +} + +/** + * Executes conventional providers for one request exactly as a generated + * request scope does. The caller increments `processLifetime.hits` before the + * call, as every generated scope does before its provider loop. + */ +export const executeProviders = async ( + options: ExecuteProvidersOptions, +): Promise>> => { + const values: Record = { + processLifetime: providerProcessLifetimeValue(options.processLifetime), + }; + for (const provider of options.providers) { + const factory = provider.module.default; + if (typeof factory !== 'function') { + throw new TypeError(providerFactoryMissingMessage(provider.key, provider.source)); + } + try { + values[provider.key] = await (factory as (context: { + readonly invocation: unknown; + readonly signal: AbortSignal; + }) => unknown)({ invocation: options.invocation, signal: options.signal }); + } catch (error) { + throw new Error(providerFailedMessage(provider.key, provider.source, error), { cause: error }); + } + } + return values; +}; diff --git a/packages/agent-bundle/src/rstest/index.ts b/packages/agent-bundle/src/rstest/index.ts index 46611b3d6..516ff23f2 100644 --- a/packages/agent-bundle/src/rstest/index.ts +++ b/packages/agent-bundle/src/rstest/index.ts @@ -131,6 +131,7 @@ export const agentBundleRstest = async ( export type { AgentBundleTestManifest, + TestableProviderDescriptor, TestableRouteDescriptor, TestableStateDescriptor, } from '../test/manifest.ts'; diff --git a/packages/agent-bundle/src/rstest/setup-module.ts b/packages/agent-bundle/src/rstest/setup-module.ts index fc68a5dc7..54d6a6da0 100644 --- a/packages/agent-bundle/src/rstest/setup-module.ts +++ b/packages/agent-bundle/src/rstest/setup-module.ts @@ -37,6 +37,8 @@ const specifier = (source: string): string => source.replaceAll('\\', '/'); export const routeTestSetupSource = (manifest: AgentBundleTestManifest): string => { const loaders = renderableRoutes(manifest) .map((route) => ` ${JSON.stringify(route.id)}: () => import(${JSON.stringify(specifier(route.source))}),`); + const providerLoaders = (manifest.providers ?? []) + .map((provider) => ` ${JSON.stringify(provider.id)}: () => import(${JSON.stringify(specifier(provider.source))}),`); return [ '// @generated by agent-bundle/rstest. Do not edit: rerun Rstest to regenerate.', '//', @@ -48,6 +50,9 @@ export const routeTestSetupSource = (manifest: AgentBundleTestManifest): string ...loaders, ' },', ` manifest: JSON.parse(${JSON.stringify(JSON.stringify(manifest))}),`, + ...(providerLoaders.length === 0 + ? [] + : [' providerLoaders: {', ...providerLoaders, ' },']), ...(manifest.state === undefined ? [] : [` stateLoader: () => import(${JSON.stringify(specifier(manifest.state.source))}),`]), diff --git a/packages/agent-bundle/src/test/cli.ts b/packages/agent-bundle/src/test/cli.ts index 7cc616086..b7bbe39f7 100644 --- a/packages/agent-bundle/src/test/cli.ts +++ b/packages/agent-bundle/src/test/cli.ts @@ -21,9 +21,11 @@ import type * as AgentRuntime from '@agent-bundle/runtime'; import { CliInputError, runGeneratedCliEntry } from '../cli-entry.ts'; import type { CliRenderedEvent } from '../cli-entry.ts'; +import { createProviderProcessLifetime } from '../routes/provider-execution.ts'; import type { CompiledCliCommand } from '../routes/types.ts'; import { AgentTestError, captured } from './errors.ts'; import { CLI_DISPATCH_PROOF_LEVEL, type AgentBundleTestManifest } from './manifest.ts'; +import { claimProcessHit, mountProviders } from './providers.ts'; import { registeredRouteLoader, testManifest } from './registry.ts'; import { prepareCliRenderHost, type HarnessOptionsArguments, type RenderRouteContextInit } from './render.ts'; import type { AgentRouteModule, RenderedRouteProvenance } from './types.ts'; @@ -42,8 +44,9 @@ export interface InvokeCliOptionsBase { /** * Dispatch options; `context` carries the request-scope overrides for the - * dispatched command over the runtime's request contract and is required once - * the project declares providers (see {@link RenderRouteContextInit}). + * dispatched command over the runtime's request contract; omitting it (or its + * `providers`) mounts the project's conventional providers exactly as the + * generated executable does (see {@link RenderRouteContextInit}). */ export type InvokeCliOptions = InvokeCliOptionsBase & RenderRouteContextInit; @@ -163,6 +166,9 @@ export const invokeCli = async ( const runtime = await loadRuntime(); const context = options.context ?? {}; const signal = options.signal ?? new AbortController().signal; + // One simulated executable per invocation: the generated CLI creates its + // process identity at module load, so every separate run starts at hit 1. + const processLifetime = createProviderProcessLifetime(); const renderedCommands = manifest.cliCommands.filter((command) => command.rendered); let executed: CompiledCliCommand | undefined; @@ -183,6 +189,7 @@ export const invokeCli = async ( manifest, modules: renderedModules, onValidated: (validated) => { value = validated; }, + processLifetime, provenance: { kind: 'cli', manifestDigest: manifest.digest, @@ -228,6 +235,15 @@ export const invokeCli = async ( throw new CliInputError(error instanceof Error ? error.message : String(error)); } const root = process.cwd(); + // Same provider invocation the generated plain-command path builds (#366). + const providers = await mountProviders({ + explicit: context.providers, + invocation: { kind: 'cli', props: { args: execution.args, command: commandPath(command) } }, + manifest, + processHit: claimProcessHit(processLifetime), + provenance: { ...provenance, kind: 'cli', routeId: command.routeId, source: 'manifest', targets: [] }, + signal: execution.signal, + }); const result = await runtime.runAgentRequest({ capabilities: { command: runtime.unavailable(), @@ -238,6 +254,7 @@ export const invokeCli = async ( host: runtime.unavailable('unsupported-surface'), workspace: runtime.available({ root }, 'derived'), ...context, + providers, invocation: { kind: 'cli', operationId: command.routeId, diff --git a/packages/agent-bundle/src/test/index.ts b/packages/agent-bundle/src/test/index.ts index d6d94fb30..3d5145f3f 100644 --- a/packages/agent-bundle/src/test/index.ts +++ b/packages/agent-bundle/src/test/index.ts @@ -47,11 +47,12 @@ export type { CompileTestManifestOptions, TestManifestPluginIdentity, TestableAppDescriptor, + TestableProviderDescriptor, TestableRouteDescriptor, TestableStateDescriptor, } from './manifest.ts'; export { AGENT_TEST_REGISTRY_VERSION, registerTestRoutes, testManifest } from './registry.ts'; -export type { AgentTestRouteRegistry } from './registry.ts'; +export type { AgentProviderModuleLoader, AgentTestRouteRegistry } from './registry.ts'; export { AgentTestError } from './errors.ts'; export type { AgentTestErrorCode } from './errors.ts'; export { renderRoute, renderRouteEvents } from './render.ts'; diff --git a/packages/agent-bundle/src/test/manifest.ts b/packages/agent-bundle/src/test/manifest.ts index 0906748cf..b8ca6cda5 100644 --- a/packages/agent-bundle/src/test/manifest.ts +++ b/packages/agent-bundle/src/test/manifest.ts @@ -4,9 +4,12 @@ import type { Diagnostic } from '../core/diagnostics.ts'; import { stableJson } from '../core/digest.ts'; import { deepFreeze } from '../core/freeze.ts'; import type { NormalizedMcpApp, NormalizedStateDefinition } from '../core/types.ts'; +import { orderedProviders } from '../routes/provider-execution.ts'; +import { providerKeyFromName } from '../routes/providers.ts'; import type { CompiledAgentRoute, CompiledCliCommand, + CompiledProvider, CompiledRouteGraph, CompiledRouteKind, } from '../routes/types.ts'; @@ -154,6 +157,21 @@ export interface TestableStateDescriptor { readonly source: string; } +/** + * One conventional `src/providers/.ts` context provider the harness + * mounts for every manifest request scope, exactly as the generated entries + * do (#313). `key` is the camel-cased request-context key. + */ +export interface TestableProviderDescriptor { + readonly id: string; + readonly key: string; + readonly name: string; + /** Project-relative POSIX path of the provider module. */ + readonly relativePath: string; + /** Absolute provider module path. */ + readonly source: string; +} + /** One normalized MCP App declaration addressable by the browser proof level. */ export interface TestableAppDescriptor { readonly _meta?: Readonly>; @@ -201,6 +219,12 @@ export interface AgentBundleTestManifest { readonly projectRoot: string; /** The level the manifest and its registered loaders alone supply; every other level stamps its own. */ readonly proofLevel: AgentTestProofLevel; + /** + * Conventional request context providers, in the execution order every + * generated request scope uses; the harness mounts them automatically unless + * a test passes `context.providers`. Absent when the project declares none. + */ + readonly providers?: readonly TestableProviderDescriptor[]; readonly routes: Readonly>; /** Conventional project state mounted automatically for manifest route renders. */ readonly state?: TestableStateDescriptor; @@ -222,6 +246,16 @@ const descriptorOf = (route: CompiledAgentRoute): TestableRouteDescriptor => ({ source: route.source, }); +const providerDescriptors = ( + providers: readonly CompiledProvider[], +): readonly TestableProviderDescriptor[] => orderedProviders(providers).map((provider) => ({ + id: provider.id, + key: providerKeyFromName(provider.name), + name: provider.name, + relativePath: provider.provenance.relativePath, + source: provider.source, +})); + const graphRoutes = (graph: CompiledRouteGraph): readonly CompiledAgentRoute[] => [ ...(graph.cli?.routes ?? []), ...graph.events, @@ -304,6 +338,7 @@ export const testManifestFromRouteGraph = (input: { plugin: input.plugin ?? FALLBACK_PLUGIN_IDENTITY, projectRoot: input.projectRoot, proofLevel: ROUTE_UNIT_PROOF_LEVEL, + ...(input.graph.providers.length === 0 ? {} : { providers: providerDescriptors(input.graph.providers) }), routes, ...(input.state === undefined ? {} diff --git a/packages/agent-bundle/src/test/mcp.ts b/packages/agent-bundle/src/test/mcp.ts index af03c154b..ad31b76c5 100644 --- a/packages/agent-bundle/src/test/mcp.ts +++ b/packages/agent-bundle/src/test/mcp.ts @@ -22,8 +22,10 @@ import type { } from '@agent-bundle/runtime/state'; import type { createGeneratedRuntimeState } from '@agent-bundle/runtime/mount'; +import { createProviderProcessLifetime } from '../routes/provider-execution.ts'; import { AgentTestError, captured } from './errors.ts'; import { MCP_IN_MEMORY_PROOF_LEVEL, type AgentBundleTestManifest } from './manifest.ts'; +import { claimProcessHit, mountProviders } from './providers.ts'; import { registeredRouteLoader, testManifest } from './registry.ts'; import type { HarnessOptionsArguments, RenderRouteContextInit } from './render.ts'; import type { RenderedRouteProvenance, TestableRouteDescriptor } from './types.ts'; @@ -73,8 +75,9 @@ export interface InMemoryMcpSessionOptionsBase< /** * Session options; `context` holds the request-scoped overrides applied to - * every route render in this session and is required once the project - * declares providers (see {@link RenderRouteContextInit}). + * every route render in this session; omitting it (or its `providers`) mounts + * the project's conventional providers exactly as the generated server does + * (see {@link RenderRouteContextInit}). */ export type InMemoryMcpSessionOptions< TState = unknown, @@ -311,6 +314,9 @@ export const openInMemoryMcpServer = async < // the same warm host wrapper the artifact uses — it simply renders here // instead of in a spawned thread. const artifactEpoch = `${manifest.plugin.name}@${manifest.plugin.version}`; + // One process identity per open server, like the artifact's Flight worker: + // every request this session handles shares it, and a new server starts fresh. + const processLifetime = createProviderProcessLifetime(); const runtimeState = options.state === undefined ? undefined : dependencies.createGeneratedRuntimeState(options.state); @@ -328,8 +334,23 @@ export const openInMemoryMcpServer = async < details: [`registered: ${Object.keys(routes).sort().join(', ')}`], }); } + // The hit is claimed before state bindings are awaited, in the + // generated worker's order, so a failed or slow binding still consumes + // this request's hit and concurrent requests keep arrival order. + const processHit = claimProcessHit(processLifetime); const bindings = await runtimeState?.requestBindings({ signal: request.signal }); try { + // Conventional providers run before the scope opens, over the same + // tool invocation the generated Flight worker hands them. + const descriptor = manifest.routes[route.id]; + const providers = await mountProviders({ + explicit: context.providers, + invocation: request.invocation, + manifest, + processHit, + ...(descriptor === undefined ? {} : { provenance: routeProvenance(descriptor, manifest) }), + signal: request.signal, + }); return streamOf(await dependencies.runAgentRequest({ // Mirror the Flight worker boundary while allowing the documented // harness context seam to override forwarded transport identity. @@ -338,6 +359,7 @@ export const openInMemoryMcpServer = async < session: transport.session, workspace: transport.workspace, ...context, + providers, invocation: { kind: 'tool' as const, operationId: route.id, diff --git a/packages/agent-bundle/src/test/providers.ts b/packages/agent-bundle/src/test/providers.ts new file mode 100644 index 000000000..028c4e94e --- /dev/null +++ b/packages/agent-bundle/src/test/providers.ts @@ -0,0 +1,112 @@ +import type { AgentProviderValues } from '@agent-bundle/runtime'; + +import { + executeProviders, + providerProcessLifetimeValue, + type ExecutableProvider, + type ProviderProcessLifetime, + type ProviderProcessLifetimeValue, +} from '../routes/provider-execution.ts'; +import { AgentTestError } from './errors.ts'; +import type { AgentBundleTestManifest, TestableProviderDescriptor } from './manifest.ts'; +import { registeredProviderLoader } from './registry.ts'; +import type { RenderedRouteProvenance } from './types.ts'; + +/** + * Conventional request context providers for harness request scopes. + * + * Every generated request scope discovers `src/providers/*` and executes them + * once per request before `runAgentRequest` (#313, #366). The harness does the + * same for every manifest-backed render, dispatch, and in-memory projection, + * through the shared execution helper the generated scopes mirror, so a test + * observes the provider map the artifact would mount. A test that passes + * `context.providers` opts out: the explicit map is used verbatim, exactly as + * the runtime's request contract reads it. + * + * What the harness simulates per executable is the framework-owned process + * identity (`processLifetime`), not module evaluation. Provider modules load + * through the generated setup's static loaders, so one Rstest worker evaluates + * each module once and every simulated CLI invocation, route render, and + * in-memory server in that worker shares its module-level state — the same + * way the worker shares the route modules themselves. A real artifact + * evaluates the module afresh in every CLI process and Flight worker, so a + * provider's module-level cache, counter, or singleton is only proven by the + * packed and projected proof levels that spawn the artifact; a route-unit test + * that needs cold module state should substitute a fixture through + * `context.providers` or reset that state between calls. + */ + +export interface MountProvidersOptions { + /** Explicit provider values from the test; when present they win and nothing is discovered. */ + readonly explicit: AgentProviderValues | undefined; + /** The surface-specific provider invocation the generated scope would pass (`tool`, `event`, `cli`, `script`). */ + readonly invocation: unknown; + /** Absent for a module rendered directly: no project, so nothing to discover. */ + readonly manifest: AgentBundleTestManifest | undefined; + /** + * This request's claimed hit on the simulated executable's process identity + * (see {@link claimProcessHit}); mounted verbatim as `providers.processLifetime`. + */ + readonly processHit: ProviderProcessLifetimeValue; + readonly provenance?: RenderedRouteProvenance; + readonly signal: AbortSignal; +} + +const loadProvider = async ( + manifest: AgentBundleTestManifest, + descriptor: TestableProviderDescriptor, + provenance: RenderedRouteProvenance | undefined, +): Promise => { + const loader = registeredProviderLoader(manifest, descriptor.id); + if (loader === undefined) { + throw new AgentTestError( + 'manifest-unavailable', + `Context provider ${descriptor.id} (${descriptor.relativePath}) is compiled but no test-time module loader is registered for it.`, + { + ...(provenance === undefined ? {} : { provenance }), + recovery: 'Build the Rstest configuration with agentBundleRstest() so the generated setup registers provider loaders, or pass context.providers explicitly to skip conventional provider discovery.', + }, + ); + } + return { key: descriptor.key, module: await loader(), source: descriptor.relativePath }; +}; + +/** + * Claims one request's hit on a simulated executable's process identity and + * snapshots it in the same synchronous step, exactly where the generated + * scopes do: before any state binding or provider module `await`, so a + * concurrent request on the same identity cannot move this request's value. + * + * Callers scope the identity as the artifact scopes its module-level + * `processLifetime`: one per CLI invocation (each generated executable starts + * at hit 1), one per rendered route request, and one per open in-memory MCP + * server session (shared by every request that session handles). It is never + * shared across unrelated helper calls, so a provider branching on `hits` or + * `instanceId` cannot observe warmth the artifact would not exhibit. + */ +export const claimProcessHit = (processLifetime: ProviderProcessLifetime): ProviderProcessLifetimeValue => { + processLifetime.hits += 1; + return providerProcessLifetimeValue(processLifetime); +}; + +/** + * The `providers` value for one harness request scope: the explicit map when + * the test supplied one, otherwise the project's conventional providers + * executed in the generated order over the claimed process hit. + */ +export const mountProviders = async (options: MountProvidersOptions): Promise => { + if (options.explicit !== undefined) return options.explicit; + if (options.manifest === undefined) { + return { processLifetime: options.processHit }; + } + const providers: ExecutableProvider[] = []; + for (const descriptor of options.manifest.providers ?? []) { + providers.push(await loadProvider(options.manifest, descriptor, options.provenance)); + } + return executeProviders({ + invocation: options.invocation, + processLifetime: { ...options.processHit }, + providers, + signal: options.signal, + }); +}; diff --git a/packages/agent-bundle/src/test/registry.ts b/packages/agent-bundle/src/test/registry.ts index 00886d34a..7234b924b 100644 --- a/packages/agent-bundle/src/test/registry.ts +++ b/packages/agent-bundle/src/test/registry.ts @@ -16,16 +16,25 @@ export const AGENT_TEST_REGISTRY_SYMBOL_KEY = 'agent-bundle/test-route-registry' const REGISTRY_SYMBOL = Symbol.for(AGENT_TEST_REGISTRY_SYMBOL_KEY); -export const AGENT_TEST_REGISTRY_VERSION = 3; +/** + * Bumped whenever the registry layout changes so a setup module and the + * helpers reading it never silently disagree about what the registry carries. + * 4: `providerLoaders` (conventional context providers mounted by the harness). + */ +export const AGENT_TEST_REGISTRY_VERSION = 4; export type AgentStateModuleLoader = () => Promise<{ readonly default: AgentStateDefinition; }>; +export type AgentProviderModuleLoader = () => Promise<{ readonly default?: unknown }>; + export interface AgentTestRouteRegistry { /** Lazy loaders keyed by compiled route id, so a test only compiles the routes it renders. */ readonly loaders: Readonly>; readonly manifest: AgentBundleTestManifest; + /** Lazy loaders keyed by compiled provider id; present only when the project declares providers. */ + readonly providerLoaders?: Readonly>; readonly stateLoader?: AgentStateModuleLoader; readonly version: number; } @@ -114,6 +123,16 @@ export const registeredStateLoader = ( return registry.stateLoader; }; +/** The provider-module loader generated beside the registered manifest for one compiled provider id. */ +export const registeredProviderLoader = ( + manifest: AgentBundleTestManifest, + providerId: string, +): AgentProviderModuleLoader | undefined => { + const registry = registered(); + if (registry === undefined || !producedRegisteredLoaders(registry, manifest)) return undefined; + return registry.providerLoaders?.[providerId]; +}; + /** The registered manifest's identity, so a loader miss can name the mismatch that caused it. */ export const registeredManifestIdentity = (): { readonly digest: string; readonly projectRoot: string } | undefined => { const registry = registered(); diff --git a/packages/agent-bundle/src/test/render.ts b/packages/agent-bundle/src/test/render.ts index af8dd334e..0d838b771 100644 --- a/packages/agent-bundle/src/test/render.ts +++ b/packages/agent-bundle/src/test/render.ts @@ -26,9 +26,11 @@ import type { GeneratedCliRenderContext, GeneratedCliRenderSession, } from '../cli-entry.ts'; +import { createProviderProcessLifetime, type ProviderProcessLifetime } from '../routes/provider-execution.ts'; import type { CompiledCliCommand } from '../routes/types.ts'; import { AgentTestError, captured } from './errors.ts'; import { ROUTE_UNIT_PROOF_LEVEL, type AgentBundleTestManifest } from './manifest.ts'; +import { claimProcessHit, mountProviders } from './providers.ts'; import { registeredManifestIdentity, registeredRouteLoader, @@ -47,24 +49,30 @@ import type { * request contract. `host`, `session`, `actor`, `workspace`, and * `capabilities` are the identity-injection seam for context-dependent route * tests; construct observed values with `available` or `unavailable` from - * `@agent-bundle/runtime`. + * `@agent-bundle/runtime`. `providers` is the opt-out for conventional + * provider discovery: when present it is mounted verbatim; when absent the + * harness executes the project's `src/providers/*` exactly as the generated + * request scopes do. It stays optional even once the generated + * `.agent-bundle/routes.d.ts` augmentation declares provider keys — omitting + * it runs the real providers, which is the artifact's behavior — while an + * explicit map must still carry every declared key, so a fixture cannot leave + * a promised value `undefined`. */ -export type RenderRouteContext = Omit & { +export type RenderRouteContext = Omit & { readonly invocation?: Omit; readonly progress?: AgentProgressReporter; + readonly providers?: AgentProviderValues; }; /** - * The `context` member of every harness call. The harness installs fixture - * values instead of executing `src/providers/*`, so once the project's - * generated `.agent-bundle/routes.d.ts` augmentation declares required provider - * keys, `context` (and its `providers`) becomes mandatory: a test cannot omit - * the fixtures while the route's types promise them. Provider-free projects - * keep `context` optional. + * The `context` member of every harness call. Unlike a direct + * `runAgentRequest`, where `providers` becomes mandatory once the augmentation + * declares keys because nothing else would supply them, a harness call + * mounts the project's conventional providers itself, so `context` is always + * optional: omitting it observes what the artifact mounts, and passing + * `context.providers` substitutes a complete fixture map. */ -export type RenderRouteContextInit = Record extends AgentProviderValues - ? { readonly context?: RenderRouteContext } - : { readonly context: RenderRouteContext }; +export type RenderRouteContextInit = { readonly context?: RenderRouteContext }; export interface RenderRouteOptionsBase { /** CLI route arguments; `cli` routes only. */ @@ -84,13 +92,11 @@ export interface RenderRouteOptionsBase { export type RenderRouteOptions = RenderRouteOptionsBase & RenderRouteContextInit; /** - * The trailing options parameter of every harness entry point. Provider-free - * projects may omit it; once the generated augmentation declares provider - * keys it is mandatory, so no harness call can silently skip the fixtures. + * The trailing options parameter of every harness entry point. It is always + * optional: a call that omits it mounts the project's conventional providers + * exactly as the generated request scopes do (see {@link RenderRouteContextInit}). */ -export type HarnessOptionsArguments = Record extends AgentProviderValues - ? readonly [options?: Options] - : readonly [options: Options]; +export type HarnessOptionsArguments = readonly [options?: Options]; export interface RenderedRoute { /** The final Agent Document the real renderer produced. */ @@ -211,9 +217,50 @@ const cliArguments = ( ); }; +/** + * The executable surface name the generated entry records and hands to + * providers, derived like the artifact derives it: a routed CLI command is + * its space-joined command path (`tooling report`), a script is its + * path-derived name (`script:tooling-summary` -> `tooling-summary`), and an + * event route is its canonical event. The compiled command graph is the + * authority for command paths; without a manifest (module-direct renders) + * the harness falls back to the route id's own path segments. + */ +const executableSurface = ( + kind: RenderableRouteKind, + routeId: string, + manifest: AgentBundleTestManifest | undefined, +): string => { + switch (kind) { + case 'prompt': + case 'resource': + case 'tool': + return protocolName(routeId); + case 'event-route': + return routeId.startsWith('event:') ? routeId.slice('event:'.length) : routeId; + case 'cli': { + // Only authored `src/cli/**` commands have a `cli` route kind. Projected + // MCP commands (`command.mcp`) carry their tool's route id, so a request + // for one resolves as that `tool` route above, exactly like the generated + // entry's `command.mcp !== undefined` branch. + const command = manifest?.cliCommands.find((candidate) => + candidate.mcp === undefined && candidate.routeId === routeId); + if (command !== undefined) return command.path.join(' '); + return (routeId.startsWith('cli:') ? routeId.slice('cli:'.length) : routeId).replaceAll('/', ' '); + } + case 'script': + return routeId.startsWith('script:') ? routeId.slice('script:'.length) : routeId; + default: { + const exhaustive: never = kind; + throw new AgentTestError('unsupported-route-kind', `Unsupported renderable route kind ${String(exhaustive)}.`); + } + } +}; + const invocationFor = ( kind: RenderableRouteKind, routeId: string, + surface: string, options: RenderRouteOptions, provenance: RenderedRouteProvenance, ): AgentRenderInvocation => { @@ -229,20 +276,14 @@ const invocationFor = ( // The generated server names the canonical event, not the route id, and // carries the host envelope as `payload`; the harness matches both so a // route sees the props the artifact would hand it. - return { - kind: 'event', - props: { - event: routeId.startsWith('event:') ? routeId.slice('event:'.length) : routeId, - payload: (options.input ?? {}) as never, - }, - }; + return { kind: 'event', props: { event: surface, payload: (options.input ?? {}) as never } }; case 'cli': - return { kind: 'cli', props: { args: cliArguments(options, provenance), command: routeId } }; + // The generated executable passes `command.path.join(' ')`, never the + // route id, so providers branching on `command` see the artifact's value. + return { kind: 'cli', props: { args: cliArguments(options, provenance), command: surface } }; case 'script': - return { - kind: 'script', - props: { input: cliArguments(options, provenance) as never, name: routeId }, - }; + // The generated script passes its path-derived name (`tooling-summary`). + return { kind: 'script', props: { input: cliArguments(options, provenance) as never, name: surface } }; default: { const exhaustive: never = kind; throw new AgentTestError( @@ -309,14 +350,20 @@ const componentProps = ( } }; -/** The request-scope invocation the generated server opens for one route. */ +/** + * The request-scope invocation the generated entry opens for one route: + * `operationId` is the route id and `surface` the executable surface name on + * every kind, exactly as the generated MCP server, CLI, and script shells + * record them. + */ const requestInvocation = ( invocation: AgentRenderInvocation, routeId: string, + surface: string, ): AgentInvocationInput => ({ kind: invocation.kind, - ...(invocation.kind === 'tool' ? { operationId: routeId } : {}), - surface: invocation.kind === 'tool' ? protocolName(routeId) : routeId, + operationId: routeId, + surface, }); const componentOf = ( @@ -564,13 +611,14 @@ interface FlightDispatcherOptions { readonly contextProgress?: AgentProgressReporter; readonly limits?: Partial; readonly renderer: Renderer; - readonly requestInit: (request: AgentRenderDispatch) => AgentRequestInit; + /** Async so conventional providers execute inside the request, before the scope opens, as generated scopes do. */ + readonly requestInit: (request: AgentRenderDispatch) => Promise; } const createFlightDispatcher = (options: FlightDispatcherOptions): AgentRuntime.AgentRenderDispatcher => options.renderer.createAgentRenderDispatcher({ execute: async (request) => streamOf(await options.renderer.runAgentRequest({ - ...options.requestInit(request), + ...(await options.requestInit(request)), progress: progressFor(options.collected, options.contextProgress, request.progress), signal: request.signal, }, async () => drain(options.renderer.renderAgentFlight( @@ -587,6 +635,8 @@ export interface PrepareCliRenderHostOptions { readonly manifest: AgentBundleTestManifest; readonly modules: ReadonlyMap; readonly onValidated: (value: unknown) => void; + /** The invoking CLI's process identity; the rendered command runs inside that same simulated executable. */ + readonly processLifetime: ProviderProcessLifetime; readonly provenance: RenderedRouteProvenance; readonly signal: AbortSignal; } @@ -667,8 +717,16 @@ export const prepareCliRenderHost = async ( componentProps: (request) => ({ input: parsed, signal: request.signal }), contextProgress: context.progress, renderer, - requestInit: (request) => { + requestInit: async (request) => { const root = process.cwd(); + const providers = await mountProviders({ + explicit: context.providers, + invocation, + manifest: options.manifest, + processHit: claimProcessHit(options.processLifetime), + provenance: { ...options.provenance, routeId: command.routeId }, + signal: request.signal, + }); return { capabilities: { command: renderer.unavailable(), @@ -680,6 +738,7 @@ export const prepareCliRenderHost = async ( workspace: renderer.available({ root }, 'derived'), ...context, ...mounted.context, + providers, invocation: command.mcp === undefined ? { kind: 'cli', @@ -735,10 +794,14 @@ const prepareRender = async ( ): Promise => { const resolved = await resolveTarget(target, options); const renderer = await loadRenderer(); - const invocation = invocationFor(resolved.kind, resolved.provenance.routeId, options, resolved.provenance); + const surface = executableSurface(resolved.kind, resolved.provenance.routeId, resolved.manifest); + const invocation = invocationFor(resolved.kind, resolved.provenance.routeId, surface, options, resolved.provenance); const collected: AgentProgressUpdate[] = []; const context = options.context ?? {}; const signal = options.signal ?? new AbortController().signal; + // A route-unit render stands in for one fresh executable serving one + // request; nothing is warm across renders, so each starts at hit 1. + const processLifetime = createProviderProcessLifetime(); const mounted = await mountManifestState(resolved.manifest, resolved.provenance, context, renderer, signal); const dispatcher = createFlightDispatcher({ collected, @@ -747,11 +810,21 @@ const prepareRender = async ( contextProgress: context.progress, limits: options.limits, renderer, - requestInit: (request) => ({ + requestInit: async (request) => ({ ...context, ...mounted.context, + // The render invocation is exactly what the generated Flight worker + // receives as `message.invocation`, so providers see the same shape. + providers: await mountProviders({ + explicit: context.providers, + invocation: request.invocation, + manifest: resolved.manifest, + processHit: claimProcessHit(processLifetime), + provenance: resolved.provenance, + signal: request.signal, + }), invocation: { - ...requestInvocation(request.invocation, resolved.provenance.routeId), + ...requestInvocation(request.invocation, resolved.provenance.routeId, surface), ...context.invocation, kind: request.invocation.kind, }, diff --git a/packages/agent-bundle/tests/entry-shell.test.ts b/packages/agent-bundle/tests/entry-shell.test.ts index c90e2f3c4..9e792d65e 100644 --- a/packages/agent-bundle/tests/entry-shell.test.ts +++ b/packages/agent-bundle/tests/entry-shell.test.ts @@ -17,6 +17,12 @@ import { mcpServerRuntimePath, mcpServerRuntimeSpecifier, } from '../src/build/entry-shell.ts'; +import { + executeProviders, + orderedProviders, + providerFactoryMissingMessage, + providerFailedMessage, +} from '../src/routes/provider-execution.ts'; const execFile = promisify(executeFile); @@ -351,8 +357,14 @@ it('generates the warm react-server Flight worker separately from the MCP dispat expect(source).toContain('/project/src/mcp/curator/tools/inspect.tsx'); expect(source).toContain('/project/src/events/tool/after.tsx'); expect(source).toContain("message.invocation.kind === 'event'"); + // The worker resolves the event route by its hook identity but mounts the + // compiled route id as `operationId`, the same id the hook shell, the + // lifecycle replay, and the test harness use for that route. + expect(source).toContain( + '"hook:event-route:tool-after": Object.freeze({ event: "tool/after", id: "event:tool/after", kind: \'event-route\'', + ); expect(createHash('sha256').update(source).digest('hex')).toBe( - '2b9feba295b3a77cd14bdee6527379837a9a21712e649c545d35d1fed107245d', + '36f042498df1933c6321bd21e4585599a0d39e5ddb3890657bd660c322f4cc23', ); expect(generate({ artifactEpoch: 'route-fixture@1.2.3', @@ -559,6 +571,12 @@ it('mounts deterministic per-request providers for plain routed CLI commands (#3 expect(withProviders.indexOf('for (const provider of providers)')).toBeLessThan( withProviders.indexOf('const result = await runAgentRequest({'), ); + // The request's hit is claimed and snapshotted in one synchronous step + // before any await, so concurrent requests cannot move each other's value. + expect(withProviders).toContain( + 'processLifetime.hits += 1;\n const processHit = { hits: processLifetime.hits, instanceId: processLifetime.instanceId, pid: processLifetime.pid };', + ); + expect(withProviders).toContain('const providerValues = { processLifetime: processHit };'); // A project without providers still mounts only the framework-owned process identity. const withoutProviders = entryShellModule.generatedCliBinEntrySource({ @@ -567,9 +585,7 @@ it('mounts deterministic per-request providers for plain routed CLI commands (#3 routes: [route], }); expect(withoutProviders).not.toContain('const providers = Object.freeze(['); - expect(withoutProviders).toContain( - 'providers: { processLifetime: { hits: processLifetime.hits, instanceId: processLifetime.instanceId, pid: processLifetime.pid } },', - ); + expect(withoutProviders).toContain('providers: { processLifetime: processHit },'); expect(withoutProviders).not.toContain('import * as provider0'); }); @@ -617,6 +633,84 @@ it('mounts deterministic per-request providers in rendered route workers', () => expect(bridge).toContain("worker.postMessage({ id, invocation, props, request, routeId, type: 'render' })"); }); +it('keeps the generated provider loop and the in-process execution helper identical', async () => { + const providers = [ + { + id: 'provider:zeta', + name: 'zeta', + provenance: { kind: 'conventional' as const, relativePath: 'src/providers/zeta.ts' }, + source: '/project/src/providers/zeta.ts', + }, + { + id: 'provider:alpha-value', + name: 'alpha-value', + provenance: { kind: 'conventional' as const, relativePath: 'src/providers/alpha-value.ts' }, + source: '/project/src/providers/alpha-value.ts', + }, + ]; + const source = entryShellModule.generatedRenderedRouteWorkerSource({ + providers, + routes: [{ + config: {}, + id: 'cli:report', + kind: 'cli', + provenance: { kind: 'conventional', relativePath: 'src/cli/report.tsx' }, + source: '/project/src/cli/report.tsx', + }], + }); + + // Ordering: the harness manifest and the generated registry sort identically. + expect(orderedProviders(providers).map((provider) => provider.name)).toEqual(['alpha-value', 'zeta']); + expect(source.indexOf('key: "alphaValue"')).toBeLessThan(source.indexOf('key: "zeta"')); + + // Messages: the generated template literals evaluate to the helper's text. + const evaluate = (template: string, bindings: Record): string => + template.replaceAll(/\$\{([^}]+)\}/gu, (_match, expression: string) => bindings[expression] ?? `<${expression}>`); + const missing = /throw new TypeError\(`([^`]+)`\)/u.exec(source)?.[1]; + const failed = /throw new Error\(`([^`]+)`, \{ cause: error \}\)/u.exec(source)?.[1]; + expect(missing).toBeDefined(); + expect(failed).toBeDefined(); + expect(evaluate(missing!, { 'provider.key': 'alphaValue', 'provider.source': 'src/providers/alpha-value.ts' })) + .toBe(providerFactoryMissingMessage('alphaValue', 'src/providers/alpha-value.ts')); + expect(evaluate(failed!, { + 'error instanceof Error ? error.message : String(error)': 'boom', + 'provider.key': 'alphaValue', + 'provider.source': 'src/providers/alpha-value.ts', + })).toBe(providerFailedMessage('alphaValue', 'src/providers/alpha-value.ts', new Error('boom'))); + + // Behavior: processLifetime seeded first, deterministic order, fail-closed on both defects. + const lifetime = { hits: 3, instanceId: 'instance-1', pid: 42 }; + const calls: string[] = []; + const values = await executeProviders({ + invocation: { kind: 'cli', props: { args: [], command: 'report' } }, + processLifetime: lifetime, + providers: [ + { key: 'alphaValue', module: { default: (context: { invocation: unknown }) => { calls.push('alphaValue'); return context.invocation; } }, source: 'src/providers/alpha-value.ts' }, + { key: 'zeta', module: { default: async () => { calls.push('zeta'); return 'z'; } }, source: 'src/providers/zeta.ts' }, + ], + signal: new AbortController().signal, + }); + expect(Object.keys(values)).toEqual(['processLifetime', 'alphaValue', 'zeta']); + expect(values).toEqual({ + alphaValue: { kind: 'cli', props: { args: [], command: 'report' } }, + processLifetime: { hits: 3, instanceId: 'instance-1', pid: 42 }, + zeta: 'z', + }); + expect(calls).toEqual(['alphaValue', 'zeta']); + await expect(executeProviders({ + invocation: undefined, + processLifetime: lifetime, + providers: [{ key: 'zeta', module: {}, source: 'src/providers/zeta.ts' }], + signal: new AbortController().signal, + })).rejects.toThrow('Context provider "zeta" (src/providers/zeta.ts) must default-export a factory.'); + await expect(executeProviders({ + invocation: undefined, + processLifetime: lifetime, + providers: [{ key: 'zeta', module: { default: () => { throw new Error('boom'); } }, source: 'src/providers/zeta.ts' }], + signal: new AbortController().signal, + })).rejects.toThrow('Context provider "zeta" (src/providers/zeta.ts) failed: boom'); +}); + it('conditionally emits generated state mounting without leaking sqlite into volatile or stateless entries', () => { const route = { config: {}, diff --git a/packages/agent-bundle/tests/generated-route-server.test.ts b/packages/agent-bundle/tests/generated-route-server.test.ts index 734366eea..61c77cf78 100644 --- a/packages/agent-bundle/tests/generated-route-server.test.ts +++ b/packages/agent-bundle/tests/generated-route-server.test.ts @@ -918,7 +918,7 @@ it('renders composite plugin events through each concrete host in one warm runti ' const context = await agent();', ' const processLifetime = context.providers.processLifetime as { hits: number; instanceId: string };', ' const host = context.host.state === "available" ? context.host.value.name : "unavailable";', - ' return createElement(Agent.Result, null, createElement(Agent.Context, null, `${host}:tool/after:${String(processLifetime.hits)}:${processLifetime.instanceId}`));', + ' return createElement(Agent.Result, null, createElement(Agent.Context, null, `${host}:${context.invocation.operationId}|${context.invocation.surface}:${String(processLifetime.hits)}:${processLifetime.instanceId}`));', '}', '', ].join('\n')), @@ -973,11 +973,14 @@ it('renders composite plugin events through each concrete host in one warm runti }, { AGENT_BUNDLE_HOOK_HOST: undefined, PLUGIN_ROOT: undefined }); const firstContext = (claude as { hookSpecificOutput: { additionalContext: string } }) .hookSpecificOutput.additionalContext; - const instanceId = firstContext.slice('claude:tool/after:1:'.length); + // The worker mounts the compiled route id as `operationId` and the + // canonical event as `surface` — the same pair the hook shell, the + // lifecycle replay, and `renderRoute` record for this route. + const instanceId = firstContext.slice('claude:event:tool/after|tool/after:1:'.length); expect(instanceId).not.toBe(''); expect(claude).toEqual({ hookSpecificOutput: { - additionalContext: `claude:tool/after:1:${instanceId}`, + additionalContext: `claude:event:tool/after|tool/after:1:${instanceId}`, hookEventName: 'PostToolUse', }, }); @@ -993,7 +996,7 @@ it('renders composite plugin events through each concrete host in one warm runti transcript_path: null, }, { AGENT_BUNDLE_HOOK_HOST: undefined, PLUGIN_ROOT: output })).resolves.toEqual({ hookSpecificOutput: { - additionalContext: `codex:tool/after:2:${instanceId}`, + additionalContext: `codex:event:tool/after|tool/after:2:${instanceId}`, hookEventName: 'PostToolUse', }, }); @@ -1008,7 +1011,7 @@ it('renders composite plugin events through each concrete host in one warm runti tool_output: '{"ok":true}', tool_use_id: 'tool-cursor', }, { AGENT_BUNDLE_HOOK_HOST: undefined, PLUGIN_ROOT: undefined })).resolves.toEqual({ - additional_context: `cursor:tool/after:3:${instanceId}`, + additional_context: `cursor:event:tool/after|tool/after:3:${instanceId}`, }); await expect(runHook(sharedSession.output, { diff --git a/packages/agent-bundle/tests/packed-stdio-projection.test.ts b/packages/agent-bundle/tests/packed-stdio-projection.test.ts index 4cadf0ace..74e91351a 100644 --- a/packages/agent-bundle/tests/packed-stdio-projection.test.ts +++ b/packages/agent-bundle/tests/packed-stdio-projection.test.ts @@ -126,6 +126,7 @@ it('serves compiled routes and durable state across packed process restarts', as 'publish-notice', 'strict-report', 'ticket', + 'tooling', 'unavailable', 'wait', ]); diff --git a/packages/agent-bundle/tests/projection/cli-dispatch.test.ts b/packages/agent-bundle/tests/projection/cli-dispatch.test.ts index 2fd16d67d..d41045e26 100644 --- a/packages/agent-bundle/tests/projection/cli-dispatch.test.ts +++ b/packages/agent-bundle/tests/projection/cli-dispatch.test.ts @@ -36,10 +36,13 @@ describe('the CLI dispatch level', () => { 'harness publish-notice', 'harness strict-report', 'harness ticket', + 'harness tooling', 'harness unavailable', 'harness wait', 'inventory', 'report', + 'tooling inspect', + 'tooling report', ], proofLevel: 'cli-dispatch', }); diff --git a/packages/agent-bundle/tests/projection/mcp-in-memory.test.ts b/packages/agent-bundle/tests/projection/mcp-in-memory.test.ts index a3b760a85..737d1b24d 100644 --- a/packages/agent-bundle/tests/projection/mcp-in-memory.test.ts +++ b/packages/agent-bundle/tests/projection/mcp-in-memory.test.ts @@ -31,7 +31,7 @@ describe('the in-memory MCP projection level', () => { it('registers every compiled route kind on the real generated server', async () => { const surface = await listMcpSurface(); - expect(surface.tools).toEqual(['catalog', 'context', 'echo', 'journal', 'lifecycle', 'mutation-probe', 'publish-notice', 'strict-report', 'ticket', 'unavailable', 'wait']); + expect(surface.tools).toEqual(['catalog', 'context', 'echo', 'journal', 'lifecycle', 'mutation-probe', 'publish-notice', 'strict-report', 'ticket', 'tooling', 'unavailable', 'wait']); expect(surface.prompts).toEqual(['summarize']); expect(surface.resources).toEqual(['harness://notes']); expect(surface.provenance).toMatchObject({ @@ -48,6 +48,7 @@ describe('the in-memory MCP projection level', () => { 'tool:harness/publish-notice', 'tool:harness/strict-report', 'tool:harness/ticket', + 'tool:harness/tooling', 'tool:harness/unavailable', 'tool:harness/wait', ], diff --git a/packages/agent-bundle/tests/projection/providers.test.ts b/packages/agent-bundle/tests/projection/providers.test.ts new file mode 100644 index 000000000..17dbb7b03 --- /dev/null +++ b/packages/agent-bundle/tests/projection/providers.test.ts @@ -0,0 +1,247 @@ +import { setTimeout as sleep } from 'node:timers/promises'; + +import { describe, expect, it } from '@rstest/core'; +import { createMemoryStateDriver, defineState, type AgentStateDriver } from '@agent-bundle/runtime/state'; +import { z } from 'zod'; + +import { cliJson, invokeCli } from '../../src/test/cli.ts'; +import { AgentTestError } from '../../src/test/errors.ts'; +import { invokeMcpTool, openInMemoryMcpServer } from '../../src/test/mcp.ts'; +import { renderRoute } from '../../src/test/render.ts'; +import { testManifest } from '../../src/test/registry.ts'; + +/** + * Conventional request context providers reach every harness request scope + * the way they reach every generated request scope (#313, #366): discovered + * from the compiled manifest, executed once per request in the generated + * order, and mounted at `providers.` beside the framework-owned + * `processLifetime`. A test that passes `context.providers` opts out and the + * explicit map is used verbatim. + */ +describe('conventional providers through the harness', () => { + it('names the compiled providers in the manifest in the generated execution order', () => { + const manifest = testManifest(); + + expect(manifest.providers).toEqual([{ + id: 'provider:library-tooling', + key: 'libraryTooling', + name: 'library-tooling', + relativePath: 'src/providers/library-tooling.ts', + source: expect.stringMatching(/route-harness[\\/]src[\\/]providers[\\/]library-tooling\.ts$/u), + }]); + }); + + it('mounts providers for a plain routed CLI command with the cli invocation', async () => { + const run = await invokeCli(['tooling', 'inspect']); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + expect(cliJson(run)).toEqual({ + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { kind: 'cli', surface: 'tooling inspect', tool: 'ffprobe 6.1' }, + processLifetime: { hits: 1, instanceId: expect.any(String), pid: process.pid }, + }); + }); + + it('mounts providers for a rendered routed CLI command with the cli invocation', async () => { + const run = await invokeCli(['tooling', 'report', '--json']); + + expect(run.exitCode).toBe(0); + expect(run.stderr).toBe(''); + expect(cliJson(run)).toEqual({ + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { kind: 'cli', surface: 'tooling report', tool: 'ffprobe 6.1' }, + }); + }); + + it('mounts providers for a projected MCP command with the tool invocation', async () => { + const run = await invokeCli(['harness', 'tooling', '--json']); + + expect(run.exitCode).toBe(0); + expect(cliJson(run)).toEqual({ + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { kind: 'tool', surface: 'tool:harness/tooling', tool: 'ffprobe 6.1' }, + processLifetime: { hits: 1, instanceId: expect.any(String), pid: process.pid }, + }); + }); + + it('mounts providers for an MCP route through the real in-memory server', async () => { + const call = await invokeMcpTool('tooling'); + + expect(call.isError).toBe(false); + expect(call.structuredContent).toEqual({ + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { kind: 'tool', surface: 'tool:harness/tooling', tool: 'ffprobe 6.1' }, + processLifetime: { hits: 1, instanceId: expect.any(String), pid: process.pid }, + }); + }); + + it('mounts providers for an MCP route at the route-unit level, including when it is also a projected CLI command', async () => { + // `harness tooling` is projected onto the CLI from this tool; its command + // carries the tool's route id, so rendering it takes the tool branch the + // generated entry takes for `command.mcp !== undefined`. + expect(testManifest().cliCommands.find((command) => command.mcp?.tool === 'tooling')?.routeId).toBe('tool:harness/tooling'); + const rendered = await renderRoute('tool:harness/tooling'); + + expect(rendered.result).toEqual({ + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { kind: 'tool', surface: 'tool:harness/tooling', tool: 'ffprobe 6.1' }, + processLifetime: { hits: 1, instanceId: expect.any(String), pid: process.pid }, + }); + }); + + it('mounts providers for a rendered script with the script name the generated script passes', async () => { + const rendered = await renderRoute('script:tooling-summary', { args: ['--fast', 'a.mp4'] }); + + // The generated script passes `name: 'tooling-summary'`, never the route id. + expect(rendered.result).toEqual({ + arguments: 2, + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { kind: 'script', surface: 'tooling-summary', tool: 'ffprobe 6.1' }, + }); + }); + + it('mounts providers for a rendered CLI route at the route-unit level with the command path', async () => { + const rendered = await renderRoute('cli:tooling/report'); + + // The generated executable passes `command.path.join(' ')`, never the route id. + expect(rendered.result).toEqual({ + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: { kind: 'cli', surface: 'tooling report', tool: 'ffprobe 6.1' }, + }); + }); + + it('gives every CLI invocation its own fresh process identity, like a separate generated executable', async () => { + type Lifetime = { processLifetime: { hits: number; instanceId: string; pid: number } }; + const first = cliJson(await invokeCli(['tooling', 'inspect'])) as Lifetime; + const second = cliJson(await invokeCli(['tooling', 'inspect'])) as Lifetime; + + expect(first.processLifetime).toEqual({ hits: 1, instanceId: expect.any(String), pid: process.pid }); + expect(second.processLifetime).toEqual({ hits: 1, instanceId: expect.any(String), pid: process.pid }); + expect(second.processLifetime.instanceId).not.toBe(first.processLifetime.instanceId); + }); + + it('shares one process identity across the requests of one open in-memory MCP server only', async () => { + type Lifetime = { processLifetime: { hits: number; instanceId: string; pid: number } }; + const lifetimeOf = (result: unknown): Lifetime['processLifetime'] => + ((result as { structuredContent: Lifetime }).structuredContent).processLifetime; + + await using session = await openInMemoryMcpServer(); + const first = lifetimeOf(await session.client.callTool({ arguments: {}, name: 'tooling' })); + const second = lifetimeOf(await session.client.callTool({ arguments: {}, name: 'tooling' })); + await using other = await openInMemoryMcpServer(); + const elsewhere = lifetimeOf(await other.client.callTool({ arguments: {}, name: 'tooling' })); + const convenience = lifetimeOf(await invokeMcpTool('tooling')); + + // The same warm server serves both calls, exactly like the artifact's + // Flight worker; a second server, and the open-call-close convenience + // helper, are separate processes that start at hit 1. + expect(first).toEqual({ hits: 1, instanceId: expect.any(String), pid: process.pid }); + expect(second).toEqual({ hits: 2, instanceId: first.instanceId, pid: process.pid }); + expect(elsewhere).toEqual({ hits: 1, instanceId: expect.any(String), pid: process.pid }); + expect(elsewhere.instanceId).not.toBe(first.instanceId); + expect(convenience).toEqual({ hits: 1, instanceId: expect.any(String), pid: process.pid }); + expect(convenience.instanceId).not.toBe(first.instanceId); + }); + + it('hands concurrent requests on one server distinct hit counts, snapshotted before provider loading', async () => { + type Lifetime = { processLifetime: { hits: number; instanceId: string } }; + await using session = await openInMemoryMcpServer(); + + const results = await Promise.all( + Array.from({ length: 4 }, () => session.client.callTool({ arguments: {}, name: 'tooling' })), + ); + const lifetimes = results.map((result) => (result as { structuredContent: Lifetime }).structuredContent.processLifetime); + + // Like the generated worker, each request captures its own hit right after + // the increment; awaiting provider loaders must not let a concurrent + // request move it. + expect(lifetimes.map((lifetime) => lifetime.hits).sort((left, right) => left - right)).toEqual([1, 2, 3, 4]); + expect(new Set(lifetimes.map((lifetime) => lifetime.instanceId)).size).toBe(1); + }); + + it('claims the hit before awaiting state bindings, so hits follow arrival order like the generated worker', async () => { + type Lifetime = { processLifetime: { hits: number; instanceId: string } }; + const definition = defineState({ + events: { changed: z.object({ value: z.string() }).strict() }, + id: 'providers/request-state', + initial: { value: '' }, + lifetime: 'request', + reduce: (_state, event) => ({ value: event.payload.value }), + schema: z.object({ value: z.string() }).strict(), + }); + const inner = createMemoryStateDriver({ lifetime: 'request' }); + let projectOpens = 0; + const driver: AgentStateDriver = { + ...inner, + open: async (opened) => { + // Only the first request's project store is slow to open; the second + // request's bindings resolve first. + if (opened.id === definition.id && projectOpens++ === 0) await sleep(150); + return inner.open(opened); + }, + }; + await using session = await openInMemoryMcpServer({ state: { definition, driver } }); + + const [first, second] = await Promise.all([ + session.client.callTool({ arguments: {}, name: 'tooling' }), + session.client.callTool({ arguments: {}, name: 'tooling' }), + ]); + const lifetimeOf = (result: unknown): Lifetime['processLifetime'] => + (result as { structuredContent: Lifetime }).structuredContent.processLifetime; + + // The generated worker increments and snapshots before `requestBindings`; + // the request that arrived first keeps hit 1 even though its state + // bindings resolved last. + expect(lifetimeOf(first).hits).toBe(1); + expect(lifetimeOf(second).hits).toBe(2); + expect(projectOpens).toBe(2); + }); + + it('gives every route-unit render a fresh process identity', async () => { + type Result = { processLifetime: { hits: number; instanceId: string } }; + const first = (await renderRoute('tool:harness/tooling')).result as Result; + const second = (await renderRoute('tool:harness/tooling')).result as Result; + + expect(first.processLifetime.hits).toBe(1); + expect(second.processLifetime.hits).toBe(1); + expect(second.processLifetime.instanceId).not.toBe(first.processLifetime.instanceId); + }); + + it('uses an explicit context.providers map verbatim instead of discovering providers', async () => { + const [plain, rendered, tool] = await Promise.all([ + invokeCli(['tooling', 'inspect'], { + context: { providers: { libraryTooling: 'stubbed', processLifetime: { hits: 1, instanceId: 'test', pid: 1 } } }, + }), + renderRoute('script:tooling-summary', { context: { providers: { other: true } } }), + invokeMcpTool('tooling', { context: { providers: {} } }), + ]); + + expect(cliJson(plain)).toEqual({ + keys: ['libraryTooling', 'processLifetime'], + libraryTooling: 'stubbed', + processLifetime: { hits: 1, instanceId: 'test', pid: 1 }, + }); + expect(rendered.result).toEqual({ arguments: 0, keys: ['other'] }); + expect(tool.structuredContent).toEqual({ keys: [] }); + }); + + it('fails a request closed when a provider factory throws, naming the provider like the generated scope', async () => { + const message = 'Context provider "libraryTooling" (src/providers/library-tooling.ts) failed: ffprobe is not installed'; + + const run = await invokeCli(['harness', 'tooling', '--input', '{"failProvider":true}']); + expect(run.exitCode).toBe(1); + expect(run.stdout).toBe(''); + expect(run.stderr).toContain(message); + + let error: unknown; + try { + await renderRoute('tool:harness/tooling', { input: { failProvider: true } }); + } catch (caught) { + error = caught; + } + expect(error).toBeInstanceOf(AgentTestError); + expect((error as AgentTestError).code).toBe('render-failed'); + expect((error as AgentTestError).message).toContain(message); + }); +}); diff --git a/packages/agent-bundle/tests/provider-typegen.test.ts b/packages/agent-bundle/tests/provider-typegen.test.ts index 34dc11abb..cbb4264c3 100644 --- a/packages/agent-bundle/tests/provider-typegen.test.ts +++ b/packages/agent-bundle/tests/provider-typegen.test.ts @@ -113,9 +113,11 @@ it('types (await agent()).providers. from the generated provider declaratio 'export const wrong = async (): Promise => (await agent()).providers.library;', '', ].join('\n')), - // Contexts that do not run src/providers/* — a custom runAgentRequest host - // or a route-unit fixture — must supply the declared keys, or the handler's - // typed `providers.library` would dereference undefined at runtime. + // A custom runAgentRequest host runs no src/providers/*, so it must supply + // the declared keys, or the handler's typed `providers.library` would + // dereference undefined at runtime. The harness mounts the project's + // providers itself, so a call without `context` is legal and observes the + // real values; an explicit `context.providers` fixture must be complete. writeProjectFile(root, 'custom-scope.ts', [ "import { runAgentRequest } from '@agent-bundle/runtime';", "import { renderRoute } from 'agent-bundle/test';", @@ -125,6 +127,8 @@ it('types (await agent()).providers. from the generated provider declaratio 'export const complete = async (): Promise => {', " await runAgentRequest({ invocation: { kind: 'tool' }, providers: { buildNumber: 7, library } }, async () => undefined);", " await renderRoute('tool:curator/status', { context: { providers: { buildNumber: 7, library } } });", + " await renderRoute('tool:curator/status');", + " await renderRoute('tool:curator/status', { input: {} });", '};', '', ].join('\n')), @@ -138,7 +142,6 @@ it('types (await agent()).providers. from the generated provider declaratio "import type { LibraryContext } from './src/providers/library.js';", "const library: LibraryContext = { stages: ['discover'], surface: 'tool' };", "export const partial = renderRoute('tool:curator/status', { context: { providers: { library } } });", - "export const absent = renderRoute('tool:curator/status');", '', ].join('\n')), ]); @@ -160,7 +163,6 @@ it('types (await agent()).providers. from the generated provider declaratio expect(missingProviders).toHaveLength(1); expect(missingProviders[0]).toContain("Property 'providers' is missing"); const missingFixture = typecheck(root, 'missing-fixture.ts'); - expect(missingFixture).toHaveLength(2); + expect(missingFixture).toHaveLength(1); expect(missingFixture[0]).toContain("Property '\"buildNumber\"' is missing"); - expect(missingFixture[1]).toContain('Expected 2 arguments, but got 1.'); }); diff --git a/packages/agent-bundle/tests/route-unit/render-route.test.ts b/packages/agent-bundle/tests/route-unit/render-route.test.ts index ca1ed8511..9343d3a04 100644 --- a/packages/agent-bundle/tests/route-unit/render-route.test.ts +++ b/packages/agent-bundle/tests/route-unit/render-route.test.ts @@ -263,7 +263,7 @@ describe('renderRoute through the real renderer', () => { .toHaveValue(undefined); }); - it('mounts provider fixture values through the context seam instead of executing provider modules', async () => { + it('mounts explicit provider fixture values through the context seam instead of discovering providers', async () => { const library = { stages: ['discover', 'curate'], tooling: { ffmpeg: { available: false } } }; const Providers = async (): Promise => { const { providers } = await agent(); @@ -287,12 +287,14 @@ describe('renderRoute through the real renderer', () => { library, }); - // A render without fixtures observes an empty, frozen provider map — the - // harness never runs conventional src/providers modules on the test's behalf. + // A module rendered directly has no compiled manifest, so there is nothing + // to discover: it observes only the framework-owned process identity, the + // same map a generated scope without providers mounts. Manifest routes + // execute the project's conventional providers (projection/providers.test.ts). const unfixtured = await renderRoute({ default: Providers as never }, { routeId: 'tool:harness/providers (module)', }); - expectDocument(unfixtured).toHaveValue({ frozen: true, keys: [], library: undefined }); + expectDocument(unfixtured).toHaveValue({ frozen: true, keys: ['processLifetime'], library: undefined }); }); it('serves useAgent() synchronously inside a rendered Server Component', async () => { diff --git a/packages/agent-bundle/tests/support/contract-matrix-fixtures.ts b/packages/agent-bundle/tests/support/contract-matrix-fixtures.ts index 34a456dc0..0cd1bb063 100644 --- a/packages/agent-bundle/tests/support/contract-matrix-fixtures.ts +++ b/packages/agent-bundle/tests/support/contract-matrix-fixtures.ts @@ -101,6 +101,7 @@ export const routeHarnessContractFixtures = (): Record => { + const ids = new Set(); + if (status.build.state === 'building') ids.add(status.build.activeAttempt.id); + if ('lastAttempt' in status.build && status.build.lastAttempt !== undefined) ids.add(status.build.lastAttempt.id); + return ids; +}; + +const pollIntervalMs = 25; + +/** + * Replaces one watched source and waits for the dev server's own + * watcher-driven rebuild of that write to finish, returning the completed + * attempt. This is the readiness signal a source edit needs: the coordinator + * publishes every attempt through `status()`, and the watcher mints exactly + * one invalidation per write (path-signature dedupe, #329), so the first + * completed attempt the session did not know before the write is the + * write's build. Issuing a manual rebuild in the same window instead races + * the watcher for a second, redundant epoch whose arrival time depends on + * load — the race behind the retired `{ retry: 2 }` guards in + * `examples-real.e2e.test.ts`. + */ +export const replaceWatchedSourceAndAwaitRebuild = async ( + session: WatchedBuildSession, + projectRoot: string, + path: string, + content: string, + options: AwaitWatcherRebuildOptions, +): Promise => { + const known = attemptIds(session.status()); + await replaceWatchedSource(projectRoot, path, content); + const deadline = Date.now() + options.timeoutMs; + for (;;) { + const status = session.status(); + if ( + status.build.state !== 'building' + && status.build.lastAttempt !== undefined + && !known.has(status.build.lastAttempt.id) + ) { + return status.build.lastAttempt; + } + if (Date.now() >= deadline) { + throw new Error( + `Timed out after ${String(options.timeoutMs)}ms waiting for the watcher rebuild of ${path}; ` + + `known attempts ${JSON.stringify([...known])}; last status ${JSON.stringify(status.build)}.`, + ); + } + await sleep(pollIntervalMs); + } +}; diff --git a/packages/agent-bundle/tests/test-harness-manifest.test.ts b/packages/agent-bundle/tests/test-harness-manifest.test.ts index d181fc360..74f7faa25 100644 --- a/packages/agent-bundle/tests/test-harness-manifest.test.ts +++ b/packages/agent-bundle/tests/test-harness-manifest.test.ts @@ -67,9 +67,12 @@ describe('the compiled test manifest', () => { 'cli:db/migrate', 'cli:inventory', 'cli:report', + 'cli:tooling/inspect', + 'cli:tooling/report', 'event:tool/after', 'prompt:harness/summarize', 'resource:harness/notes', + 'script:tooling-summary', 'tool:harness/catalog', 'tool:harness/context', 'tool:harness/echo', @@ -79,10 +82,18 @@ describe('the compiled test manifest', () => { 'tool:harness/publish-notice', 'tool:harness/strict-report', 'tool:harness/ticket', + 'tool:harness/tooling', 'tool:harness/unavailable', 'tool:harness/wait', ]); expect(manifest.diagnostics).toEqual([]); + expect(manifest.providers).toEqual([{ + id: 'provider:library-tooling', + key: 'libraryTooling', + name: 'library-tooling', + relativePath: 'src/providers/library-tooling.ts', + source: resolve(fixtureRoot, 'src/providers/library-tooling.ts'), + }]); expect(manifest.routes['tool:harness/echo']).toEqual({ config: { annotations: { readOnlyHint: true }, @@ -157,6 +168,24 @@ describe('the compiled test manifest', () => { rendered: true, routeId: 'cli:report', }, + { + aliases: [], + description: 'Reports the request providers a plain command observes.', + exitCode: 'zero', + options: [], + path: ['tooling', 'inspect'], + rendered: false, + routeId: 'cli:tooling/inspect', + }, + { + aliases: [], + description: 'Renders the request providers a rendered command observes.', + exitCode: 'zero', + options: [], + path: ['tooling', 'report'], + rendered: true, + routeId: 'cli:tooling/report', + }, ]); const inputOption = { description: 'Tool input as one JSON object.', @@ -198,6 +227,7 @@ describe('the compiled test manifest', () => { projected('publish-notice', 'Publishes a durable notice for a later session event.', true), projected('strict-report', 'Returns a closed-object report that rejects unknown serialized keys.', true), projected('ticket', 'Returns a cargo-conductor-shaped ticket status with optional diagnostics fields.', true), + projected('tooling', 'Reports the request providers an MCP tool observes.', false), projected('unavailable', 'Returns a typed unavailable result for projection checks.', true), projected('wait', 'Waits until aborted or holdMs elapses, for cancellation contract proof.', true), ]); @@ -299,6 +329,15 @@ describe('the generated route registry', () => { expect(source).toContain('app:harness/panel'); }); + it('registers a loader for every conventional provider so the harness mounts them like the entry shell', () => { + const providerLoaders = /providerLoaders: \{\n(?[\s\S]*?)\n {2}\},/u.exec(source)?.groups?.body ?? ''; + + expect(providerLoaders).toContain('"provider:library-tooling": () => import('); + expect(providerLoaders).toContain('/src/providers/library-tooling.ts'); + // A project without providers emits no loader table at all. + expect(routeTestSetupSource({ ...manifest, providers: undefined })).not.toContain('providerLoaders'); + }); + it('carries the manifest and the registry version the helpers require', () => { expect(source).toContain(`version: ${String(AGENT_TEST_REGISTRY_VERSION)}`); expect(source).toContain('globalThis[Symbol.for("agent-bundle/test-route-registry")]'); diff --git a/packages/agent-bundle/tests/watched-files-support.test.ts b/packages/agent-bundle/tests/watched-files-support.test.ts new file mode 100644 index 000000000..42d2ee38c --- /dev/null +++ b/packages/agent-bundle/tests/watched-files-support.test.ts @@ -0,0 +1,100 @@ +import { mkdir, mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, beforeEach, describe, expect, it } from '@rstest/core'; + +import type { CompletedBuildAttempt, ProjectStatus, RunningBuildAttempt } from '../src/dev/types.ts'; +import { replaceWatchedSourceAndAwaitRebuild } from './support/watched-files.ts'; + +const running = (id: string): RunningBuildAttempt => Object.freeze({ + diagnostics: Object.freeze([]), + id, + outcome: 'running', + sourceRevision: 'rev', + startedAt: '2026-09-02T00:00:00.000Z', +}); + +/** A completed attempt; failed so the fake needs no artifact epoch. The wait is outcome-agnostic. */ +const completed = (id: string): CompletedBuildAttempt => Object.freeze({ + completedAt: '2026-09-02T00:00:01.000Z', + diagnostics: Object.freeze([ + Object.freeze({ code: 'AB7201', message: `attempt ${id} failed`, severity: 'error' as const }), + ] as const), + id, + outcome: 'failed', + sourceRevision: 'rev', + startedAt: '2026-09-02T00:00:00.000Z', +}); + +const idle = (lastAttempt?: CompletedBuildAttempt): ProjectStatus => Object.freeze({ + artifact: Object.freeze({ state: 'missing' }), + build: Object.freeze(lastAttempt === undefined ? { state: 'idle' } : { lastAttempt, state: 'idle' }), + source: Object.freeze({ diagnostics: Object.freeze([]), state: 'ready' }), +}); + +const building = (active: RunningBuildAttempt, lastAttempt?: CompletedBuildAttempt): ProjectStatus => Object.freeze({ + artifact: Object.freeze({ state: 'missing' }), + build: Object.freeze({ activeAttempt: active, ...(lastAttempt === undefined ? {} : { lastAttempt }), state: 'building' }), + source: Object.freeze({ diagnostics: Object.freeze([]), state: 'ready' }), +}); + +/** + * The readiness wait behind the `examples-real.e2e` source edits: the first + * completed attempt the session did not already know is the write's build. + * A session that reports the pre-write attempt, or a still-running one, is + * not ready yet. + */ +describe('replaceWatchedSourceAndAwaitRebuild', () => { + let root: string; + let project: string; + + beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), 'agent-bundle-watched-files-')); + project = join(root, 'project'); + await mkdir(project); + }); + + afterEach(async () => { + await rm(root, { force: true, recursive: true }); + }); + + it('returns the first completed attempt that was unknown before the write, after the write landed', async () => { + const before = completed('attempt-0'); + const sequence: ProjectStatus[] = [ + idle(before), + idle(before), + building(running('attempt-1'), before), + idle(completed('attempt-1')), + ]; + let reads = 0; + const session = { status: () => sequence[Math.min(reads++, sequence.length - 1)]! }; + const path = join(project, 'source.ts'); + + const attempt = await replaceWatchedSourceAndAwaitRebuild(session, project, path, 'export const value = 2;\n', { timeoutMs: 5_000 }); + + expect(attempt.id).toBe('attempt-1'); + expect(attempt.outcome).toBe('failed'); + expect(await readFile(path, 'utf8')).toBe('export const value = 2;\n'); + // The pre-write status read seeds the known set; the wait began only after the write. + expect(reads).toBeGreaterThanOrEqual(4); + }); + + it('does not accept the pre-write attempt as the write\'s build', async () => { + const before = completed('attempt-0'); + const session = { status: () => idle(before) }; + + await expect(replaceWatchedSourceAndAwaitRebuild(session, project, join(project, 'source.ts'), 'x', { timeoutMs: 120 })) + .rejects.toThrow(/Timed out after 120ms waiting for the watcher rebuild .*known attempts \["attempt-0"\]/u); + }); + + it('treats an attempt that was already running before the write as known', async () => { + const active = running('attempt-1'); + const sequence: ProjectStatus[] = [building(active), idle(completed('attempt-1'))]; + let reads = 0; + const session = { status: () => sequence[Math.min(reads++, sequence.length - 1)]! }; + + await expect(replaceWatchedSourceAndAwaitRebuild(session, project, join(project, 'source.ts'), 'x', { timeoutMs: 120 })) + .rejects.toThrow(/known attempts \["attempt-1"\]/u); + }); +}); diff --git a/packages/workbench/tests/examples-real.e2e.test.ts b/packages/workbench/tests/examples-real.e2e.test.ts index 4a9d87273..866bacfef 100644 --- a/packages/workbench/tests/examples-real.e2e.test.ts +++ b/packages/workbench/tests/examples-real.e2e.test.ts @@ -15,10 +15,36 @@ import { writeExampleReport, } from './support/example-acceptance.ts'; import { timeScale } from '../../agent-bundle/tests/support/time-scale.ts'; -import { replaceWatchedSource } from '../../agent-bundle/tests/support/watched-files.ts'; +import { + replaceWatchedSourceAndAwaitRebuild, + type WatchedBuildSession, +} from '../../agent-bundle/tests/support/watched-files.ts'; import { buildWorkbench, e2e, workbenchAssets, workbenchUrl } from './support/workbench-e2e.ts'; const browserTimeout = 15_000 * timeScale; +/** One watcher debounce plus a full development rebuild of an example under gate load. */ +const rebuildTimeout = 60_000 * timeScale; + +/** + * Edits one watched source and waits for the dev server's own rebuild of that + * edit to complete before the browser is asked about it. A source write must + * not be paired with an immediate manual rebuild: the watcher rebuilds the + * same write on its own, and two builds per edit mint two epochs whose + * relative timing depends on load — the second one flips the Workbench's + * build identity while it may still be loading capabilities for the first. + * Waiting on the coordinator's published attempt makes one edit exactly one + * build, so no retry is needed to absorb that race. + */ +const editWatchedSource = async ( + server: WatchedBuildSession, + projectRoot: string, + path: string, + content: string, + expectedOutcome: 'failed' | 'succeeded', +): Promise => { + const attempt = await replaceWatchedSourceAndAwaitRebuild(server, projectRoot, path, content, { timeoutMs: rebuildTimeout }); + expect(attempt.outcome).toBe(expectedOutcome); +}; const waitForExampleValue = async ( page: Parameters[0], @@ -40,20 +66,6 @@ const waitForExampleValue = async ( return value; }; -const rebuildFromCurrentPage = async (page: Parameters[0]): Promise => { - const status = await page.evaluate(async () => { - const sessionResponse = await fetch('/api/project/session'); - const session = await sessionResponse.json() as { readonly token: string }; - const response = await fetch('/api/project/rebuild', { - body: JSON.stringify({ paths: [] }), - headers: { 'content-type': 'application/json', 'x-agent-bundle-session': session.token }, - method: 'POST', - }); - return response.status; - }); - expect(status).toBe(200); -}; - e2e('drives the populated Skills Starter in real Chrome', { timeout: 90_000 }, async ({ page }) => { await buildWorkbench(); const server = await startDevServer({ @@ -108,9 +120,7 @@ e2e('drives the populated Skills Starter in real Chrome', { timeout: 90_000 }, a } }); -// #122's delayed duplicate event is signature-gated; this retry still covers the first -// Chokidar event racing the immediate manual rebuild after each source write. -e2e('reveals, retains, repairs, and removes capabilities without reloading Chrome', { retry: 2, timeout: 120_000 }, async ({ page }) => { +e2e('reveals, retains, repairs, and removes capabilities without reloading Chrome', { timeout: 120_000 * timeScale }, async ({ page }) => { await buildWorkbench(); const project = await copyExample('skills-starter'); const configPath = join(project.root, 'agent-bundle.config.ts'); @@ -137,8 +147,7 @@ e2e('reveals, retains, repairs, and removes capabilities without reloading Chrom await expect(page.getByRole('link', { name: 'Hooks', exact: true })).toHaveCount(0, { timeout: browserTimeout }); await expect(page.getByRole('link', { name: 'Playground', exact: true })).toHaveCount(0, { timeout: browserTimeout }); - await writeFile(configPath, hookConfig); - await rebuildFromCurrentPage(page); + await editWatchedSource(server, project.root, configPath, hookConfig, 'succeeded'); await expect(page.getByRole('link', { name: 'Hooks', exact: true })).toBeVisible({ timeout: browserTimeout }); await expect(page.getByRole('link', { name: 'Playground', exact: true })).toBeVisible({ timeout: browserTimeout }); await page.getByRole('link', { name: 'Hooks', exact: true }).click(); @@ -146,8 +155,7 @@ e2e('reveals, retains, repairs, and removes capabilities without reloading Chrom await expect(page.locator('#hook-binding option')).not.toHaveCount(0, { timeout: browserTimeout }); await captureExampleState(page, 'skills-starter', 'capability-revealed'); - await writeFile(hookSource, 'export default () => ({\n'); - await rebuildFromCurrentPage(page); + await editWatchedSource(server, project.root, hookSource, 'export default () => ({\n', 'failed'); await page.getByRole('link', { name: 'Overview', exact: true }).click(); await waitForSettledWorkbench(page); await expect(page.getByRole('heading', { name: /Diagnostics \([1-9]/u })).toBeVisible({ timeout: browserTimeout }); @@ -156,8 +164,7 @@ e2e('reveals, retains, repairs, and removes capabilities without reloading Chrom await expect(page.getByRole('link', { name: 'Playground', exact: true })).toBeVisible({ timeout: browserTimeout }); await captureExampleState(page, 'skills-starter', 'capability-stale'); - await writeFile(hookSource, healthyHook); - await rebuildFromCurrentPage(page); + await editWatchedSource(server, project.root, hookSource, healthyHook, 'succeeded'); await expect(page.getByRole('heading', { name: 'Diagnostics (0)' })).toBeVisible({ timeout: browserTimeout }); await expect(page.locator('.build-health')).toContainText('Current build', { timeout: browserTimeout }); await captureExampleState(page, 'skills-starter', 'capability-repaired'); @@ -165,8 +172,7 @@ e2e('reveals, retains, repairs, and removes capabilities without reloading Chrom await page.getByRole('link', { name: 'Hooks', exact: true }).click(); await waitForSettledWorkbench(page); await expect(page.locator('#hook-binding option')).not.toHaveCount(0, { timeout: browserTimeout }); - await writeFile(configPath, originalConfig); - await rebuildFromCurrentPage(page); + await editWatchedSource(server, project.root, configPath, originalConfig, 'succeeded'); await expect(page).toHaveURL(new URL('#overview', server.url).href, { timeout: browserTimeout }); await expect(page.getByRole('link', { name: 'Hooks', exact: true })).toHaveCount(0, { timeout: browserTimeout }); await expect(page.getByRole('link', { name: 'Playground', exact: true })).toHaveCount(0, { timeout: browserTimeout }); @@ -179,9 +185,7 @@ e2e('reveals, retains, repairs, and removes capabilities without reloading Chrom } }); -// #122's delayed duplicate event is signature-gated; this retry still covers the first -// Chokidar event racing the immediate manual rebuild after each source write. -e2e('drives Hooks, scripts, logs, diagnostics, and repair in real Chrome', { retry: 2, timeout: 150_000 }, async ({ page }) => { +e2e('drives Hooks, scripts, logs, diagnostics, and repair in real Chrome', { timeout: 150_000 * timeScale }, async ({ page }) => { await buildWorkbench(); const project = await copyExample('hooks-and-scripts'); const hookSource = join(project.root, 'src', 'hooks', 'session-start.ts'); @@ -270,25 +274,18 @@ e2e('drives Hooks, scripts, logs, diagnostics, and repair in real Chrome', { ret await expect(page.locator('.logs-details').first()).toHaveAttribute('open', ''); await captureExampleState(page, 'hooks-and-scripts', 'logs-populated'); - await writeFile(hookSource, 'export default () => ({\n'); + // The stale-diagnostic and repair journey rides the watcher's own rebuild + // of each edit; the Rebuild button's manual path is overview.e2e's claim. await page.getByRole('link', { name: 'Overview' }).click(); - const failedRebuild = page.waitForResponse((response) => response.url() === `${server.url}/api/project/rebuild` && response.request().method() === 'POST'); - await page.getByRole('button', { name: 'Rebuild' }).click(); - await failedRebuild; + await waitForSettledWorkbench(page); + await editWatchedSource(server, project.root, hookSource, 'export default () => ({\n', 'failed'); await expect(page.getByRole('heading', { name: /Diagnostics \([1-9]/u })).toBeVisible({ timeout: browserTimeout }); await expect(page.locator('.build-health')).toContainText('Last good build', { timeout: browserTimeout }); - await page.waitForTimeout(500); - await expect(page.locator('.build-health')).toContainText('Last good build', { timeout: browserTimeout }); await captureExampleState(page, 'hooks-and-scripts', 'diagnostic-stale'); - await writeFile(hookSource, healthyHook); - const repaired = page.waitForResponse((response) => response.url() === `${server.url}/api/project/rebuild` && response.request().method() === 'POST' && response.ok()); - await page.getByRole('button', { name: 'Rebuild' }).click(); - await repaired; + await editWatchedSource(server, project.root, hookSource, healthyHook, 'succeeded'); await expect(page.getByRole('heading', { name: 'Diagnostics (0)' })).toBeVisible({ timeout: browserTimeout }); await expect(page.locator('.build-health')).toContainText('Current build', { timeout: browserTimeout }); - await page.waitForTimeout(500); - await expect(page.locator('.build-health')).toContainText('Current build', { timeout: browserTimeout }); await captureExampleState(page, 'hooks-and-scripts', 'diagnostic-repaired'); await expectHealthyExamplePage(ledger); await writeExampleReport(); @@ -575,9 +572,7 @@ e2e('drives every populated MCP App workflow surface in real Chrome', { timeout: } }); -// #122's delayed duplicate event is signature-gated; this retry still covers the first -// Chokidar event racing the immediate manual rebuild after each staged source replacement. -e2e('renders the flagship compiled route catalog by server and kind in real Chrome', { retry: 2, timeout: 150_000 }, async ({ page }) => { +e2e('renders the flagship compiled route catalog by server and kind in real Chrome', { timeout: 150_000 * timeScale }, async ({ page }) => { await buildWorkbench(); const project = await copyExample('audiobook-curator'); const conversionSource = join(project.root, 'src', 'conversion.ts'); @@ -707,8 +702,7 @@ e2e('renders the flagship compiled route catalog by server and kind in real Chro // A prepared source revision can move ahead while a failed rebuild keeps // the published epoch intact. Reloading the same browser page re-reads that // prepared manifest and must identify it as stale until a repair publishes. - await replaceWatchedSource(project.root, conversionSource, `${healthyConversion}\nconst = ;\n`); - await rebuildFromCurrentPage(page); + await editWatchedSource(server, project.root, conversionSource, `${healthyConversion}\nconst = ;\n`, 'failed'); await page.reload(); await waitForSettledWorkbench(page); await expect(page.locator('.route-state')).toHaveText('stale', { timeout: browserTimeout }); @@ -718,8 +712,7 @@ e2e('renders the flagship compiled route catalog by server and kind in real Chro ); await captureExampleState(page, 'audiobook-curator', 'routes-catalog-stale'); - await replaceWatchedSource(project.root, conversionSource, healthyConversion); - await rebuildFromCurrentPage(page); + await editWatchedSource(server, project.root, conversionSource, healthyConversion, 'succeeded'); await waitForSettledWorkbench(page); await expect(page.locator('.route-state')).toHaveText('current', { timeout: browserTimeout }); await expect(page.locator('.routes-page-heading')).toContainText( diff --git a/packages/workbench/tests/logs-real.e2e.test.ts b/packages/workbench/tests/logs-real.e2e.test.ts index 8ad816c69..9628f163c 100644 --- a/packages/workbench/tests/logs-real.e2e.test.ts +++ b/packages/workbench/tests/logs-real.e2e.test.ts @@ -1,11 +1,10 @@ -import { writeFile } from 'node:fs/promises'; - import { expect } from '@rstest/playwright'; import { createWorkbenchAssetSource } from '../../agent-bundle/src/dev/workbench-assets.ts'; import { startDevServer } from '../../agent-bundle/src/dev/workbench-server.ts'; import { createProjectFixture, removeProjectFixture } from '../../agent-bundle/tests/helpers/project-fixture.ts'; import { timeScale } from '../../agent-bundle/tests/support/time-scale.ts'; +import { replaceWatchedSource } from '../../agent-bundle/tests/support/watched-files.ts'; import { buildWorkbench, e2e, workbenchAssets, workbenchUrl } from './support/workbench-e2e.ts'; const browserTimeout = 12_000 * timeScale; @@ -31,7 +30,9 @@ e2e('shows real producer logs with replay, filters, redaction, responsive layout const replay = await (await replayed).json() as { readonly replay: Readonly<{ readonly records: readonly unknown[] }> }; expect(replay.replay.records.length).toBeGreaterThan(0); - await writeFile(project.skillSource, `${project.skillMarkdown}\nSource change for Logs E2E.\n`); + // One atomic replacement is one watcher invalidation; a truncating write + // can split into two under load and log "Project source changed." twice. + await replaceWatchedSource(project.root, project.skillSource, `${project.skillMarkdown}\nSource change for Logs E2E.\n`); await expect(page.getByText('Project source changed.')).toBeVisible({ timeout: browserTimeout }); await expect(page.locator('.logs-entries > li').first()).toBeVisible({ timeout: browserTimeout }); await expect(page.locator('.logs-entry-level').first()).toBeVisible();