Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/xref-registered-route-ids.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"agent-bundle": patch
"@agent-bundle/runtime": patch
---

Carry the route registration that `.agent-bundle/routes.d.ts` places on `@agent-bundle/runtime`'s `Register` through the rest of the public API, not only `renderRoute`, the way TanStack Router's one `Register` reaches `Link to`, `useNavigate`, and `RoutesByPath`. In `agent-bundle/test`, `invokeMcpTool` and `getMcpPrompt` now check a literal wire name against the registered tool/prompt names and type `input` from that route — of the literal `server` when one is passed, which is itself checked against the compiled server names (`McpInvocationOptions<Input, Server>`, `McpRouteNameConstraint`, `McpRouteInput`, `McpServerConstraint`, `McpRouteServer`); the `fixtures` of `runContractMatrix`, `runPackedContractMatrix`, `runDevEpochContractMatrix`, and `runInstalledHostContractMatrix` type each registered key's `input`, `inputs`, `cancellation.input`, and lifecycle transitions (`ContractRouteFixtures`, `ContractRouteFixture<Input>`, `ContractLifecycleFixture<Input>`, `ContractLifecycleTransition<Input>`) while MCP App keys and dynamic records stay legal; and `invokeCli` reports `CliInvocation.routeId` as a `RegisteredRouteId` (`argv` is unchanged). In `agent-bundle/eval`, `expectMcpCall` and `expectNoMcpCall` check a literal `tool` against the registered tools of a literal project `server` (`ExpectMcpCallOptions`, `ExpectNoMcpCallOptions`, `EvalMcpToolConstraint`); third-party servers stay free. `@agent-bundle/runtime` adds `RegisteredMcpRouteKind`, `RegisteredMcpServerName`, `RegisteredMcpRouteName`, and `RegisteredMcpRouteId` for the server and protocol names a registered id encodes. Type-only: nothing changes at run time, and every surface keeps its `string`/`unknown` shape when no project has registered. (#494)
18 changes: 18 additions & 0 deletions docs/framework-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,24 @@ lookups, a directly imported module target is unaffected, and a project that
excludes the generated declarations (or has not built yet) sees the
unregistered types — any string, `unknown` input, `unknown` result.

That one registration is read by every public surface that takes or yields a
route id or route payload, the way TanStack Router's `Register` reaches `Link
to`, `useNavigate`, and `RoutesByPath`: `invokeMcpTool` and `getMcpPrompt`
check their wire name against the registered tool/prompt names (the last
segment of a `tool:`/`prompt:` id) — of the literal `server` when one is
passed, since the session mounts only that server's routes — and type
`input` from that route;
`runContractMatrix` and the packed, dev-epoch, and installed-host matrices type
the inputs of each registered `fixtures` key (an App route key stays untyped —
Apps register no contract); `invokeCli` reports `routeId` as a registered id;
and `agent-bundle/eval`'s `expectMcpCall`/`expectNoMcpCall` check a literal
`tool` against the registered tools of a literal project `server`.
`RegisteredMcpServerName`, `RegisteredMcpRouteName`, and `RegisteredMcpRouteId`
expose the server and protocol names an id encodes. `readMcpResource` (a wire
URI), `runScript` (no registered contract), the wire `structuredContent`
(object-valued documents only), and `agent-bundle/api` (an arbitrary project
`root`) stay `string`/`unknown` on purpose.

### What reaches the MCP wire

The final Agent Document of a tool route lowers to one `CallToolResult`:
Expand Down
14 changes: 14 additions & 0 deletions packages/agent-bundle/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,20 @@ name that surface for wrappers. A value typed `string`, a module target, or a
program without the generated file sees the previous types — any id, `unknown`
input and result.

The registration flows to every harness surface that takes a route id or
payload, not only `renderRoute`: `invokeMcpTool('find', { input })` and
`getMcpPrompt` check the wire name against the registered tool/prompt names and
type `input` from that route — of the literal `server`, when passed
(`RegisteredMcpServerName`, `RegisteredMcpRouteName`,
and `RegisteredMcpRouteId` name what a `tool:<server>/<name>` id encodes); the
contract matrices type each registered key of `fixtures` while an MCP App key
or a dynamic `Record<string, ContractRouteFixture>` stays legal; `invokeCli`
reports `routeId` as a registered id (`argv` is untouched); and
`agent-bundle/eval`'s `expectMcpCall` checks a literal `tool` against the
registered tools of a literal project `server`. `readMcpResource` (a URI),
`runScript` (scripts register no contract), and `structuredContent` (carried
only for object-valued documents) deliberately stay untyped.

Conventional request context providers (`src/providers/*`, see
[entry conventions](../../docs/entry-conventions.md#request-context-providers-power-tier))
are mounted automatically for every manifest-backed helper — `renderRoute`,
Expand Down
43 changes: 35 additions & 8 deletions packages/agent-bundle/src/eval/assertions.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import type { RegisteredMcpRouteName, RegisteredMcpServerName } from '@agent-bundle/runtime';

import { digest } from '../core/digest.ts';
import { EvalDefinitionError } from './errors.ts';
import { claudeSemanticGraderId } from './graders.ts';
Expand All @@ -18,15 +20,36 @@ export interface EvalEvidenceOptions {
readonly minimumEvidence?: ActivationEvidence;
}

export interface ExpectMcpCallOptions extends EvalEvidenceOptions {
/**
* The constraint an MCP-call assertion's `tool` must satisfy for `Server`.
* Once the generated `.agent-bundle/routes.d.ts` registers the project's
* routes and `server` is a literal naming one of the project's own compiled
* MCP servers, a literal `tool` must be one of that server's registered tool
* names (`find` for `tool:curator/find`) — a typo is rejected naming the
* alternatives. A server outside the registration (an assertion about a
* third-party MCP server the host also exposes), a value typed `string`, or an
* unregistered project keeps `string`, exactly as before. The `& string`
* reduces the alias instantiation to a literal union so a rejection lists the
* server's tool names.
*/
export type EvalMcpToolConstraint<Server extends string, Tool> = Server extends RegisteredMcpServerName
? string extends Tool ? string : RegisteredMcpRouteName<'tool', Server> & string
: string;

/**
* `server` is the MCP server name as the host trace records it and `tool` the
* wire tool name; both are inferred from literals so `tool` checks against the
* project's registered tools of that server (see {@link EvalMcpToolConstraint}).
*/
export interface ExpectMcpCallOptions<Server extends string = string, Tool extends string = string> extends EvalEvidenceOptions {
readonly atLeast?: number;
readonly server: string;
readonly tool: string;
readonly server: Server;
readonly tool: (Tool & EvalMcpToolConstraint<Server, Tool>) | EvalMcpToolConstraint<Server, Tool>;
}

export interface ExpectNoMcpCallOptions extends EvalEvidenceOptions {
readonly server: string;
readonly tool?: string;
export interface ExpectNoMcpCallOptions<Server extends string = string, Tool extends string = string> extends EvalEvidenceOptions {
readonly server: Server;
readonly tool?: (Tool & EvalMcpToolConstraint<Server, Tool>) | EvalMcpToolConstraint<Server, Tool>;
}

export interface ExpectOutcomeOptions extends EvalEvidenceOptions {
Expand Down Expand Up @@ -95,7 +118,9 @@ export const expectExitCode = (
return Object.freeze({ ...expectation, id: assertionId(expectation.kind, expectation) });
};

export const expectMcpCall = (options: ExpectMcpCallOptions): EvalMcpCallAssertion => {
export const expectMcpCall = <Server extends string, Tool extends string>(
options: ExpectMcpCallOptions<Server, Tool>,
): EvalMcpCallAssertion => {
const expectation = {
atLeast: requireCount(options.atLeast ?? 1, 'Expected MCP call count', 1),
kind: 'mcp-call' as const,
Expand All @@ -106,7 +131,9 @@ export const expectMcpCall = (options: ExpectMcpCallOptions): EvalMcpCallAsserti
return Object.freeze({ ...expectation, id: assertionId(expectation.kind, expectation) });
};

export const expectNoMcpCall = (options: ExpectNoMcpCallOptions): EvalNoMcpCallAssertion => {
export const expectNoMcpCall = <Server extends string, Tool extends string>(
options: ExpectNoMcpCallOptions<Server, Tool>,
): EvalNoMcpCallAssertion => {
const expectation = {
kind: 'no-mcp-call' as const,
minimumEvidence: requireMinimumEvidence(options.minimumEvidence, 'observed'),
Expand Down
1 change: 1 addition & 0 deletions packages/agent-bundle/src/eval/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ export {
} from './assertions.ts';
export type {
EvalEvidenceOptions,
EvalMcpToolConstraint,
ExpectMcpCallOptions,
ExpectNoMcpCallOptions,
ExpectNoSkillActivationOptions,
Expand Down
14 changes: 12 additions & 2 deletions packages/agent-bundle/src/test/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
* wire behavior.
*/
import type * as AgentRuntime from '@agent-bundle/runtime';
import type { RegisteredRouteId } from '@agent-bundle/runtime';

import { CliInputError, runGeneratedCliEntry } from '../cli-entry.ts';
import type { CliRenderedEvent } from '../cli-entry.ts';
Expand Down Expand Up @@ -63,7 +64,15 @@ export interface CliInvocation {
*/
readonly exitCode: number;
readonly provenance: CliDispatchProvenance;
readonly routeId?: string;
/**
* The compiled route the shell executed: a `cli:` route, or the `tool:`
* route behind a projected MCP command. Typed from the project's route
* registration once `.agent-bundle/routes.d.ts` is in the program (both
* kinds register), `string` without it; absent for help, `--version`, and
* usage failures. `argv` itself stays `readonly string[]` — it is the shell's
* input, not a route id.
*/
readonly routeId?: RegisteredRouteId;
/** Everything the shell wrote to its diagnostic stream. */
readonly stderr: string;
/** Everything the shell wrote to stdout, including rendered Markdown, TTY, JSON, or NDJSON output. */
Expand Down Expand Up @@ -292,7 +301,8 @@ export const invokeCli = async (

return Object.freeze({
argv: Object.freeze([...argv]),
...(executed === undefined ? {} : { command: commandPath(executed), routeId: executed.routeId }),
// The compiled command graph's ids are the ones the registration lists.
...(executed === undefined ? {} : { command: commandPath(executed), routeId: executed.routeId as RegisteredRouteId }),
exitCode,
provenance,
stderr: err,
Expand Down
60 changes: 45 additions & 15 deletions packages/agent-bundle/src/test/contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
* Boundaries without an event runtime report identity as not-applicable.
*/
import type { Client } from '@modelcontextprotocol/client';
import type { RegisteredRouteId, RegisteredRouteInput } from '@agent-bundle/runtime';

import {
requestEventRuntimeStatus,
Expand Down Expand Up @@ -92,20 +93,24 @@ export type ContractLifecyclePhase =
| 'repeated-progress'
| 'terminal';

export interface ContractLifecycleTransition {
/**
* One lifecycle phase's invocation. `Input` is the route's input, bound to
* its registered type by {@link ContractRouteFixtures} for a registered id.
*/
export interface ContractLifecycleTransition<Input = unknown> {
readonly expectedStructuredContent: unknown;
readonly input: unknown;
readonly input: Input;
readonly phase: ContractLifecyclePhase;
readonly progressNotifications: number;
readonly renderedTextIncludes?: string;
}

export interface ContractLifecycleFixture {
export interface ContractLifecycleFixture<Input = unknown> {
readonly state?: {
readonly budget?: {
readonly codePath: readonly string[];
readonly expectedCode: string;
readonly input: unknown;
readonly input: Input;
readonly revisionPath: readonly string[];
};
readonly catalog?: {
Expand All @@ -114,7 +119,7 @@ export interface ContractLifecycleFixture {
};
readonly durability?: {
readonly expectedStructuredContent: unknown;
readonly input: unknown;
readonly input: Input;
};
readonly idempotency?: {
readonly phase: ContractLifecyclePhase;
Expand All @@ -132,7 +137,7 @@ export interface ContractLifecycleFixture {
};
};
/** Pure deterministic phase driver; transport and assertions remain framework-owned. */
readonly transitionDriver: () => readonly ContractLifecycleTransition[];
readonly transitionDriver: () => readonly ContractLifecycleTransition<Input>[];
}

/**
Expand All @@ -144,17 +149,25 @@ export interface ContractResourceFixture {
readonly kind: 'resource';
}

export interface ContractRouteFixture {
/**
* One route's contract fixture. Every `input` here is handed to the route as
* its tool or prompt input, so `Input` is that route's input type:
* {@link ContractRouteFixtures} binds it to the registered input for a
* registered route id and leaves it `unknown` otherwise. `previousResults`
* stays `unknown` by design — those payloads come from previous server
* versions and need not match the current schema's type.
*/
export interface ContractRouteFixture<Input = unknown> {
/**
* `'resource'` marks a resource/MCP App fixture (see `ContractResourceFixture`).
* Omit it for tool and prompt fixtures; a legacy `{}` still covers a
* resource or app route.
*/
readonly kind?: ContractResourceFixture['kind'];
/** Valid input for the sweep invocation (tools/prompts; resources need none). */
readonly input?: unknown;
readonly input?: Input;
/** Additional valid inputs — e.g. one per declared status/discriminant value. */
readonly inputs?: readonly unknown[];
readonly inputs?: readonly Input[];
/** Declared serialized-result compatibility policy. REQUIRED for tool routes. */
readonly resultCompat?: ResultCompatPolicy;
/**
Expand All @@ -168,11 +181,28 @@ export interface ContractRouteFixture {
* `abortAfterMs` (default 50ms); an invocation that settles before the abort
* fires is reported `not-applicable`, not `failed`.
*/
readonly cancellation?: { readonly abortAfterMs?: number; readonly input?: unknown };
readonly cancellation?: { readonly abortAfterMs?: number; readonly input?: Input };
/** Optional stateful replay over this matrix run's single open client. */
readonly lifecycle?: ContractLifecycleFixture;
readonly lifecycle?: ContractLifecycleFixture<Input>;
}

/**
* Route id -> fixture, the `fixtures` member of every contract-matrix entry
* point. Once the generated `.agent-bundle/routes.d.ts` registers the
* project's routes, a registered id's fixture carries that route's registered
* input in `input`, `inputs`, `cancellation.input`, and its lifecycle
* transitions (a mistyped literal is rejected at the key), while any other
* key — an MCP App route, which the registration never lists, or a value typed
* `Record<string, ContractRouteFixture>` built dynamically — stays legal with
* `unknown` inputs, exactly as before. Without a registration every key is a
* string and every input `unknown`.
*/
export type ContractRouteFixtures = {
readonly [Id in RegisteredRouteId]?: ContractRouteFixture<RegisteredRouteInput<Id>>;
} & {
readonly [routeId: string]: ContractRouteFixture;
};

/**
* How MCP App routes are covered at boundaries that register app resources.
*
Expand Down Expand Up @@ -217,7 +247,7 @@ export interface ContractMatrixOptions extends InMemoryMcpSessionOptions {
* the server must be covered. App routes are not registered at
* `mcp-in-memory`; entries for them are accepted and ignored.
*/
readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
readonly fixtures: ContractRouteFixtures;
/** Accepted for parity with the other entry points; apps are never registered here. */
readonly apps?: ContractAppCoverage;
/** Reopens the same durable store after the matrix closes its initial in-memory session. */
Expand Down Expand Up @@ -261,7 +291,7 @@ export interface PackedContractMatrixOptions {
* the server must be covered; app routes are auto-covered unless
* `apps: 'explicit'`.
*/
readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
readonly fixtures: ContractRouteFixtures;
readonly manifest: AgentBundleTestManifest;
readonly server?: string;
/** An already-open packed session; this entry point never opens or closes it. */
Expand All @@ -286,7 +316,7 @@ export interface DevEpochContractMatrixSession {
export interface DevEpochContractMatrixOptions {
/** App route coverage at this boundary; defaults to `'auto'`. */
readonly apps?: ContractAppCoverage;
readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
readonly fixtures: ContractRouteFixtures;
readonly manifest: AgentBundleTestManifest;
readonly server?: string;
/** An already-open epoch-pinned generated stdio session; this entry point never opens or closes it. */
Expand All @@ -296,7 +326,7 @@ export interface DevEpochContractMatrixOptions {
export interface InstalledHostContractMatrixOptions {
/** App route coverage at this boundary; defaults to `'auto'`. */
readonly apps?: ContractAppCoverage;
readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
readonly fixtures: ContractRouteFixtures;
readonly manifest: AgentBundleTestManifest;
readonly server?: string;
/** An already-open installed-host session; this entry point never opens or closes it. */
Expand Down
5 changes: 5 additions & 0 deletions packages/agent-bundle/src/test/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ export type {
ContractCheckStatus,
ContractEventRuntimeAddress,
ContractResourceFixture,
ContractRouteFixtures,
ContractLifecycleFixture,
ContractLifecyclePhase,
ContractLifecycleTransition,
Expand Down Expand Up @@ -147,6 +148,10 @@ export type {
McpProjectionProvenance,
McpPromptResult,
McpResourceRead,
McpRouteInput,
McpRouteNameConstraint,
McpRouteServer,
McpServerConstraint,
McpSurfaceListing,
McpToolInvocation,
} from './mcp.ts';
Expand Down
Loading
Loading