fix(test): cover MCP App routes in the packed contract matrix without hand-enumerated fixtures (#401)

.changeset/401-contract-matrix-app-coverage.md

@@ -0,0 +1,5 @@
+---
+"agent-bundle": patch
+---
+
+Stop requiring hand-enumerated MCP App fixtures in `runPackedContractMatrix`, `runInstalledHostContractMatrix`, and `runDevEpochContractMatrix`: app routes are auto-covered at those levels with the new `apps: 'auto'` default (`coverage` passes with a reason naming the `ui://` resource sweep; `apps: 'explicit'` restores the fixture requirement), declare a resource or app fixture as `{ kind: 'resource' }` (`ContractResourceFixture`; legacy `{}` still accepted), keep apps `not-applicable` at `mcp-in-memory`, and report the `cancellation` check as `not-applicable` ("invocation completed before abort; use an input that stays in flight") instead of a `contract-violation` when the aborted call settled before `abortAfterMs` elapsed. The `agent-bundle/test` docs now state per level whether apps are covered. No diagnostic codes change. (#417)

packages/agent-bundle/README.md

@@ -487,8 +487,9 @@ the current schema, rejection of negative inputs derived from the advertised
 `listTools` input JSON Schema, and mid-flight cancellation hygiene. In-memory
 transport may pass structured values without serialization; the matrix closes
 that gap with an explicit `JSON.parse(JSON.stringify(...))` round-trip before
-validation. MCP Apps are reported as not-applicable for surface registration
-because the in-memory level does not register them.
+validation. MCP Apps are not registered at this level: every app route reports
+`surface-completeness` as `not-applicable` and receives no coverage or sweep
+check, and app fixture entries are accepted and ignored.
 
 **`runPackedContractMatrix` (`packed-stdio` / `packed-deleted-source`)** runs
 against an already-open packed session (the single packed journey owns session
@@ -502,6 +503,24 @@ version-skew (including their per-lifecycle-phase variants) are reported
 server validates every tool result through its bundled `resultSchema` before
 returning; a successful sweep invocation is that evidence.
 
+**MCP App coverage per level.** Fixtures must cover every compiled tool, prompt,
+and resource route on the server. App routes are covered at every boundary that
+registers app resources — `packed-stdio`, `packed-deleted-source`,
+`host-install`, and `dev-epoch` — where `surface-completeness` requires the
+compiled `ui://` URI in `listResources` and `sweep` reads that resource. With
+the default `apps: 'auto'` an app route needs no fixture entry: `coverage`
+passes with a reason naming the auto-covered sweep. An explicit
+`{ kind: 'resource' }` (or legacy `{}`) entry is always accepted, and
+`apps: 'explicit'` makes a missing app entry a `coverage` failure again. At
+`mcp-in-memory` apps are never registered, so `apps` has no effect there.
+
+The `cancellation` fixture aborts the invocation after `abortAfterMs`
+(default 50ms) and requires it to settle rejected. Its input must stay in
+flight past that point: when the invocation settles before the abort fires
+the check is `not-applicable` ("invocation completed before abort; use an
+input that stays in flight"), and it only `fails` when the abort was delivered
+mid-flight and the call still settled without rejecting.
+
 Lifecycle fixtures replay
 `unknown → queued → running → first-progress → repeated-progress → terminal`
 over the matrix's one open client. The framework validates every phase's
@@ -560,11 +579,12 @@ await runContractMatrix({
 });
 
 // Packed: pass an already-open session and a manifest compiled before source removal.
+// App routes are auto-covered here; `{ kind: 'resource' }` names one explicitly.
 await runPackedContractMatrix({
   eventRuntime: { endpointId: packedEventRuntimeEndpointId },
   session: packedSession,
   manifest: compiledManifest,
-  fixtures: { /* same shape */ },
+  fixtures: { /* same shape, optionally 'app:library/dashboard': { kind: 'resource' } */ },
 });
 
 await using installedSession = await openInstalledHostMcpServer({

packages/agent-bundle/src/config/dev-contracts.ts

@@ -131,9 +131,12 @@ const validateFixture = (value: unknown, routeId: string): ContractRouteFixture
   if (!isRecord(value)) return invalid(`Fixture ${JSON.stringify(routeId)} must be an object.`);
   fields(
     value,
-    ['cancellation', 'input', 'inputs', 'lifecycle', 'previousResults', 'resultCompat'],
+    ['cancellation', 'input', 'inputs', 'kind', 'lifecycle', 'previousResults', 'resultCompat'],
     `Fixture ${JSON.stringify(routeId)}`,
   );
+  if (value.kind !== undefined && value.kind !== 'resource') {
+    return invalid(`Fixture ${JSON.stringify(routeId)} kind must be "resource" when provided.`);
+  }
   if (value.resultCompat !== undefined && value.resultCompat !== 'additive' && value.resultCompat !== 'closed') {
     return invalid(`Fixture ${JSON.stringify(routeId)} resultCompat must be "additive" or "closed".`);
   }

packages/agent-bundle/src/test/contract.ts

@@ -15,7 +15,8 @@
  * behavior, and previous-server payload acceptance. In-memory transport may
  * pass structured values without serialization; the explicit
  * `JSON.parse(JSON.stringify(...))` round-trip closes that gap. MCP Apps are
- * not registered at this level.
+ * not registered at this level: every app route reports `surface-completeness`
+ * as `not-applicable` and receives no coverage, sweep, or other check.
  *
  * **`runPackedContractMatrix` (`packed-stdio` / `packed-deleted-source`)** runs
  * against an already-open packed session. It proves process stdio evidence for
@@ -27,6 +28,15 @@
  * every tool result through its bundled `resultSchema` before returning; a
  * successful sweep invocation is that evidence.
  *
+ * **MCP App coverage.** At every boundary that registers app resources
+ * (`packed-stdio`, `packed-deleted-source`, `host-install`, `dev-epoch`) app
+ * routes ARE part of the matrix: `surface-completeness` requires the compiled
+ * `ui://` URI in `listResources`, and `sweep` reads that resource. With the
+ * default `apps: 'auto'` an app route needs no fixture entry — `coverage`
+ * passes with a reason naming the auto-covered sweep. An explicit
+ * `{ kind: 'resource' }` (or legacy `{}`) fixture is always accepted;
+ * `apps: 'explicit'` restores the requirement that every app route be listed.
+ *
  * Stateful lifecycle fixtures replay over one open client at every boundary.
  * Same-store restart callbacks add boundary-local durability evidence; a run
  * without one reports restart durability as not-applicable.
@@ -125,7 +135,22 @@ export interface ContractLifecycleFixture {
   readonly transitionDriver: () => readonly ContractLifecycleTransition[];
 }
 
+/**
+ * The explicit fixture form for a resource or MCP App route: the sweep reads
+ * the route's wire URI and no input, policy, or lifecycle applies. Declaring
+ * it on a tool or prompt route is a coverage failure.
+ */
+export interface ContractResourceFixture {
+  readonly kind: 'resource';
+}
+
 export interface ContractRouteFixture {
+  /**
+   * `'resource'` marks a resource/MCP App fixture (see `ContractResourceFixture`).
+   * Omit it for tool and prompt fixtures; a legacy `{}` still covers a
+   * resource or app route.
+   */
+  readonly kind?: ContractResourceFixture['kind'];
   /** Valid input for the sweep invocation (tools/prompts; resources need none). */
   readonly input?: unknown;
   /** Additional valid inputs — e.g. one per declared status/discriminant value. */
@@ -139,13 +164,28 @@ export interface ContractRouteFixture {
   readonly previousResults?: readonly unknown[];
   /**
    * Cancellation case: invocation aborted mid-flight must settle rejected and
-   * leave the session usable.
+   * leave the session usable. The input must stay in flight past
+   * `abortAfterMs` (default 50ms); an invocation that settles before the abort
+   * fires is reported `not-applicable`, not `failed`.
    */
   readonly cancellation?: { readonly abortAfterMs?: number; readonly input?: unknown };
   /** Optional stateful replay over this matrix run's single open client. */
   readonly lifecycle?: ContractLifecycleFixture;
 }
 
+/**
+ * How MCP App routes are covered at boundaries that register app resources.
+ *
+ * - `'auto'` (default): app routes need no fixture entry; `coverage` passes
+ *   and the sweep reads the compiled `ui://` resource. Explicit entries are
+ *   still accepted.
+ * - `'explicit'`: every compiled app route must have a fixture entry
+ *   (`{ kind: 'resource' }`), matching the rule for every other route kind.
+ *
+ * Has no effect at `mcp-in-memory`, where apps are never registered.
+ */
+export type ContractAppCoverage = 'auto' | 'explicit';
+
 export type ContractMatrixClient = Pick<
   Client,
   'callTool' | 'getPrompt' | 'listPrompts' | 'listResources' | 'listTools' | 'readResource'
@@ -158,8 +198,14 @@ export interface ContractMatrixRestartSession {
 export interface ContractMatrixOptions extends InMemoryMcpSessionOptions {
   readonly manifest?: AgentBundleTestManifest;
   readonly server?: string;
-  /** Route id -> fixture. Every compiled non-app route on the server must be covered. */
+  /**
+   * Route id -> fixture. Every compiled tool, prompt, and resource route on
+   * the server must be covered. App routes are not registered at
+   * `mcp-in-memory`; entries for them are accepted and ignored.
+   */
   readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
+  /** Accepted for parity with the other entry points; apps are never registered here. */
+  readonly apps?: ContractAppCoverage;
   /** Reopens the same durable store after the matrix closes its initial in-memory session. */
   readonly restart?: () => Promise<ContractMatrixRestartSession>;
 }
@@ -192,8 +238,15 @@ export type ContractEventRuntimeAddress =
   | { readonly endpoint?: never; readonly endpointId: string };
 
 export interface PackedContractMatrixOptions {
+  /** App route coverage at this boundary; defaults to `'auto'`. */
+  readonly apps?: ContractAppCoverage;
   /** Read-only status address for the generated event runtime, when this artifact has one. */
   readonly eventRuntime?: ContractEventRuntimeAddress;
+  /**
+   * Route id -> fixture. Every compiled tool, prompt, and resource route on
+   * the server must be covered; app routes are auto-covered unless
+   * `apps: 'explicit'`.
+   */
   readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
   readonly manifest: AgentBundleTestManifest;
   readonly server?: string;
@@ -217,6 +270,8 @@ export interface DevEpochContractMatrixSession {
 }
 
 export interface DevEpochContractMatrixOptions {
+  /** App route coverage at this boundary; defaults to `'auto'`. */
+  readonly apps?: ContractAppCoverage;
   readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
   readonly manifest: AgentBundleTestManifest;
   readonly server?: string;
@@ -225,6 +280,8 @@ export interface DevEpochContractMatrixOptions {
 }
 
 export interface InstalledHostContractMatrixOptions {
+  /** App route coverage at this boundary; defaults to `'auto'`. */
+  readonly apps?: ContractAppCoverage;
   readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
   readonly manifest: AgentBundleTestManifest;
   readonly server?: string;
@@ -861,7 +918,12 @@ const runSweep = async (
       if (uri === undefined) {
         return failed(`${descriptor.kind} route config exports no uri to read`);
       }
-      const read = await client.readResource({ uri }) as { contents?: unknown };
+      let read: { contents?: unknown };
+      try {
+        read = await client.readResource({ uri }) as { contents?: unknown };
+      } catch (error) {
+        return failed(`readResource threw for ${JSON.stringify(uri)}: ${error instanceof Error ? error.message : captured(error)}`);
+      }
       const contents = Array.isArray(read.contents) ? read.contents : [];
       return contents.length === 0
         ? failed(`readResource returned no contents for ${JSON.stringify(uri)}`)
@@ -1027,9 +1089,16 @@ const runCancellation = async (
     signal: controller.signal,
     timeout: settleWithinMs,
   });
-  const timer = setTimeout(() => controller.abort(), abortAfterMs);
+  // The timer and the settlement race; `abortFired` is read the moment the
+  // call settles so the verdict reflects whether the abort was ever delivered
+  // while the invocation was still in flight.
+  let abortFired = false;
+  const timer = setTimeout(() => {
+    abortFired = true;
+    controller.abort();
+  }, abortAfterMs);
   const result = await Promise.race([
-    call.then((settled) => ({ kind: 'settled' as const, settled })),
+    call.then((settled) => ({ abortedInFlight: abortFired, kind: 'settled' as const, settled })),
     new Promise<{ kind: 'timeout' }>((resolve) => {
       setTimeout(() => resolve({ kind: 'timeout' }), settleWithinMs);
     }),
@@ -1038,6 +1107,13 @@ const runCancellation = async (
   if (result.kind === 'timeout') {
     return failed(`callTool did not settle within ${String(settleWithinMs)}ms after abort`);
   }
+  if (!result.abortedInFlight) {
+    return notApplicable(
+      result.settled.threw
+        ? `invocation rejected before abort (${result.settled.error instanceof Error ? result.settled.error.message : captured(result.settled.error)}); use an input that stays in flight past ${String(abortAfterMs)}ms.`
+        : `invocation completed before abort; use an input that stays in flight past ${String(abortAfterMs)}ms.`,
+    );
+  }
   if (!result.settled.threw) {
     return failed('aborted callTool settled without throwing or rejecting.');
   }
@@ -1387,15 +1463,58 @@ const finalizeContractMatrixReport = (
   );
 };
 
+const AUTO_APP_FIXTURE: ContractResourceFixture = Object.freeze({ kind: 'resource' });
+
+const APP_AUTO_COVERAGE_REASON =
+  'app route auto-covered (apps: "auto"): the sweep reads its compiled MCP App resource URI.';
+
+interface ResolvedRouteFixture {
+  readonly coverage: ContractCheckOutcome;
+  readonly fixture: ContractRouteFixture | undefined;
+}
+
+const resolveRouteFixture = (
+  descriptor: TestableRouteDescriptor,
+  fixture: ContractRouteFixture | undefined,
+  apps: ContractAppCoverage,
+): ResolvedRouteFixture => {
+  if (fixture === undefined) {
+    if (descriptor.kind !== 'app') {
+      return { coverage: failed('compiled route has no fixture entry'), fixture };
+    }
+    switch (apps) {
+      case 'auto':
+        return { coverage: passedWithReason(APP_AUTO_COVERAGE_REASON), fixture: AUTO_APP_FIXTURE };
+      case 'explicit':
+        return {
+          coverage: failed('compiled app route has no fixture entry (apps: "explicit"); add { kind: "resource" } or use apps: "auto"'),
+          fixture,
+        };
+      default: {
+        const exhaustive: never = apps;
+        return { coverage: failed(`unsupported apps coverage mode ${String(exhaustive)}`), fixture: undefined };
+      }
+    }
+  }
+  if (fixture.kind === 'resource' && descriptor.kind !== 'resource' && descriptor.kind !== 'app') {
+    return {
+      coverage: failed(`fixture kind "resource" declared for a ${descriptor.kind} route; resource fixtures apply to resource and app routes only`),
+      fixture: undefined,
+    };
+  }
+  return { coverage: passed(), fixture };
+};
+
 const executeContractMatrix = async (options: {
+  readonly apps: ContractAppCoverage;
   readonly boundary: MatrixBoundaryCapabilities;
   readonly client: ContractMatrixClient;
   readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
   readonly manifest: AgentBundleTestManifest;
   readonly provenance: ContractMatrixProvenance;
   readonly serverName: string;
 }): Promise<ContractMatrixReport> => {
-  const { boundary, client, fixtures, manifest, provenance, serverName } = options;
+  const { apps, boundary, client, fixtures, manifest, provenance, serverName } = options;
   const failures: ContractMatrixFailure[] = [];
   const matrixChecks: Record<string, ContractCheckOutcome> = {};
   const routeReports: Record<string, ContractRouteReport> = {};
@@ -1443,15 +1562,13 @@ const executeContractMatrix = async (options: {
     }
 
     const checks: Record<string, ContractCheckOutcome> = {};
-    const fixture = fixtures[descriptor.id];
+    const { coverage, fixture } = resolveRouteFixture(descriptor, fixtures[descriptor.id], apps);
 
     checks[CHECK_COVERAGE] = outcomeFromCheck(
       failures,
       descriptor.id,
       CHECK_COVERAGE,
-      fixture === undefined
-        ? failed('compiled route has no fixture entry')
-        : passed(),
+      coverage,
     );
 
     checks[CHECK_SURFACE] = outcomeFromCheck(
@@ -1731,6 +1848,7 @@ export const runContractMatrix = async (
         },
       });
     return await executeContractMatrix({
+      apps: options.apps ?? 'auto',
       boundary,
       client: session.client,
       fixtures: options.fixtures,
@@ -1753,6 +1871,7 @@ export const runDevEpochContractMatrix = async (
 ): Promise<ContractMatrixReport> => {
   const serverName = resolveServerName(options.manifest, options.server);
   return executeContractMatrix({
+    apps: options.apps ?? 'auto',
     boundary: DEV_EPOCH_BOUNDARY,
     client: options.session.client,
     fixtures: options.fixtures,
@@ -1765,12 +1884,15 @@ export const runDevEpochContractMatrix = async (
 /**
  * Runs the contract matrix against an already-open packed stdio session.
  * Never opens or closes the session; stamps the session's own proof level.
+ * Compiled MCP App routes are covered here (surface + `ui://` sweep) and
+ * auto-covered without a fixture entry unless `apps: 'explicit'`.
  */
 export const runPackedContractMatrix = async (
   options: PackedContractMatrixOptions,
 ): Promise<ContractMatrixReport> => {
   const serverName = resolveServerName(options.manifest, options.server);
   return executeContractMatrix({
+    apps: options.apps ?? 'auto',
     boundary: packedBoundaryFromSession(
       options.session,
       options.eventRuntime,
@@ -1794,6 +1916,7 @@ export const runInstalledHostContractMatrix = async (
 ): Promise<InstalledHostContractMatrixReport> => {
   const serverName = resolveServerName(options.manifest, options.server);
   const matrix = await executeContractMatrix({
+    apps: options.apps ?? 'auto',
     boundary: installedHostBoundaryFromSession(options.session),
     client: options.session.client,
     fixtures: options.fixtures,