Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/contract-matrix-stage3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"agent-bundle": minor
---

Add the stage-3 installed-host contract matrix boundary (#218). `openInstalledHostMcpServer` verifies and discovers a clean installed host layout, spawns its emitted MCP command over stdio, and observes the live initialize identity. `runInstalledHostContractMatrix` reuses the shared matrix at `host-install` proof level and reports a fail-closed source, built-artifact, installed-artifact, and running-process version quadruple with host binary, adapter, manifest/schema, and framework metadata.
51 changes: 42 additions & 9 deletions packages/agent-bundle/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ is never a receipt for another.
| `cli-dispatch` | `invokeCli`, `cliJson`, `cliNdjson` | a plain or rendered argv vector resolved and run through the routed CLI's own shell, including rendered Markdown, explicit TTY, JSON, and NDJSON modes, in-process |
| `packed-stdio` | `openPackedMcpServer`, `runPackedContractMatrix` | a built artifact's generated entry running as a real process over stdio |
| `packed-deleted-source` | `removeProjectSource`, `openPackedMcpServer({ deletedSource })`, `runPackedContractMatrix` | the packed stdio process still runs after project source and configuration are removed and verified absent |
| `host-install` | repository real-host install proof | a built bundle installed into an isolated real host home through the public install path, with registration observed through the host's own CLI |
| `host-install` | `openInstalledHostMcpServer`, `runInstalledHostContractMatrix` | a built bundle staged into an isolated host root, discovered in the emitted host format, and spawned from the installed layout |

```ts
import { cliJson, cliNdjson, expectEvents, invokeCli, invokeMcpTool } from 'agent-bundle/test';
Expand Down Expand Up @@ -322,14 +322,15 @@ once, build once, remove and verify source once, spawn once, and iterate every
per-route assertion inside that one session. The deleted-source journey also
reads the embedded MCP App resource from the generated server; it does not
prove native-host install or dispatch, or an install mode that copies the
artifact elsewhere. `host-install` is separate real-host process evidence for
built-bundle acceptance and registration, not packed provenance or session
behavior.
artifact elsewhere. `host-install` is separate installed-layout process
evidence: its deterministic adapter-simulator lane is unconditional, available
Claude and Codex binaries also prove their public install paths, and Cursor
records its unavailable non-interactive host-session surface explicitly.

### Contract matrix (`runContractMatrix` / `runPackedContractMatrix`)
### Contract matrix (`runContractMatrix` / `runPackedContractMatrix` / `runInstalledHostContractMatrix`)

The contract matrix is the framework-owned generated-plugin wire-contract suite.
Two entry points share one implementation; boundary differences are explicit
Three entry points share one implementation; boundary differences are explicit
capability flags, not forked check logic. The project supplies only fixtures —
valid inputs, a declared `resultCompat` policy for every in-memory tool route,
optional `previousResults` payloads, optional `cancellation` cases, and an
Expand Down Expand Up @@ -372,16 +373,35 @@ without one the check is honestly `not-applicable`. Packed callers should wire
that callback into the existing packed journey's restart rather than creating
a second pack/build/install path.

**Neither boundary proves:** host install, browser App HTML, artifact-rebuild
replay, or state-lifetime catalog identity.
**`runInstalledHostContractMatrix` (`host-install`)** runs against an
already-open session from `openInstalledHostMcpServer`. The opener reads the
host's emitted MCP document from the installed root, verifies the manifest,
component/resource/hook paths and artifact file digests, spawns that installed
command, and observes the running version from the live MCP `initialize`
result. Its report records source, built-artifact, installed-artifact, and
running-process versions separately and fails closed when any value is missing
or differs. Metadata records the host binary version when observed, adapter
revision, manifest/schema digest, and framework version. Module-backed checks
remain honestly not-applicable because loading project modules would cross back
into the source/build tree.

No matrix boundary proves browser App HTML, artifact-rebuild replay,
state-lifetime catalog identity, or running-process identity beyond what the
live MCP session reports; deeper runtime-instance introspection depends on
#269.

When the advertised input schema declares `additionalProperties: false`, plain
`z.object` tool routes may still strip unknown keys without a protocol failure.
The negative-inputs check records that tolerance when other generated negatives
still prove rejection paths.

```ts
import { runContractMatrix, runPackedContractMatrix } from 'agent-bundle/test';
import {
openInstalledHostMcpServer,
runContractMatrix,
runInstalledHostContractMatrix,
runPackedContractMatrix,
} from 'agent-bundle/test';

await runContractMatrix({
fixtures: {
Expand All @@ -399,6 +419,19 @@ await runPackedContractMatrix({
manifest: compiledManifest,
fixtures: { /* same shape */ },
});

await using installedSession = await openInstalledHostMcpServer({
artifactRoot,
host: 'claude',
installedRoot,
manifest: compiledManifest,
server: 'library',
});
await runInstalledHostContractMatrix({
fixtures: { /* same shape */ },
manifest: compiledManifest,
session: installedSession,
});
```

A failing matrix throws one aggregated `AgentTestError` with code
Expand Down
94 changes: 85 additions & 9 deletions packages/agent-bundle/src/test/contract.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
/**
* The generated-plugin contract matrix — framework-owned wire-contract checks
* at two proof boundaries today (`mcp-in-memory` and packed stdio).
* at three proof boundaries (`mcp-in-memory`, packed stdio, and host install).
*
* Both entry points share one implementation. Boundary differences are explicit
* All three entry points share one implementation. Boundary differences are explicit
* capability flags, not forked check logic. The project supplies only fixtures
* — valid inputs, declared result-compat policy, version-skew payloads,
* optional cancellation cases, and deterministic lifecycle transitions —
Expand All @@ -27,10 +27,16 @@
* every tool result through its bundled `resultSchema` before returning; a
* successful sweep invocation is that evidence.
*
* Stateful lifecycle fixtures replay over one open client at both boundaries.
* Stateful lifecycle fixtures replay over one open client at every boundary.
* Same-store restart callbacks add boundary-local durability evidence; a run
* without one reports restart durability as not-applicable. Neither boundary
* proves host install, browser App HTML, or state-lifetime catalog identity.
* without one reports restart durability as not-applicable.
*
* **The installed-host boundary** discovers and spawns the emitted MCP command
* from a clean installed layout. It carries static layout checks and the
* source/artifact/installed/running version quadruple from `installed.ts`.
*
* No boundary here proves browser App HTML, state-lifetime catalog identity,
* or runtime-instance identity beyond the live MCP initialize result (#269).
*/
import type { Client } from '@modelcontextprotocol/client';

Expand All @@ -46,6 +52,13 @@ import {
type InMemoryMcpSessionOptions,
type McpProjectionProvenance,
} from './mcp.ts';
import type {
InstalledHostCheckOutcome,
InstalledHostEvidenceMetadata,
InstalledHostMcpProvenance,
InstalledHostMcpSession,
InstalledHostVersionQuadruple,
} from './installed.ts';
import type { PackedMcpProvenance, PackedMcpSession } from './packed.ts';
import { registeredRouteLoader, testManifest } from './registry.ts';
import type { AgentRouteModule, TestableRouteDescriptor } from './types.ts';
Expand Down Expand Up @@ -145,7 +158,10 @@ export interface ContractRouteReport {
readonly checks: Readonly<Record<string, ContractCheckOutcome>>;
}

export type ContractMatrixProvenance = McpProjectionProvenance | PackedMcpProvenance;
export type ContractMatrixProvenance =
| InstalledHostMcpProvenance
| McpProjectionProvenance
| PackedMcpProvenance;

export interface ContractMatrixReport {
readonly provenance: ContractMatrixProvenance;
Expand All @@ -162,22 +178,46 @@ export interface PackedContractMatrixOptions {
readonly restart?: () => Promise<ContractMatrixRestartSession>;
}

export interface InstalledHostContractMatrixOptions {
readonly fixtures: Readonly<Record<string, ContractRouteFixture>>;
readonly manifest: AgentBundleTestManifest;
readonly server?: string;
/** An already-open installed-host session; this entry point never opens or closes it. */
readonly session: InstalledHostMcpSession;
}

export interface InstalledHostContractMatrixReport {
readonly checks: Readonly<Record<string, InstalledHostCheckOutcome>>;
readonly host: InstalledHostMcpSession['provenance']['host'];
readonly matrix: ContractMatrixReport;
readonly metadata: InstalledHostEvidenceMetadata;
readonly proofLevel: string;
readonly sessionEvidence: string;
readonly status: 'passed';
readonly versions: InstalledHostVersionQuadruple;
}

interface MatrixBoundaryCapabilities {
readonly canLoadRouteModules: boolean;
readonly moduleSchemaNotApplicableReason: string;
readonly proofLevel: AgentTestProofLevel;
readonly registersAppResources: boolean;
readonly recovery: string;
readonly restart?: () => Promise<ContractMatrixRestartSession>;
}

const PACKED_MODULE_SCHEMA_NOT_APPLICABLE_REASON =
'packed sessions cannot load project route modules (source may be deleted and verified absent); loading a module would silently break deleted-source proof. The packed server validates every tool result through its bundled resultSchema before returning — a successful sweep invocation is that evidence.';

const INSTALLED_HOST_MODULE_SCHEMA_NOT_APPLICABLE_REASON =
'installed-host sessions cannot load project route modules without crossing back into the source/build tree; the installed server validates every tool result through its bundled resultSchema before returning — a successful sweep invocation is that evidence.';

const IN_MEMORY_BOUNDARY: MatrixBoundaryCapabilities = Object.freeze({
canLoadRouteModules: true,
moduleSchemaNotApplicableReason: '',
proofLevel: MCP_IN_MEMORY_PROOF_LEVEL,
registersAppResources: false,
recovery: 'Fix the failing route, fixture, or declared resultCompat policy; re-run runContractMatrix.',
});

const packedBoundaryFromSession = (
Expand All @@ -189,9 +229,18 @@ const packedBoundaryFromSession = (
moduleSchemaNotApplicableReason: PACKED_MODULE_SCHEMA_NOT_APPLICABLE_REASON,
proofLevel: session.provenance.proofLevel,
registersAppResources: true,
recovery: 'Fix the failing route or fixture; re-run runPackedContractMatrix.',
...(restart === undefined ? {} : { restart }),
});

const INSTALLED_HOST_BOUNDARY: MatrixBoundaryCapabilities = Object.freeze({
canLoadRouteModules: false,
moduleSchemaNotApplicableReason: INSTALLED_HOST_MODULE_SCHEMA_NOT_APPLICABLE_REASON,
proofLevel: 'host-install',
registersAppResources: true,
recovery: 'Fix the installed layout, route, or fixture; reinstall and re-run runInstalledHostContractMatrix.',
});

const COMPAT_PROBE_KEY = '__agentBundleContractProbe';

const CHECK_SURFACE = 'surface-completeness';
Expand Down Expand Up @@ -1096,9 +1145,7 @@ const finalizeContractMatrixReport = (
`Contract matrix reported ${String(failures.length)} violation(s) at the ${boundary.proofLevel} proof level.`,
{
details,
recovery: boundary.canLoadRouteModules
? 'Fix the failing route, fixture, or declared resultCompat policy; re-run runContractMatrix.'
: 'Fix the failing route or fixture; re-run runPackedContractMatrix.',
recovery: boundary.recovery,
},
);
};
Expand Down Expand Up @@ -1444,3 +1491,32 @@ export const runPackedContractMatrix = async (
serverName,
});
};

/**
* Runs the shared contract matrix over an already-open MCP process discovered
* and spawned from a host-owned installed layout. The returned report carries
* the separately observed source/artifact/installed/running version evidence.
*/
export const runInstalledHostContractMatrix = async (
options: InstalledHostContractMatrixOptions,
): Promise<InstalledHostContractMatrixReport> => {
const serverName = resolveServerName(options.manifest, options.server);
const matrix = await executeContractMatrix({
boundary: INSTALLED_HOST_BOUNDARY,
client: options.session.client,
fixtures: options.fixtures,
manifest: options.manifest,
provenance: options.session.provenance,
serverName,
});
return Object.freeze({
checks: options.session.observation.checks,
host: options.session.observation.host,
matrix,
metadata: options.session.observation.metadata,
proofLevel: options.session.observation.proofLevel,
sessionEvidence: options.session.observation.sessionEvidence,
status: 'passed',
versions: options.session.observation.versions,
});
};
19 changes: 17 additions & 2 deletions packages/agent-bundle/src/test/index.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/**
* `agent-bundle/test` — the consumer test harness helpers.
*
* Six Node proof levels ship here, and the browser-safe seventh level ships
* Seven Node proof levels ship here, and the browser-safe eighth level ships
* from `agent-bundle/test/browser`. The repository's real-host install proof
* uses the same level convention. Each helper names the level it supplies,
* stamps it into its provenance, and prints it in every failure:
Expand All @@ -14,7 +14,7 @@
* | `packed-stdio` | `openPackedMcpServer`, `runPackedContractMatrix` | a built artifact's generated entry running as a real process over stdio |
* | `packed-deleted-source` | `removeProjectSource`, `openPackedMcpServer({ deletedSource })`, `runPackedContractMatrix` | the packed stdio process still runs after project source and configuration are removed and verified absent |
* | `browser-app` | `mountBrowserApp` (`agent-bundle/test/browser`) | production-compiled MCP App HTML mounted over the product bridge in a real browser page |
* | `host-install` | repository real-host install proof | a built bundle accepted through a real host's public install path in an isolated home, with registration observed by that host |
* | `host-install` | `openInstalledHostMcpServer`, `runInstalledHostContractMatrix` | a built bundle staged into an isolated host root, discovered in the host's emitted format, and spawned from the installed layout |
*
* A pass at one level is never a receipt for another. The `deletedSource`
* option upgrades `openPackedMcpServer` provenance only after every path in a
Expand Down Expand Up @@ -84,6 +84,7 @@ export {
export {
negativeInputsFromJsonSchema,
runContractMatrix,
runInstalledHostContractMatrix,
runPackedContractMatrix,
} from './contract.ts';
export type {
Expand All @@ -98,6 +99,8 @@ export type {
ContractMatrixRestartSession,
ContractRouteFixture,
ContractRouteReport,
InstalledHostContractMatrixOptions,
InstalledHostContractMatrixReport,
PackedContractMatrixOptions,
ResultCompatPolicy,
} from './contract.ts';
Expand All @@ -121,6 +124,18 @@ export type {
PackedMcpSession,
PackedMcpSessionOptions,
} from './packed.ts';
export { openInstalledHostMcpServer } from './installed.ts';
export type {
InstalledHostBinaryVersion,
InstalledHostCheckName,
InstalledHostCheckOutcome,
InstalledHostEvidenceMetadata,
InstalledHostMcpProvenance,
InstalledHostMcpSession,
InstalledHostObservation,
InstalledHostVersionQuadruple,
OpenInstalledHostMcpServerOptions,
} from './installed.ts';
export type {
AgentRouteModule,
AgentRouteModuleLoader,
Expand Down
Loading
Loading