Generated plugin contract matrix: catch source→artifact→dev→installed-host drift before release

[ScriptedAlchemy]

Context

This is a concrete acceptance-test child of #103 (generated consumer test harness), #133 (emitted-artifact conformance), #105 (Workbench lifecycle replay), #179 (stable dev-host proxy/HMR), and #181 (framework-owned host installs).

Developing https://github.com/ScriptedAlchemy/cargo-conductor exposed failures that all passed project unit tests and agent-bundle build, but broke only in the live plugin lifecycle:

1. MCP result-schema drift: the daemon added execArgv, then errorCount / warningCount / diagnostics; generated/runtime response validation rejected otherwise-valid conductor_status and conductor_result payloads with unrecognized_keys. Agents institutionalized a CLI+jq workaround and reran cargo commands because they stopped trusting MCP.
2. Status/result lifecycle gap: status rows intentionally omitted output tails, while result fetching skipped non-terminal rows. Running tickets therefore showed “no output until the run finishes” despite the daemon already streaming output. Source-level tests did not exercise an open MCP/UI consumer across queued→running→output→terminal transitions.
3. Source/artifact/install drift: rebuilding source did not prove that the installed Cursor plugin, spawned stdio MCP process, dashboard resource, or daemon was the new version. The Workbench tested its own MCP control surface, not the developed plugin surface; stale processes/artifacts looked healthy.
4. Target layout drift: Cursor local-plugin layout needed a hand-maintained install script and previously emitted artifacts did not match host discovery semantics (Cursor target artifacts don't match how Cursor actually loads local plugins (manifest location, hooks doc collision, root tokens) #126). Build success was not install success.
5. Forward/backward compatibility: a new daemon with an older widget/client, and a new widget/client with an older daemon, were not tested as an explicit matrix.

Requested generated contract matrix

agent-bundle test (or the #103 Rstest integration) should generate and run the same contract at five boundaries:

1. Source route

• invoke every operation with generated valid inputs;
• validate the operation result against its declared result schema;
• generate negative inputs from schema boundaries;
• assert cancellation/timeout behavior.

2. Packed artifact

• build once, delete or make source unavailable, spawn the exact generated MCP executable from the artifact;
• initialize over the real transport, list tools/resources, call every operation, fetch every bound App resource;
• assert stdout protocol purity and clean shutdown.

3. Stateful lifecycle replay

For each operation with a stateful fixture, keep one client connection open and replay:

unknown → queued → running → first progress/output → repeated progress → terminal

At every phase:

• validate structuredContent and rendered/Agent Document output;
• validate additive fields are accepted according to the declared compatibility policy;
• assert live progress is visible before terminal settlement;
• assert reconnect/restart retains durable state where declared.

The fixture API should let projects supply a deterministic transition driver; the framework owns transport, schema, render, and lifecycle assertions.

4. Version-skew matrix

Run at least:

• current server + previous generated client/widget contract;
• previous server fixture + current generated client/widget;
• unknown additive response fields;
• absent new optional fields;
• every declared status/discriminant value.

Schema strictness should be explicit per boundary: user inputs may be closed; daemon/server outputs intended for rolling upgrades must prove their forward-compat policy. Generated compile-time satisfies checks are not enough — the exact serialized payload must round-trip through the exact runtime validator.

5. Installed-host smoke / doctor

Using the same framework-generated install plan from #181:

• stage into a clean temporary host root;
• run host discovery exactly as the adapter does;
• verify manifests, component paths, MCP command, resources, hook commands, and version digests;
• spawn the configured MCP command and run the lifecycle replay;
• report source/artifact/installed/running versions separately so stale-but-healthy processes cannot pass.

Where a real host CLI is available, add an opt-in native lane; otherwise the adapter simulator is the required deterministic baseline.

Dev-server requirement

The Workbench/dev server should run this matrix against the developed plugin’s routes and resources, not only its own control tools. A rebuild should automatically rerun affected contracts and preserve last-good behavior:

• source edit → new epoch;
• passing epoch swaps in without dropping the stable host connection (Feature: dev-server HMR into agent hosts — hot-swap plugin MCP/hooks behind a stable host connection (webpack-HMR analog) #179);
• failing epoch stays inactive and surfaces contract diagnostics in the Workbench;
• open App/MCP lifecycle replay continues or clearly resets according to policy.

Acceptance proof using cargo-conductor

A fixture plugin should include:

• an operation whose response adds optional execArgv, diagnostics, and live-tail fields over successive fixture versions;
• an App that opens a running ticket and must render output before terminal completion;
• a durable record that survives a server restart;
• Cursor/Claude/Codex target artifacts staged into clean roots.

The generated matrix must fail if:

• any additive output field triggers unrecognized_keys;
• a running App says output is unavailable when the server emitted progress;
• source, built artifact, installed artifact, and running process versions disagree;
• a host adapter cannot discover or spawn its emitted target layout.

Why this belongs in Agent Bundle

Projects can unit-test their business logic, but only Agent Bundle owns all five transformations: route contract → generated runtime → packed artifact → host projection/install → live protocol/render lifecycle. Requiring each plugin to rebuild this harness guarantees inconsistent coverage and repeats the exact drift the meta-framework is meant to eliminate.

[ScriptedAlchemy]

Scope split vs #133, and staging

Boundary rule between the two issues: a check that can pass on bytes at rest belongs to #133; a check that requires observing a live protocol exchange or a spawned process belongs here.

#133 keeps (static, emitted-artifact conformance):

• pinned ecosystem schemas over rendered manifests/configs (Cursor plugin + marketplace, portable, Claude plugin.json/hooks.json, Codex openai.yaml) with negative fixtures;
• Agent Skills frontmatter lint, layout resolution/containment, root-token rules;
• the @modelcontextprotocol/conformance CI lane;
• doctor's static findings (discovery path, manifest parse, digests, command existence) — stage 1 already landed via feat(doctor): read-only agent-bundle doctor for host installs and runtime endpoints (#101 stage 1) #236/fix(doctor): match the pinned Claude registration id contract and accept versionless Cursor manifests #243, Claude plugin validate via feat(claude): validate artifacts with host devtools #193.

#218 owns (runtime contract matrix — nothing here can be proven on bytes at rest):

• every operation's serialized result round-tripped through the exact runtime resultSchema validator (the generated server validates results via resultSchema.parse; compile-time satisfies is explicitly not enough);
• declared per-boundary compatibility policy proofs: unknown additive response fields, absent new optional fields, every declared discriminant — the version-skew matrix;
• stateful lifecycle replay over one open client connection (queued→running→progress→terminal), live-progress-before-terminal, reconnect/restart durability;
• the source/artifact/installed/running version quadruple so stale-but-healthy processes cannot pass — doctor is the shared reporting surface: Adopt pinned schemas and ecosystem validators for generated plugin conformance (build-time + doctor) #133 feeds it static findings, Generated plugin contract matrix: catch source→artifact→dev→installed-host drift before release #218 adds the runtime/running-process lane.

Staging (#218 stages; this lands stage 1)

1. Stage 1 (now): the contract-matrix core at the source and packed boundaries — a generated per-operation sweep in agent-bundle/test (every compiled tool/resource/prompt invoked, serialized result round-tripped through the exact runtime validator), the additive-field/skew probes against a declared compat policy, negative inputs from schema boundaries, and cancellation behavior. The packed boundary rides the single Generate an Agent Bundle framework test harness on top of Rstest #103 packed journey (cost rule) — no second pack/build.
2. Stage 2: stateful lifecycle replay with the project-supplied deterministic transition driver, live-progress and restart-durability assertions (on the Track remaining #98 state-kernel scope: warm-runtime process state, generated-runtime state mounting, budgets, lifetime visibility #233 state kernel).
3. Stage 3: installed-host smoke + version quadruple, building on feat(test): host-install proof level with real Claude, Codex, and Cursor install proofs (#100 stage 3) #242 (host-install proof) and feat(doctor): read-only agent-bundle doctor for host installs and runtime endpoints (#101 stage 1) #236 (doctor).
4. Stage 4: Workbench dev-epoch integration (Feature: dev-server HMR into agent hosts — hot-swap plugin MCP/hooks behind a stable host connection (webpack-HMR analog) #179): rebuild reruns affected contracts, failing epoch stays inactive.

Proof-level labels stay honest per the #103/#190 discipline: a matrix pass at mcp-in-memory is never claimed as process, packed, or host evidence.

[ScriptedAlchemy]

Stage 1 landed — #272 (d062130fad53d34b21ccc43f0e1293a91cb74162)

Per the scope split above (runtime half here, bytes-at-rest conformance in #133), stage 1 ships the contract-matrix core at the two boundaries whose substrate landed this week: runContractMatrix at mcp-in-memory and runPackedContractMatrix against an already-open packed session (packed-stdio / packed-deleted-source), one shared implementation with explicit boundary capability flags and honest per-check passed / failed / not-applicable outcomes. A failing matrix throws one aggregated contract-violation error naming every route, check, and the proof-level label actually carried.

Coverage at each boundary

check	mcp-in-memory	packed stdio
surface completeness vs compiled manifest	✅ (Apps not-applicable: not registered at this level)	✅ incl. MCP App resource URIs
fixture coverage of every compiled route	✅	✅
invocation sweep (tools per fixture input, resources, prompts)	✅	✅ incl. App resource reads
serialized round-trip through the route's own resultSchema (explicit JSON round-trip closes the in-memory-transport gap)	✅	n/a — packed sessions cannot load project route modules (source may be deleted+verified absent); the packed server's bundled resultSchema validation is the evidence
declared resultCompat probe — unknown-key injection into the serialized payload; the unrecognized_keys drift from this issue's failure mode 1 is a hard failure under additive	✅ (required per tool route)	n/a (same reason)
version-skew previousResults (previous-server payloads incl. absent new optional fields must parse under the current schema)	✅	n/a (same reason)
negative inputs generated from the advertised listTools JSON Schema	✅	✅
mid-flight cancellation (bounded settle + session stays usable)	✅	✅

The route-harness fixture gained the acceptance-shaped tools: ticket (optional execArgv/diagnostics/tail over a loose schema — the cargo-conductor shape), strict-report (z.strictObject, closed policy), and wait (signal-aware, for deterministic cancellation). The packed matrix rides the single #103 packed journey — no new pack, install, build, or spawn; the durable-restart assertions are unchanged.

Gates (local, at the merged head's branch state)

pnpm build · pnpm typecheck · pnpm lint (0/0) · pnpm test:unit · pnpm test:projection · pnpm test:route-unit · packed single journey — all green. public-api-packed.test.ts fails identically on a clean origin/main checkout (ERR_MODULE_NOT_FOUND: @agent-bundle/runtime in the packed consumer) — pre-existing on main, not introduced by this change; worth its own issue.

Remaining stages (this issue stays open)

2. Stateful lifecycle replay (unknown → queued → running → progress → terminal over one open connection, live-progress-before-terminal, restart durability on the Track remaining #98 state-kernel scope: warm-runtime process state, generated-runtime state mounting, budgets, lifetime visibility #233 state kernel) with the project-supplied deterministic transition driver.
3. Installed-host smoke + the source/artifact/installed/running version quadruple, on feat(test): host-install proof level with real Claude, Codex, and Cursor install proofs (#100 stage 3) #242 (host-install) and feat(doctor): read-only agent-bundle doctor for host installs and runtime endpoints (#101 stage 1) #236 (doctor).
4. Workbench dev-epoch integration (Feature: dev-server HMR into agent hosts — hot-swap plugin MCP/hooks behind a stable host connection (webpack-HMR analog) #179).