Context
PR #1404 and its review fixes exposed a recurring class of false-green paths around agent-facing help:
- The help-conformance benchmark validates positional arity, while the production CLI can still silently discard extra positionals.
- A benchmark scorer can exist and pass without proving that it rejects the bad plan that motivated it.
- Runner/API failures are reconstructed from raw strings and can be reported alongside model-validation failures, making infrastructure instability look like model quality.
- Changes to the help-conformance harness fall through to an overly broad affected-check set, including paid/stochastic SkillGym work.
- The main harnesses have grown beyond the repository's context-safety guidance, but that guidance is not ratcheted.
These are different symptoms of the same gap: important claims are benchmark-local conventions instead of fail-closed contracts at the boundary that owns them.
Concrete production examples today include:
snapshot ignored.json # extra positional is ignored
close first second # `second` is ignored
open app url close # `close` is swallowed instead of being a lifecycle command
Goal
Make it difficult to ship another help/benchmark false green:
- Production and benchmark parsing share one positional contract.
- Every important oracle demonstrates that it can fail.
- Infrastructure failures cannot enter model scoring.
- Deterministic help-conformance checks have explicit affected-check ownership.
- Guidance harnesses remain cheap enough for an agent to read and change safely.
Proposed work
1. Enforce positional bounds at the production CLI boundary
- Add one shared positional-arity assertion derived from
CommandSchema.
- Apply it to normal CLI parsing and legacy batch positionals.
- Require an explicit schema trait for any command that intentionally accepts additional positionals.
- Add an exhaustive test that supplies
max + 1 positionals to every bounded command and expects rejection.
- Remove the benchmark's private arity implementation once the production contract is reusable.
- Update CLI help/docs and add a behavioral SkillGym case if rejecting ignored input changes the user-visible contract.
Relevant seams:
src/cli/parser/args.ts
src/commands/cli-grammar/registry.ts
- command readers such as
src/commands/management/app.ts
scripts/help-conformance-command-validator.ts
2. Require falsification fixtures for benchmark oracles
Extend case/expectation definitions so each important oracle carries:
- a minimal passing witness;
- at least one known-bad counterexample;
- where useful, a metamorphic variant that changes irrelevant nouns/values.
Add a completeness test that rejects a named expectation without falsification fixtures. Include regressions for swallowed lifecycle commands, unsupported flags/selectors, pseudo refs, shell operators, and invalid positional ordering.
3. Make runner outcomes typed and evidence honest
Replace raw-string success inference with a discriminated outcome:
type RunnerOutcome =
| { kind: 'success'; raw: string; commands: string[] }
| { kind: 'runner-error'; raw: string; message: string; reason: RunnerErrorReason };
- Only
success outcomes may reach command validation and scoring.
- A result must not contain both
runnerError and model-validation failures.
- All-error aggregates render
N/A, not 0/0 (0%).
- Add an
--evidence mode that requires at least three evaluated trials and emits paste-ready Markdown containing commit SHA, runner/model, evaluated trials, runner errors, pass rates, and help-document hashes.
- Keep single-trial mode for exploration, but do not label it stability evidence.
4. Give help-conformance tooling explicit affected-check ownership
- Add a deterministic
help-conformance check to check:affected.
- Give it explicit ownership of
scripts/help-conformance-* and focused tests.
- Treat
test/skillgym/README.md as documentation rather than model input.
- Keep stochastic/paid Claude and Codex calls non-gating; gate schemas, counterexamples, adapters, and reporting deterministically.
5. Split oversized harnesses and add a merge-base ratchet
Current sizes:
scripts/help-conformance-bench.mjs: 846 lines
test/skillgym/suites/agent-device-smoke-suite.ts: 2,807 lines
scripts/__tests__/help-conformance-bench.test.ts: 449 lines
Split by question: case definitions, runners, reporting, scoring, shared SkillGym harness, fixture cases, and guidance cases. Then add a merge-base-aware size guard:
- New implementation files cannot exceed 500 lines.
- Existing implementation files over 500 lines cannot grow.
- Files over 1,000 lines require a reviewed waiver or must shrink.
- Generated fixtures/data receive documented exemptions.
- Source/test topology stays aligned when modules split.
The ratchet should report the file, baseline/current size, applicable rule, and recovery action. It should not require a repository-wide cleanup before landing.
Suggested delivery order
- Production positional contract.
- Harness split.
- Typed outcomes, falsification fixtures, and evidence mode.
- Affected-check ownership and module-size ratchet.
Separate PRs are expected; later work should build on the production contract rather than duplicating it.
Acceptance criteria
Non-goals
- Making stochastic model calls a required PR gate.
- Treating one model trial as stability evidence.
- Duplicating command grammar or help contracts inside the benchmark.
- A blanket line-count gate without grandfathering and explicit generated-data exemptions.
Related: #1404
Context
PR #1404 and its review fixes exposed a recurring class of false-green paths around agent-facing help:
These are different symptoms of the same gap: important claims are benchmark-local conventions instead of fail-closed contracts at the boundary that owns them.
Concrete production examples today include:
Goal
Make it difficult to ship another help/benchmark false green:
Proposed work
1. Enforce positional bounds at the production CLI boundary
CommandSchema.max + 1positionals to every bounded command and expects rejection.Relevant seams:
src/cli/parser/args.tssrc/commands/cli-grammar/registry.tssrc/commands/management/app.tsscripts/help-conformance-command-validator.ts2. Require falsification fixtures for benchmark oracles
Extend case/expectation definitions so each important oracle carries:
Add a completeness test that rejects a named expectation without falsification fixtures. Include regressions for swallowed lifecycle commands, unsupported flags/selectors, pseudo refs, shell operators, and invalid positional ordering.
3. Make runner outcomes typed and evidence honest
Replace raw-string success inference with a discriminated outcome:
successoutcomes may reach command validation and scoring.runnerErrorand model-validation failures.N/A, not0/0 (0%).--evidencemode that requires at least three evaluated trials and emits paste-ready Markdown containing commit SHA, runner/model, evaluated trials, runner errors, pass rates, and help-document hashes.4. Give help-conformance tooling explicit affected-check ownership
help-conformancecheck tocheck:affected.scripts/help-conformance-*and focused tests.test/skillgym/README.mdas documentation rather than model input.5. Split oversized harnesses and add a merge-base ratchet
Current sizes:
scripts/help-conformance-bench.mjs: 846 linestest/skillgym/suites/agent-device-smoke-suite.ts: 2,807 linesscripts/__tests__/help-conformance-bench.test.ts: 449 linesSplit by question: case definitions, runners, reporting, scoring, shared SkillGym harness, fixture cases, and guidance cases. Then add a merge-base-aware size guard:
The ratchet should report the file, baseline/current size, applicable rule, and recovery action. It should not require a repository-wide cleanup before landing.
Suggested delivery order
Separate PRs are expected; later work should build on the production contract rather than duplicating it.
Acceptance criteria
CommandSchema; no parallel command/arity map remains.N/A.check:affectedselects a deterministic help-conformance check for harness changes without automatically requiring paid model runs.test/skillgym/README.mdchanges remain documentation-only.Non-goals
Related: #1404