feat(cli): compact workflow help card + version header

[thymikee]

Summary

Benchmark evidence (SWM AppControlBench, gpt-5.4-high driving the CLI on 30 Bluesky tasks) showed agent-device help workflow (~41KB) plus a separate agent-device --version handshake were mandated on every task by the shipped skill router, producing 108 boilerplate tool calls across 30 tasks (~3.6 extra model turns/task) and 1266KB of the 1899KB total tool output (67%) being help/version text.

This PR:

1. Shrinks agent-device help workflow from 41025 → 8493 bytes (-79.3%). Content that carries correctness (the open→snapshot→settle→verify→close loop, selector/@ref rules, --settle guidance, platform gotchas, error-recovery hints) stays; duplication, long prose, and rarely-needed depth move to sub-topics — nothing is deleted:
   • New help scripting: save-script/destination-guard authoring, --record-as secret-safe fills, batch JSON step shape, replay divergence/resume, agent-supervised repair (heal-by-doing), recording/tracing.
   • New help gestures: full multi-touch command shapes (pinch/rotate/transform/drag), iOS XCTest-synthesis vs Android geometric-injection quirks, macOS context-menu/wheel-scroll patterns.
   • Folded into topics that already owned the subject: iOS Allow-Paste / Android Gboard-handwriting quirks → help debugging; runner/daemon idle-timeout env vars → help physical-device; Apple-runner CI cache-key note → help validate.
2. Every help <topic> first line is now agent-device <version> — <topic> (e.g. agent-device 0.20.6 — workflow), so the skill reads the CLI version off its mandatory first help read instead of a separate agent-device --version call. skills/agent-device/SKILL.md is updated to do that (with explicit fallback semantics: a first line still reading the old agent-device help <topic> format, or a failed/unrecognized command, means the CLI predates this header and needs upgrading) and otherwise stays a thin router — no behavioral guidance was added there.
3. The compact card gains two terse behavioral rules directly in help workflow:
   • Chain confident consecutive steps with && (falling back to one command at a time when a step is uncertain — ambiguous match, network-backed result, unseen screen).
   • Before declaring a task done, confirm the requested end state is actually visible on the current screen (scrolling into view if needed) — get text alone, or stopping one screen early, is not enough.
4. Ran the repo's help-planning benchmark (scripts/help-conformance-bench.mjs, 25 cases × 2 non-agentic runners) before and after against the live CLI output (never hand-transcribed).
5. Live-validated on a simulator I created myself.

Review follow-ups (all three addressed)

• && vs. the plan validator (P1). The card's &&-chaining guidance conflicted with scripts/help-conformance-plan-validator.mjs, which classified any unquoted && as shell-projection and failed validPlanCommands — canonical help was teaching a plan shape the conformance contract rejected. Fixed by extending the validator, not by dropping the guidance: it now splits a plan line on unquoted && and validates each segment independently (&& inside a quoted selector value, e.g. label="A && B", is not a split boundary and stays intact). Added unit tests for the splitter and a chains-confident-consecutive-settle-steps conformance case that exercises a real chained plan end to end. batch stays untouched — it's deliberately stop-only, a settled design decision, not a workaround target.
• @ref placeholder contradiction (P2). The card said never to target a placeholder like @ref/@eN, then itself wrote snapshot -s @ref. Replaced with a concrete ref and an explicit label: snapshot -s @e12 (the current concrete ref). Reverted the test that had been changed to require the placeholder shape.
• Waivers → real conformance cases (P2). help scripting and help gestures now each have a dedicated conformance case instead of a "we'll add one if regressions show up" waiver: scripting-secret-safe-recorded-login (arm --save-script, --record-as a fill against an env var, publish without closing) and gestures-android-transform-then-verify (plan a combined pan/scale/rotate transform, then verify with the exact confirmation text that only appears in the gestures topic's own example — so a correct plan proves the model actually read it). Both waivers removed.
• Empty && chain operands (P2, follow-up round). The && splitter above filtered out empty segments, so a plan with a leading (&& foo), trailing (foo &&), or doubled (a && && b) operator passed validPlanCommands even though a real shell rejects all three as a syntax error — the validator would have blessed a plan that fails at execution. Fixed: an empty (post-trim) segment is now a dedicated empty-chain-operand issue instead of being silently dropped. Added regression tests for all three shapes plus confirmation the quoted-&& case is unaffected.

Before / After

Metric	Before	After
agent-device help workflow bytes	41025	8493 (-79.3%)
Mandated boilerplate turns/task (router)	version handshake + topic read (2+)	topic read only (1)
help-conformance-bench (25 cases × 2 runners, pass/total, repeat=1)	codex 15/22, haiku 14/22 → 29/44 (22-case baseline, pre-review-fixes)	codex 19/25, haiku 14/25 → 33/50 (latest full run, post-review-fixes)

Read the bench delta with the noise floor this framework actually has at repeat=1, not as a clean signal: one case with zero exposure to any change in this PR (ios-system-ui-widget-flow, docs are --help:first30 + an untouched topic) swung 10/10 → 7/10 → 1/10 across three consecutive runs — the 1/10 was a claude CLI subprocess API error, not a content issue. Across two independent full post-fix runs the combined score was 32/50 and 33/50 — statistically on par with the 29/44 (65.9%) baseline. The two things actually being tested by the new cases behave correctly: scripting-secret-safe-recorded-login passes 8/8 for both runners (after one case-authoring bug of my own — an export line the task didn't need — was found and fixed), and chains-confident-consecutive-settle-steps correctly produces a fully valid plan either way, reporting the model's choice not to chain as a soft, informative signal rather than a validator failure (matching the original benchmark finding that models chained only twice in 30 tasks even when told they could).

One real regression surfaced and was fixed before any of the numbers above: the compact Bootstrap section initially dropped the explicit "install arguments are id then artifact path" rule (kept only the example command), and both runners on workflow-install-artifact-before-open started omitting the package id from install calls. Re-added the explicit rule sentence; the case now passes for both runners consistently.

Live smoke test

Created my own simulator (never touched bench-golden* or the reserved UDID), booted it, and ran the exact sequence the skill prescribes with an isolated AGENT_DEVICE_STATE_DIR:

agent-device help workflow                                          # 8493 bytes, version-headed
agent-device open com.apple.Preferences --platform ios --device claude-helpdiet-test
agent-device snapshot -i --platform ios --device claude-helpdiet-test
agent-device press 'label="General"' --settle --platform ios --device claude-helpdiet-test
agent-device close --platform ios --device claude-helpdiet-test

All five commands worked end-to-end using only the compact card's guidance — no missing command shapes. press --settle returned a full settled diff (+14/-14) showing the General screen's content, matching the "continue from the settled diff" guidance verbatim. Cleaned up: daemon stopped, simulator shut down and deleted.

Tests

• src/cli/parser/__tests__/cli-help-topics.test.ts, src/__tests__/cli-help.test.ts, src/cli/parser/__tests__/cli-help-command-usage.test.ts — rewritten to match the new compact content, new topics, and version headers.
• scripts/__tests__/help-conformance-bench.test.ts — new unit tests for the && segment splitter (splits a chain, leaves a quoted && alone, still rejects a lone unquoted &, fails only the offending segment of a chain).
• scripts/help-conformance-cases.mjs — three new conformance cases (chains-confident-consecutive-settle-steps, scripting-secret-safe-recorded-login, gestures-android-transform-then-verify); scripts/__tests__/help-conformance-topic-coverage.test.ts — both waivers removed now that scripting/gestures have real cases.
• src/commands/management/session.test.ts — a pre-existing production test asserted save-script/--record-as content directly against help workflow; updated to follow the pointer to help scripting, where that content now lives.
• All touched suites pass locally (269+ tests in the help-specific files, 5600/5600 across the full unit-core+subprocess-stub run); oxlint and oxfmt --check are clean repo-wide. CI is green (Coverage, Lint & Format, Typecheck, Layering Guard, Smoke Tests, etc. all pass).

What was left out

• Didn't chase the exact ≤8KB target to the byte — landed at 8493B (vs the 8192B target) after restoring the install-order rule the bench caught as a real regression, plus the @e12 fix from review. Judged that trade worth it: benchmark-verified correctness over an arbitrary few hundred bytes.
• Didn't push for a cleaner single bench number by re-running further — repeat=1 noise (documented above) means chasing one more point is not signal. The unit tests are the deterministic proof for the validator fix; the bench numbers are corroborating, not load-bearing.

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	2.00 MB	1.97 MB	-24.9 kB
JS gzip	645.6 kB	632.4 kB	-13.2 kB
npm tarball	773.3 kB	766.3 kB	-7.0 kB
npm unpacked	2.71 MB	2.68 MB	-24.4 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	25.6 ms	26.3 ms	+0.7 ms
CLI --help	66.1 ms	63.9 ms	-2.2 ms

Top changed chunks:

Chunk	Raw diff	Gzip diff
dist/src/viewport-dimension.js	+78.7 kB	+25.7 kB
dist/src/cli.js	+74.0 kB	+21.2 kB
dist/src/selector-vocabulary.js	+46.0 kB	+13.9 kB
dist/src/version.js	+37.6 kB	+12.1 kB
dist/src/cli-help.js	-21.4 kB	-6.2 kB

[thymikee]

Review found three actionable gaps in the help contract:

• [P1] Do not ship && guidance while the owning plan validator rejects it. The compact workflow now says Chain confident consecutive steps with && and gives a chained command, but scripts/help-conformance-plan-validator.mjs classifies every unquoted && as shell-projection and validPlanCommands fails the case. So an agent that follows the new primary instruction produces a plan this repository declares invalid. The 32/44 repeat=1 result does not prove this path—the sampled runners simply did not consistently follow the new rule. Either keep canonical help to one command per line and pursue batching through the typed batch surface, or deliberately extend the planner contract to split/validate each chained segment, with a focused conformance case and validator tests.

• [P2] Replace the literal @ref placeholder in the compact card. The same card says never to use @ref/@eN placeholders, then instructs Truncated preview: snapshot -s @ref, not get text. The old text included the concrete runnable example snapshot -s @e7; the new test was changed to require the contradictory placeholder. Keep a concrete illustrative ref (@e12) or explicitly say “the current concrete ref,” so the highest-frequency help card does not teach an invalid target shape.

• [P2] Give the two new routing destinations conformance ownership instead of waiving them. scripting now owns security-sensitive recorded-fill/replay-repair command planning and gestures owns complex multi-touch shapes, specifically because that content left the mandatory workflow card. Adding waivers whose rationale is “add a case when regressions show up” makes regressions invisible and conflicts with the repo rule that changed command-planning guidance gets a help-conformance case. Add at least one real planning case per topic (for example secret-safe recorded fill/resume repair, and transform/pinch planning) before relying on the split as semantically preserved.

The 79% size reduction, version-header consolidation, moved-content organization, deterministic size pin, and live common-loop smoke otherwise look good. I did not apply ready-for-human while these findings remain.

[thymikee]

Thanks — all three fixed, pushed in 80ee870 and 3c47adc:

1. [P1] && vs. the plan validator. Extended scripts/help-conformance-plan-validator.mjs to split a plan line on unquoted && and validate each segment independently, instead of rejecting the whole line as one shell-projection violation. Quoted && inside a selector value (label="A && B") is not a split boundary. Added unit tests for the splitter (chain splits and validates both segments; only the offending segment of a chain fails; a quoted && doesn't split; a lone unquoted & still fails) and a chains-confident-consecutive-settle-steps conformance case. Did not touch batch — agreed that's a settled, separate design decision. Kept the card's fall-back-to-single-steps sentence as is.

2. [P2] @ref placeholder contradiction. Fixed: Truncated preview: snapshot -s @e12 (the current concrete ref), not get text. Reverted the test to demand that concrete shape (and assert the placeholder is gone).

3. [P2] Waivers → real cases. Removed both waivers. help scripting now has scripting-secret-safe-recorded-login (arm --save-script, --record-as a fill against an env var, publish without closing). help gestures now has gestures-android-transform-then-verify (plan a combined transform gesture, then verify with the exact confirmation text that only appears in the gestures topic's own example, so a correct plan proves the content was actually read).

Re-ran the full help-conformance bench (25 cases × 2 runners) after all three fixes. Two full runs landed at 32/50 and 33/50, on par with the 29/44 pre-fix baseline — read against the framework's real repeat=1 noise floor, which I made concrete this round: a case with zero exposure to anything in this PR (ios-system-ui-widget-flow) swung 10/10 → 7/10 → 1/10 across three runs (the 1/10 was a claude CLI subprocess API error, not content). scripting-secret-safe-recorded-login passes 8/8 for both runners after I fixed a bug in my own case (the task didn't need to ask for a shell export line, and the validator correctly has no grammar for one). chains-confident-consecutive-settle-steps produces a fully valid plan whether or not the model actually chains — which is honest: models don't always take the chaining option even when told they can, and that's a signal worth keeping visible, not a validator defect.

Also found and fixed a real production-test casualty while chasing this: src/commands/management/session.test.ts asserted save-script/--record-as content directly against help workflow; updated it to follow the pointer to help scripting, where that content now lives. CI is green.

Full details and the updated before/after table are in the PR description.

[thymikee]

Re-reviewed 3c47adc: the three prior findings and stale Coverage assertion are fixed. One P2 remains: splitOnUnquotedAnd() filters empty segments, so leading, trailing, or doubled && can pass validPlanCommands even though the shell rejects them. Preserve empty operands as validation errors and add leading/trailing/doubled-chain regressions. All CI is green, but this conformance hole blocks readiness.

[thymikee]

Follow-up P2 fixed, pushed in 5606eea.

splitOnUnquotedAnd() was trimming and filtering out empty segments, so a plan with a leading (&& agent-device close), trailing (agent-device press @e1 --settle &&), or doubled (agent-device open foo && && agent-device close) && passed validPlanCommands even though a real shell rejects all three as a syntax error.

Fix: empty segments are no longer dropped. parseChainSegment now checks each segment before parsing, and an empty (post-trim) one becomes a dedicated empty-chain-operand issue instead of silently vanishing — so the whole chain correctly fails validation. Quoted && (label="A && B") still doesn't split, and a plain single command with no chain still parses identically to before (verified by the existing keeps single-command results identical test, unchanged and still green).

Added 4 regression tests: leading, trailing, and doubled && each produce empty-chain-operand on the empty side while the real segment(s) still pass; and the quoted-&& case stays intact as one unsplit, valid segment.

CI is green (all checks pass, including Coverage).

[thymikee]

Re-reviewed 5606eea: the remaining empty-operand hole is fixed. Leading, trailing, and doubled && now produce empty-chain-operand; quoted && remains unsplit, and the regression tests cover each shape. Prior findings remain fixed, all CI is green, and merge state is clean. Code review clean.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-07 11:27 UTC