Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## Unreleased

- `agent-device help workflow` is now a compact ~8KB card instead of a ~41KB dump; the same depth still exists, split into `help scripting` (save-script, secret-safe fills, batch JSON, replay divergence/repair, recording) and `help gestures` (multi-touch shapes and platform quirks), plus a few paragraphs folded into the topics that already owned the subject (`help debugging`, `help physical-device`, `help validate`). Every `help <topic>` first line is now `agent-device <version> — <topic>` so an agent can read the installed version from its mandatory first help read instead of a separate `agent-device --version` call.

- `scroll` and `back` now accept `--settle` (with `--settle-quiet` and `--timeout`), collapsing scroll-then-observe and back-then-observe into one call (#1638). The response carries the same settled payload the touch commands return — verdict, changed-lines diff with fresh refs on added lines, the unchanged-interactive tail, and `refsGeneration` when the settled tree was stored — and is best-effort: it never fails the action. One difference is deliberate: `scroll`/`back` resolve no element, so the diff baseline is the session's stored pre-action tree ("the last tree you observed") rather than a freshly resolved pre-action capture. Both commands now also preserve the daemon on timeout, like the other settle-capable commands.
- Security: repository `./agent-device.json` now accepts only project-safe automation defaults. It rejects daemon endpoint/auth/transport/server settings, tenant/run/lease selectors, provider/cloud and Metro connection fields, headers, executable reporter modules, local write destinations, and other operator-controlled values before local module loading or any daemon health/RPC request. Put remote endpoint and token together in protected CI environment variables, user config, an explicit `--config` file, or the existing `connect`/`--remote-config` workflow. Daemon auth tokens no longer travel in serialized command flags.
- `viewport` is now rejected during capability admission on Apple targets instead of reaching the device and failing inside dispatch. No Apple backend can resize a screen — simulator and device geometry is fixed by the selected device type — so `viewport` on iOS/iPadOS/tvOS/macOS now fails with `UNSUPPORTED_OPERATION`, `viewport is not supported on this device`, and a hint pointing at `--platform web` and at picking a different simulator. `capabilities` no longer advertises `viewport` on Apple targets. Web viewport resizing (`agent-device viewport 1280 900 --platform web`) is unchanged, and Android was already denied.
Expand Down
74 changes: 74 additions & 0 deletions scripts/__tests__/help-conformance-bench.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -379,6 +379,80 @@ test('plan validator rejects shell projection and non-permitted executables', as
assert.ok(placeholder.issues.some(({ kind }) => kind === 'shell-projection'));
});

// The compact workflow card teaches chaining confident consecutive steps
// with an unquoted `&&`. This is the validator side of that contract: split
// on `&&` and validate each chained segment as its own agent-device command,
// instead of failing the whole line as one shell-projection violation.
test('plan validator splits an unquoted && chain into independently valid segments', async () => {
const [press, fill] = await validatePlanCommands([
'agent-device press \'label="Search"\' --settle && agent-device fill \'label="Search"\' "query" --settle',
]);
assert.equal(press.issues.length, 0);
assert.deepEqual(press.tokens, ['agent-device', 'press', 'label="Search"', '--settle']);
assert.equal(fill.issues.length, 0);
assert.deepEqual(fill.tokens, ['agent-device', 'fill', 'label="Search"', 'query', '--settle']);
});

test('plan validator fails only the offending segment of a chained plan', async () => {
const [goodFirst, badSecond] = await validatePlanCommands([
'agent-device snapshot -i && agent-device press @<search-ref> --settle',
]);
assert.equal(goodFirst.issues.length, 0);
assert.ok(badSecond.issues.some(({ kind }) => kind === 'pseudo-ref'));
});

test('plan validator does not split && inside a quoted selector value', async () => {
const [single] = await validatePlanCommands([
'agent-device fill \'label="A && B"\' "value" --settle',
]);
assert.equal(single.issues.length, 0);
assert.deepEqual(single.tokens, ['agent-device', 'fill', 'label="A && B"', 'value', '--settle']);
});

test('plan validator still rejects an unquoted lone & as a shell operator', async () => {
const [lone] = await validatePlanCommands(['agent-device open foo & agent-device close']);
assert.equal(lone.issues[0]?.kind, 'shell-projection');
});

test('plan validator keeps single-command results identical when no chain is present', async () => {
const [single] = await validatePlanCommands(['agent-device snapshot -i']);
assert.equal(single.issues.length, 0);
assert.deepEqual(single.tokens, ['agent-device', 'snapshot', '-i']);
assert.equal(single.command, 'agent-device snapshot -i');
});

// A real shell rejects && with an empty operand on either side. A validator
// that silently dropped the empty segment (instead of failing it) would
// bless a plan that fails at execution — exactly the gap review found.
test('plan validator rejects a leading && as an empty chain operand', async () => {
const [empty, closeSegment] = await validatePlanCommands(['&& agent-device close']);
assert.equal(empty.issues[0]?.kind, 'empty-chain-operand');
assert.equal(closeSegment.issues.length, 0);
});

test('plan validator rejects a trailing && as an empty chain operand', async () => {
const [pressSegment, empty] = await validatePlanCommands(['agent-device press @e1 --settle &&']);
assert.equal(pressSegment.issues.length, 0);
assert.equal(empty.issues[0]?.kind, 'empty-chain-operand');
});

test('plan validator rejects a doubled && as an empty chain operand', async () => {
const [openSegment, empty, closeSegment] = await validatePlanCommands([
'agent-device open foo && && agent-device close',
]);
assert.equal(openSegment.issues.length, 0);
assert.equal(empty.issues[0]?.kind, 'empty-chain-operand');
assert.equal(closeSegment.issues.length, 0);
});

test('plan validator still allows a quoted && to pass through a single segment unsplit', async () => {
const [single] = await validatePlanCommands([
'agent-device fill \'label="A && B"\' "value" --settle',
]);
assert.equal(single.issues.length, 0);
assert.deepEqual(single.tokens, ['agent-device', 'fill', 'label="A && B"', 'value', '--settle']);
});

test('case matchers score parsed tokens so shell quoting does not change results', async () => {
const commands = [
'agent-device open "com.example.shop"',
Expand Down
6 changes: 6 additions & 0 deletions scripts/help-conformance-case-checks.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ const EXPECTATION_SCORERS = {
),
usesSnapshotI: ({ commands }) => commands.some((command) => /\bsnapshot\b.*\s-i\b/.test(command)),
usesSettleOnMutations: ({ commands }) => allMutationsUseSettle(commands),
// help workflow teaches chaining confident consecutive steps with an
// unquoted &&. This reads the raw (pre-split) command lines, not `joined`
// (canonicalPlan flattens a chain into separate lines once the plan
// validator splits and validates each segment), so it is the only place
// that can tell whether the model actually chained.
usesConfidentChaining: ({ commands }) => commands.some((command) => /&&/.test(command)),
noWaitStable: ({ joined }) => !joined.includes('wait stable'),
verifiesNamedExpectation: ({ joined }) => /\b(wait|is|get|find)\b/.test(joined),
usesDogfoodEvidence: ({ joined }) =>
Expand Down
80 changes: 80 additions & 0 deletions scripts/help-conformance-cases.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,86 @@ export const CASES = [
{ id: 'noOpenArtifactPath', pattern: /(?:^|\n)agent-device\s+open\s+[^\n]*\.apk\b/i },
],
},
{
// Review of the compact workflow card's && guidance found the plan
// validator failed a plan that followed it (unquoted && classified as
// shell-projection). Both target elements are already named/unambiguous
// here, which is exactly the "confident consecutive steps" case the card
// describes, so usesConfidentChaining is a real (not just possible)
// expectation, and validPlanCommands proves the fixed validator accepts
// the chained shape end to end.
id: 'chains-confident-consecutive-settle-steps',
docs: ['--help:first30', 'workflow'],
task: 'The Search tab is visible, labeled "Search", and known to reveal a search field also labeled "Search" with no other candidate on screen. Plan commands to press the Search tab and fill that field with "react native", settling after each step, then close. Plan commands only.',
expectations: [
'validPlanCommands',
'fullPrefix',
'usesSettleOnMutations',
'usesConfidentChaining',
],
matchers: [
{
id: 'pressesSearchTab',
pattern: /\bagent-device\s+press\b[^\n]*label="?search"?[^\n]*--settle\b/i,
},
{
id: 'fillsSearchField',
pattern:
/\bagent-device\s+fill\b[^\n]*label="?search"?[^\n]*(?:"react native"|'react native')[^\n]*--settle\b/i,
},
],
},
{
// help scripting owns --record-as secret-safe fills and save-script
// authoring now that this content left the mandatory workflow card;
// this proves an agent can actually plan the loop from the topic alone.
id: 'scripting-secret-safe-recorded-login',
docs: ['--help:first30', 'scripting'],
task: 'Author a reusable login script for the installed app com.example.app that never records the literal password. The AD_VAR_PASSWORD environment variable is already set in your shell, so do not plan a shell export line. Arm recording on open with --save-script=login.ad, fill the password field (id="password") from AD_VAR_PASSWORD using --record-as, verify the login succeeded, then publish the script without closing the session. Plan agent-device commands only.',
expectations: ['validPlanCommands', 'fullPrefix'],
matchers: [
{
id: 'armsSaveScriptOnOpen',
pattern: /\bagent-device\s+open\s+com\.example\.app\b[^\n]*--save-script[=\s]*login\.ad/i,
},
{
id: 'recordsSecretSafeFill',
pattern:
/\bagent-device\s+fill\s+(?:'|")?id="?password"?(?:'|")?\s+"?\$AD_VAR_PASSWORD"?[^\n]*--record-as\s+PASSWORD\b/i,
},
{
id: 'verifiesLoginSucceeded',
pattern: /\b(?:wait|is|get|find)\b/i,
},
{ id: 'publishesWithoutClosing', pattern: /\bagent-device\s+session\s+save-script\b/i },
],
forbidden: [
{ id: 'noBareClose', pattern: /(?:^|\n)agent-device\s+close\b/i },
{ id: 'noNoRecordOnSecretFill', pattern: /--no-record/i },
],
},
{
// help gestures owns multi-touch shapes now that this content left the
// mandatory workflow card. The exact verification text ("pan changed
// yes") only appears in the gestures topic's own example, so a correct
// plan proves the model actually read it rather than guessing a shape.
id: 'gestures-android-transform-then-verify',
docs: ['--help:first30', 'gestures'],
task: 'On the already-open Android app, plan a combined pan/scale/rotate transform gesture centered at (200, 420) with dx=80, dy=-40, scale=2, rotate=35 degrees over 700ms, then verify the app-reported pan change using the exact confirmation text shown in the gesture reference. Plan commands only.',
expectations: ['validPlanCommands', 'fullPrefix'],
matchers: [
{
id: 'runsAndroidTransform',
pattern:
/\bagent-device\s+gesture\s+transform\s+200\s+420\s+80\s+-40\s+2\s+35\s+700\b[^\n]*--platform\s+android\b/i,
},
{
id: 'verifiesSemanticPanChange',
pattern: /\bagent-device\s+wait\s+text\s+"pan changed yes"[^\n]*--platform\s+android\b/i,
},
],
forbidden: [{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET }],
},
// Next-command quiz cases: captured output (pinned to the real renderer by
// scripts/__tests__/help-conformance-sample-outputs.test.ts) plus a task,
// scored by regex instead of the named expectation scorers above.
Expand Down
71 changes: 70 additions & 1 deletion scripts/help-conformance-plan-validator.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ const COMMAND_VALIDATOR = join(ROOT, 'scripts', 'help-conformance-command-valida
const ALLOWED_PNPM_SCRIPTS = new Set(['build', 'build:android', 'build:xcuitest', 'clean:daemon']);

export async function validatePlanCommands(commands, options = {}) {
const parsedCommands = commands.map((command) => parsePlanCommand(command));
const parsedCommands = commands.flatMap((command) => parseCommandLine(command));
const agentCommands = parsedCommands.filter(
({ tokens, issues }) => issues.length === 0 && tokens[0] === 'agent-device',
);
Expand Down Expand Up @@ -38,6 +38,75 @@ function applyCommandPolicy(parsed, agentResultState, allowedExternalCommands) {
);
}

// The compact workflow card teaches chaining confident consecutive steps with
// an unquoted `&&` (`press ... --settle && fill ... --settle`). Split on it
// before tokenizing a line so each chained segment is validated as its own
// full agent-device invocation, rather than the whole line failing as one
// shell-projection violation. A `&&` inside a quoted selector value (for
// example label="A && B") is not a chain boundary and must not split.
//
// A real shell rejects `&&` with an empty operand on either side (leading
// `&& foo`, trailing `foo &&`, or doubled `foo && && bar`): each is a syntax
// error, not two commands. The splitter below produces an empty segment for
// exactly those shapes, so parseChainSegment turns an empty (post-trim)
// segment into a validation issue instead of silently dropping it — a plan
// with one of these shapes must not be blessed by validPlanCommands when it
// would fail at execution.
function parseCommandLine(command) {
return splitOnUnquotedAnd(command).map((segment) => parseChainSegment(segment));
}

function parseChainSegment(segment) {
if (segment.trim().length > 0) return parsePlanCommand(segment.trim());
return {
command: segment,
tokens: [],
issues: [
{
kind: 'empty-chain-operand',
error:
'A && chain must have a non-empty command on both sides (no leading, trailing, or doubled &&).',
},
],
};
}

function splitOnUnquotedAnd(command) {
const state = { segments: [], current: '', quote: undefined };
for (let index = 0; index < command.length; index += 1) {
index = consumeSplitCharacter(command, index, state);
}
state.segments.push(state.current);
return state.segments;
}

function consumeSplitCharacter(command, index, state) {
const character = command[index];
if (state.quote) return consumeQuotedSplitCharacter(command, index, character, state);
if (character === "'" || character === '"') {
state.quote = character;
state.current += character;
return index;
}
if (character === '&' && command[index + 1] === '&') {
state.segments.push(state.current);
state.current = '';
return index + 1;
}
state.current += character;
return index;
}

function consumeQuotedSplitCharacter(command, index, character, state) {
if (character === '\\' && state.quote === '"' && index + 1 < command.length) {
state.current += character + command[index + 1];
return index + 1;
}
state.current += character;
if (character === state.quote) state.quote = undefined;
return index;
}

function parsePlanCommand(command) {
const tokenized = tokenize(command);
const issues = [];
Expand Down
18 changes: 7 additions & 11 deletions skills/agent-device/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,7 @@ description: Automates Apple-platform apps (iOS, tvOS, macOS), Android devices,

# agent-device

Router only. Private setup before using this skill:

```bash
agent-device --version
```

If that fails but the user may have installed `agent-device` globally, check the user's configured login/interactive shell and environment before using `npx`. Resolve the command the same way the user would from a normal terminal session, then run the absolute binary path if found. This may require inspecting shell startup behavior or package-manager/global bin locations; do not assume the Codex process `PATH` is the user's `PATH`.

Require `agent-device >= 0.20.0`; older CLIs lack the current help topics and Vega OS routing. If older, stop and tell the user to upgrade the trusted install or approve an exact-version npm command. Do not run `npm install -g agent-device@latest` or `npx -y agent-device@latest` autonomously, and do not include version/upgrade commands in final plans.

Before your first agent-device command or plan, read the smallest version-matched CLI guide that fits the task:
Router only. Before your first agent-device command or plan, read the smallest version-matched CLI guide that fits the task — this single read also replaces a separate `agent-device --version` check:

```bash
agent-device help manual-qa # scripted/manual QA, acceptance checks, checklist execution
Expand All @@ -24,10 +14,16 @@ agent-device help dogfood # exploratory app dogfooding and evidence collecti
agent-device help workflow # fallback reference for general app driving or mixed tasks
```

That topic's first line is `agent-device <version> — <topic>` (for example `agent-device 0.21.0 — workflow`). Read the version from it instead of running `agent-device --version` separately. If the first line instead reads `agent-device help <topic>` with no version — or the command fails, or the topic is unrecognized — the installed CLI predates this header and its current help topics/Vega OS routing. Stop and tell the user to upgrade the trusted install or approve an exact-version npm command. Do not run `npm install -g agent-device@latest` or `npx -y agent-device@latest` autonomously, and do not include version/upgrade commands in final plans.

If `agent-device` fails outright but the user may have installed it globally, check the user's configured login/interactive shell and environment before using `npx`. Resolve the command the same way the user would from a normal terminal session, then run the absolute binary path if found. This may require inspecting shell startup behavior or package-manager/global bin locations; do not assume the Codex process `PATH` is the user's `PATH`.

Read additional topics only when relevant:

```bash
agent-device help debugging
agent-device help scripting # save-script, secret-safe fills, batch JSON, replay repair
agent-device help gestures # multi-touch gesture shapes and platform quirks
agent-device help react-native
agent-device help react-devtools
agent-device help cdp
Expand Down
Loading
Loading