test(ios): expand simulator e2e coverage

[thymikee]

Summary

This PR turns the iOS simulator test app into a durable end-to-end proving ground for the public agent-device surface. It adds catalog-complete command ownership, realistic cross-command journeys, and a smoke/full tier split so pull requests get fast confidence while expensive lifecycle and artifact cases remain comprehensive.

Test coverage added

• A registry-derived coverage contract assigns an executable owner to all 53 public commands. Most commands are exercised live on a simulator; host-restricted commands point to named contract tests, unsupported TV input is verified through capability denial, and the viewport gap remains explicitly tracked by fix(ios): align simulator viewport capability with backend support #1407.
• Four PR smoke journeys cover:
  • inventory, capability discovery, doctor, cached fixture installation, and device selection
  • cold-start deep links, exact route navigation, snapshots, selectors, click/press/long-press, native alert get/dismiss/accept, modal interaction, waits, and back navigation
  • fill/type/focus, keyboard dismissal, and snapshot diff
  • screenshot artifact validity and session close/lease cleanup
• Five full/nightly journeys cover:
  • foreground/background restoration, Home and app switcher, clipboard, orientation, appearance settings, permission-state recovery, scrolling, custom app events, and app state
  • logs, performance metrics, batch results, paginated event history, trace diagnostics, React Native detection, and playable screen recordings
  • replay, test-suite/JUnit output, directional swipes, pan/fling/pinch/rotate gestures, and long-list edge recovery
  • simulator shutdown, boot, reinstall, and restored fixture identity
  • executable ownership for known gaps
• Eight cross-command behavior contracts prevent command-by-command coverage from hiding broken user journeys: cold deep-link navigation, keyboard lifecycle, background/foreground resume, modal open/close, permission recovery, interrupted system UI, long-list recovery, and host-focus preservation.
• Assertions prove observable outcomes—app state, native device state, typed data, pixels, files, logs, and artifacts—rather than treating a zero exit code as coverage.

Project bugs fixed

The new journeys exposed and now pin regressions for:

• Apple alert acceptance omitted the standard iOS Open action used by deep-link confirmation dialogs.
• Orientation reported the requested value without verifying XCUIDevice reached that state.
• iOS simulator recording could report readiness before simctl created its output, and cleanup could mishandle a rejected process monitor.
• Recording trace diagnostics were not always attached to an existing session unless debug logging was enabled.
• Scroll-edge recovery could derive an ambiguous scope from non-unique container values.
• Screenshot scale lookup could fail during legitimate bounded CoreSimulator stalls.

Fast, reusable CI

• The test app is built once per native platform fingerprint and reused by simulator jobs instead of being rebuilt inside each smoke run.
• Producer and consumer use one platform-scoped artifact-name resolver and accept only artifacts from the designated workflow at the exact head or a trusted default-branch push.
• Collision, wrong-workflow, unrelated-head, malformed-fingerprint, API-outage, queued, failed, and absent-producer cases have executable regression coverage with bounded fallback.
• XCUITest runner preparation reuses exact cache hits, build concurrency is scoped per artifact and PR/ref, fork PRs avoid duplicate iOS native builds, and Android keeps its platform-specific producer path.
• CI step names and macOS runner selection are aligned around what each step actually does.

Scope: 74 touched files (+4,538/-390) across the fixture app, CI workflows/actions, simulator E2E suites, documentation, provider tests, recording/diagnostics, scroll recovery, and Apple runner behavior. The scope deliberately expanded beyond the original test suite because the live tests uncovered production defects and required trustworthy, reusable native-build infrastructure.

Tracks #320: the remaining live system permission-prompt accept/dismiss/recovery journey stays explicitly owned there; this PR does not claim to close it.
Tracks #1407 for the admitted iOS viewport backend gap.

Validation

Exact-head validation at 2887483 is fully green:

• all PR static, typecheck, Fallow, format, integration, coverage, FreeRange, CodeQL, packaging, and platform smoke checks passed
• iOS Release built and published the canonical fixture artifact in 17m50s
• iOS Smoke fetched that artifact without an inline fixture build; targeted runner XCTest regressions, public CLI preflight, Settings replay, and host-focus preservation passed
• fixture-backed iOS simulator E2E passed in 1m50s, including the formerly failing Open confirmation and exact catalog destination
• Android smoke passed in 6m47s; Android Release built and published its canonical artifact in 26m35s
• focused alert acceptance, orientation, recording lifetime/cleanup, trace, scroll-edge, artifact provenance, artifact naming, and producer-fallback regressions passed
• the static coverage gate verifies that every public command and every required cross-command behavior retains an executable owner

No residual CI failures or actionable review findings remain.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-28 18:09 UTC

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.87 MB	1.87 MB	+1.6 kB
JS gzip	599.0 kB	599.5 kB	+433 B
npm tarball	714.8 kB	715.7 kB	+894 B
npm unpacked	2.50 MB	2.50 MB	+3.3 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	28.1 ms	30.3 ms	+2.1 ms
CLI --help	59.2 ms	58.5 ms	-0.7 ms

Top changed chunks:

Chunk	Raw diff	Gzip diff
dist/src/internal/daemon.js	+943 B	+252 B
dist/src/recording-export-quality.js	+207 B	+50 B
dist/src/sdk-android-adb.js	+35 B	+10 B
dist/src/runner-disposal.js	+17 B	+7 B
dist/src/runner-client.js	+4 B	+6 B

[thymikee]

Code review is clean at 00632a5: no actionable implementation or regression-test defect found. #320’s fixture-coverage intent is met without deleting the system-app coverage, #1407 remains explicitly owned as a live typed viewport known gap, and the catalog-derived command/behavior manifests point to executable scenario evidence. The production trace/diagnostics changes and fixture artifact reuse flow are coherent.

P2 owner action: the PR body must disclose the required touched-file count and scope expansion. This is a 38-file (+2,751/-51) change spanning fixture app, workflows, docs, E2E suites, and two production diagnostics/daemon paths; add that statement before merge.

Current-head required checks observed so far are passing, with iOS smoke, Coverage, FreeRange, Linux, and Android still in progress and no failures. The body records a full local simulator run (1/1, 186s); independent live iOS CI evidence is pending. Code review is clean, so ready-for-human may be applied while CI completes; if CI fails, remove it and address the failure.

[thymikee]

Re-reviewed 2cd3b7a after the latest CI-stabilization delta. No actionable code or test finding remains. The Android helper cache keys include the helper sources, packaging scripts, version, and the exact android/snapshot-helper/debug.keystore used to sign the IME helper. The iOS permission-reset restart/assertion is sound. All 27 checks pass, including the changed fixture E2E smoke and host-focus assertion, so the existing ready-for-human label remains justified.

P2 owner action remains and the numbers have expanded: update the PR body with the required scope disclosure for the current 41-file (+2,836/-78) diff spanning fixture app, workflows/actions, docs, E2E suites, and production diagnostics/daemon paths. This is required before merge.

[thymikee]

Re-reviewed 86759f1. P1 test-route failure: resetMicrophonePermissionAndRestart() now immediately reads id="automation-microphone-permission" after restoring the Automation screen, but that control is below the initial viewport. The existing clickMicrophonePermission() path deliberately handles offscreen_selector by scrolling; the new precondition assertion bypasses that recovery.

The full simulator run at this exact head fails in Replay Nightly run 30231730509, Run full fixture-backed iOS simulator E2E, with COMMAND_FAILED: Selector did not match: id="automation-microphone-permission" at live-full-scenarios.ts:170. This is a deterministic test-route defect, not infrastructure. Make the assertion after the control is brought into view, or give the reset helper equivalent bounded scroll/retry behavior, then rerun the full tier.

The 27 PR checks are green because they exercise the smoke tier; they do not cover this affected full:lifecycle-system path. ready-for-human must be removed until the full-tier failure is fixed. The PR-body scope disclosure also remains missing and now covers 41 files (+2,836/-78).

[thymikee]

Exact-head full-tier validation is still red on 0c937df (Replay Nightly run 30234556565). The run now fails earlier in smoke:automation-input: after the relaunch/deep-link step, the app remains on the Home/Gesture Lab surface and wait text "Automation lab" 10000 times out. Because the run never reaches full:lifecycle-system, the new inactive-app permission reset and scroll-aware assertion do not yet have passing live evidence, so the prior P1 cannot be confirmed closed. Please diagnose/rerun the full tier and require it to reach and pass the permission scenario. The 27 PR checks are green but cover the smoke tier only.

Also update the PR body with the required large-scope disclosure (currently 42 files, +2,876/-78).

[thymikee]

Re-reviewed 1f40a9a. The latest delta has no new code finding: re-delivering the trusted cold deep link directly addresses the earlier Home/Gesture Lab failure, and orientation now validates native XCUIDevice readback instead of assuming the app window must resize.

Readiness is still blocked on live evidence. There is no exact-head Replay Nightly run for 1f40a9a, so the full iOS route has not reached and passed the previously failing permission lifecycle. The ordinary PR checks cover smoke/static gates and cannot close that P1. Run the full Replay Nightly at this head and require full:lifecycle-system to pass before applying ready-for-human.

The PR body also still needs the required scope disclosure: 49 files, +2,973/-92, expanding from E2E coverage into Apple runner/core orientation contract behavior.

[thymikee]

P1 remains on exact head 1f40a9a. Replay Nightly run 30238489078 now gets past the deep-link failure and reaches full:lifecycle-system, so that earlier blocker is fixed. The required permission route still fails deterministically: state clear, microphone TCC reset, reopen, route restore, control discovery, and click id="automation-request-microphone" all succeed; then alert wait 15000 times out with no prompt. Runner/session diagnostics are healthy—no restart, transport, simulator-boot, or infrastructure failure—so increasing the timeout did not fix the route; the prompt is absent, not late.

Diagnose why the fixture request does not produce a system permission prompt, assert the app-visible permission result on the no-prompt path, and rerun the exact-head full tier through a passing permission denial/reset/recovery cycle. Keep ready-for-human withheld.

[thymikee]

P1: 4ca3c900 avoids the failing permission-prompt route instead of fixing the behavior required by linked issue #320. The delta deletes prompt wait/dismiss/accept, renames permission-prompt-recovery to permission-state-recovery, and substitutes direct settings permission reset/grant/deny/reset assertions. That is useful deterministic settings coverage, but the separate app-native alert scenario does not prove the requested system permission prompt accept/dismiss/recovery journey. Keep the state assertions, but restore executable permission-prompt recovery evidence or stop claiming this PR closes #320 and move that requirement to an explicitly owned follow-up.

No exact-head Replay Nightly or iOS smoke/full evidence exists yet for the replacement route. GitHub also reports the branch conflicting/dirty against merged main; update the branch and rerun the required checks/full tier. The PR body scope disclosure remains missing and is now 51 files, +3,157/-96.

[thymikee]

Two P1 blockers remain on the exact head:

1. Closes #320 is still unsupported. The linked issue explicitly requires a real permission-prompt accept/dismiss/recovery journey, but full:lifecycle-system now terminates the app and injects settings permission reset/grant/deny/reset, explicitly avoiding the transient system prompt. That is useful permission-state coverage, but it is not evidence for prompt interaction or recovery. Restore a live prompt accept/dismiss/recovery path, or stop closing test: move replay coverage toward stable mobile app patterns #320 and link a concrete owner for the omitted journey.

2. Exact-head Replay Nightly run 30245882004 fails deterministically in the full iOS E2E. events 100 only returns entries 0–99, while the first required press event has moved past that page after the added Settings wait events; the assertion at live-full-scenarios.ts:280 therefore fails. Make the assertion pagination-aware (or otherwise query the relevant range) and provide a green exact-head full nightly.

The PR body also needs the required scope disclosure: 51 files, +3,158/−96.

[thymikee]

Re-review of a7f2ce32: not ready.

• P1 remains — test: move replay coverage toward stable mobile app patterns #320 is still not satisfied. full:lifecycle-system still uses direct reset/grant/deny/reset state injection and contains no real permission-prompt accept/dismiss/recovery journey, while the body still says Closes #320. Restore that live prompt flow or stop closing test: move replay coverage toward stable mobile app patterns #320 and link a concrete owner for it; fix(ios): align simulator viewport capability with backend support #1407 is unrelated viewport work.
• The new pagination implementation coherently fixes the prior fixed-page assertion in code and has focused tests, but it is not live-proven yet. Exact-head Replay Nightly 30248536233 is red: the full iOS tier fails earlier at record stop (failed to stop recording: Note: No display specified…), so it never reaches the paginated-events assertion. This failure is outside the three-file pagination delta, but a green exact-head full tier is still required before readiness.
• The body still needs the required scope disclosure: 52 files, +3,321/−96.

Ordinary checks are green and GitHub reports CLEAN/MERGEABLE; ready-for-human remains withheld because of the confirmed #320 gap and red required validation.

[thymikee]

Re-reviewed exact head 6769e4ea. The recorder-lifetime delta is sound and non-vacuously tested, and ordinary checks are green. Two P1 blockers remain: (1) Closes #320 is still inaccurate—the full lifecycle scenario uses direct permission reset/grant/deny/reset and explicitly avoids the real system permission prompt, so it does not cover prompt accept/dismiss/recovery; either restore that live journey or stop claiming #320. (2) exact-head Replay Nightly run 30255398120 is still in progress and has not reached the full fixture-backed iOS E2E, so the prior record-stop/full-route failure is not yet evidenced resolved. P2: the PR body still omits required scope disclosure (58 files, +3,743/−191 across CI, docs, E2E, provider tests, and daemon recording behavior). Keep ready-for-human withheld.

[thymikee]

Re-reviewed exact head 8fde9e46. Not ready.

• P1 — dependency-cache key silently drops Node major. .github/actions/setup-test-app-dependencies/action.yml runs node -p 'process.versions.node.split(\\".\\")[0]'; the backslashes reach Node and throw SyntaxError: Expected unicode escape. Exact-head Test App Build Cache logs then save test-app-dependencies-macOS-ARM64-node-pnpm11.1.2-…, with an empty Node major, because the surrounding echo masks the substitution failure. This can reuse native node_modules across Node majors. Remove the escaping (for example split(".") as actual JS), then rerun producer/consumer evidence.\n- P1 — Closes #320 remains unsupported. The full lifecycle scenario injects reset/grant/deny/reset state and explicitly avoids the transient system prompt; it never demonstrates prompt accept/dismiss/recovery. Restore that live path or stop closing test: move replay coverage toward stable mobile app patterns #320 and create/own the follow-up.\n- P2 — scope disclosure remains missing. Current diff is 59 files, +3,789/−198 across CI/actions, fixture app, docs, E2E/provider tests, and recording/diagnostics paths.\n\nReplay Nightly 30255398120 passed at prior code-relevant head 6769e4e, including full fixture E2E and record stop, so that prior blocker is resolved. Ordinary exact-head checks are green, but the cache-key failure is masked; no exact-head full nightly exists for the cache-only delta.

[thymikee]

Re-reviewed exact head 274e6da. The cache P1 is not fixed. .github/actions/setup-test-app-dependencies/action.yml still executes node -p 'process.versions.node.split(\\".\\")[0]'; the backslashes that caused SyntaxError: Expected unicode escape remain. Renaming outputs to node_major / pnpm_version does not make Node produce a value, so the key will still collapse to …-node-pnpm… and can reuse native dependencies across Node majors. Remove the JS escaping and require exact-head producer/consumer logs showing a populated node<major> key.\n\nThe other blockers also remain: Closes #320 still lacks a real permission-prompt accept/dismiss/recovery journey, and the required 59-file (+3,789/−198) scope disclosure is absent. Prior full/record-stop evidence at 6769e4e remains valid for that route but does not validate this cache delta. Keep ready-for-human withheld.

[thymikee]

P1: setup-fixture-app restores the newest non-expired repository artifact by fingerprinted name alone, then unpacks and executes it. The producer workflow’s fork guard does not establish consumer trust: the repository artifact query does not verify that the selected artifact came from the designated Test App Build Cache workflow/run or same-repository provenance. Filter candidates to a trusted producer/run and repository identity before download; otherwise take the safe inline fallback.

The latest commit correctly removes the superseded ios-system-ui waiver, so that prior deterministic Coverage failure is addressed, but exact-head Coverage and iOS/device evidence are still pending; the successful full live run cited in the PR is from an older head. Also refresh the stale scope/validation disclosure (current base-to-head scope exceeds the stated 59 files).

The registry-derived coverage manifest itself is structurally sound and honestly owns the #1407/#320 gaps.

[thymikee]

Re-reviewed exact head ee94efa. P1 remains: untrusted fixture artifact selection.

setup-fixture-app still selects the first non-expired repository artifact by fingerprinted name and immediately downloads/unpacks it. The new producer inputs and queued-run handling only govern a cache miss/wait; they never establish that a cache hit came from the designated Test App Build Cache iOS Release producer, its expected run/head, or trusted repository provenance. The safe inline fallback is therefore bypassed for a colliding artifact.

Filter/select candidates through the trusted producer run (and reject ambiguity/untrusted provenance), then fall back inline when no trusted artifact is available. Add a collision regression proving an untrusted same-name artifact cannot be restored, plus exact-head producer/consumer evidence.

The queued-producer fallback itself is correctly added and covered. Current PR checks, including Coverage and the macOS fixture E2E smoke, are green, but they do not validate this provenance boundary.

[thymikee]

Updated review at bf578bdf — two merge blockers remain.

1. Artifact provenance: fixture cache hits still do not prove trusted provenance. query_artifact accepts the first non-expired name match, while producer/head checks run only after a miss. Require the selected artifact to be tied to an approved Test App Build Cache producer run and trusted producer repository (with the applicable producer/head relationship validated), then add a collision regression proving an untrusted same-name artifact is rejected in favor of inline build or a trusted candidate. The current consumer run at bf578bdf restored a fingerprint artifact produced at an earlier head; native-fingerprint reuse can be valid, but the action does not establish its provenance.

2. Current iOS smoke: the fixture fetch succeeded; the failure is later. The first home snapshot was sparse with reasonCode: budget after an 8.96s XCTest capture, but the E2E immediately requires the title. Add bounded retry/recovery for that typed condition, or provide fresh passing live evidence after a targeted fix. This is runtime/test reliability under CI load, not GitHub infrastructure or an artifact-download failure.

[thymikee]

P1 — The latest wait/scoped snapshot fixes the prior sparse home observation, but the next navigation assertion is vacuous: after clicking id="automation-continue-catalog", assertWaitText(context, 'Catalog') can pass on the unchanged Automation screen because that screen already contains the button text ‘Continue to catalog.’ Exact-head iOS smoke demonstrates this: it records the navigation evidence, then times out waiting for Settings while the current surface is still Automation Lab. Wait for a unique catalog landmark such as id="catalog-title" and diagnose/fix the click route if it does not appear. This is a deterministic test-route/runtime failure, not an infrastructure flake.

[thymikee]

Re-reviewed 7756367: the navigation false positive is fixed—the flow now waits for unique id="catalog-title" before recording evidence, and the focused test pins that route. PR is still not ready: the previously reported artifact-provenance P1 is unchanged, GitHub reports DIRTY/CONFLICTING, and only CodeQL ran at this head, so the route that failed on the previous head has no exact-head iOS smoke evidence. Resolve/recompute the conflict and require a passing exact-head iOS smoke after the provenance fix.

[thymikee]

Re-reviewed current head 44b51cf after the rebase. The exact-destination navigation fix is good: targeting id="catalog-title" removes the ambiguous Catalog-text match and the regression test pins that behavior. I found no new issue in this delta.

The PR is still not ready because the previously confirmed P1 fixture-artifact provenance blocker remains unchanged: setup-fixture-app may accept the first non-expired same-name repository artifact without proving it came from the trusted Test App Build Cache workflow/run/repository. Validate trusted producer identity and run provenance before consuming the artifact, cover name-collision/untrusted-producer cases, and fall back safely otherwise.

GitHub now reports the branch mergeable; fresh exact-head CI is queued/running, but this review is not waiting for CI and the provenance finding independently blocks readiness.

[thymikee]

The latest iOS Smoke failure exposed a real Apple runner bug rather than a reason to add another E2E fallback.

• alert get correctly found the deep-link confirmation and returned Cancel, Open, and the Open in “Agent Device Tester”? message. The E2E now asserts that actual public CLI response shape instead of assuming the separate typed system-runtime envelope.
• alert accept did not classify Open as affirmative, so the next command would have failed even if the response assertion were weakened. The Apple runner now treats the exact Open label as an accept action.
• Added a focused XCTest that was red before the fix and green after it, and included it in the targeted iOS CI regression list.
• No snapshot click, retry, redelivery, or selector workaround was added.

Local evidence at 288748332: focused XCTest red → green, XCUITest build green, actionlint/format/static coverage green, and the fix-specific strict Fallow audit clean. Fresh exact-head iOS Smoke is now required to prove the live dialog accept and catalog navigation path.

[thymikee]

Re-reviewed exact head 2887483. The deep-link confirmation delta is clean: Open is classified through the existing affirmative alert path, the targeted XCTest pins that behavior, and the fixture E2E now asserts the actual runner response shape before exercising alert accept. Static, unit, integration, coverage, fallow, format, and Swift compatibility checks are green; iOS Smoke is still running the decisive targeted and live checks. Code review is clean and ready for human review.

[thymikee]

Exact-head CI evidence at 2887483:\n\n- iOS Release built and published the canonical trusted fixture artifact successfully in 17m50s.\n- iOS Smoke fetched that artifact successfully; no inline fixture build was used.\n- Targeted Apple runner regressions passed, including testAlertAcceptTreatsOpenAsAffirmative.\n- Public CLI preflight and Settings replay passed.\n- Fixture-backed iOS simulator E2E passed in 1m50s, including the deep-link confirmation alert accept and exact catalog destination.\n- Host-focus preservation and artifact upload passed.\n- Android Release built and published its canonical fixture artifact successfully; its cold Gradle build completed in 25m52s.\n\nAll exact-head PR checks are green. Review is clean with no outstanding actionable comment.