Skip to content

fix(test): give the iOS scroll search a capture-sized wait budget - #1514

Merged
thymikee merged 1 commit into
mainfrom
fix/ios-smoke-wait-capture-stall
Jul 31, 2026
Merged

fix(test): give the iOS scroll search a capture-sized wait budget#1514
thymikee merged 1 commit into
mainfrom
fix/ios-smoke-wait-capture-stall

Conversation

@thymikee

Copy link
Copy Markdown
Member

Fixes the iOS Smoke Tests lane, red on main for the last three runs.

The failure

✖ live iOS simulator fixture E2E
  AssertionError: step: wait for id="automation-longpress" after scroll (attempt 4)
  command: agent-device wait id="automation-longpress" 1000 --platform ios --udid … --json
  "message": "wait timed out for selector: id=\"automation-longpress\"",
  "details": { "reason": "wait_capture_stalled", "captureStalled": true, "timeoutMs": 1000 }
  scenario: smoke:automation-input
      at assertElementTextAfterScrolling (…/ios-simulator-e2e/live-assertions.ts:28:21)

ios.yml on main: 209ac83df ❌, 1fc916918 ❌, 6067b9a48 ❌, 32b9db2d7 ⚪ cancelled, 0e51007b0 ⚪ cancelled, e70fac7b4 ✅ last green.

Why it started failing

wait_capture_stalled is not "element not found". createWaitPolling only classifies a timeout as capture-stalled when the capture that timed out was the first one — the one handed the wait's entire budget (src/commands/interaction/runtime/wait-polling.ts:40-56). So the message is precisely: one snapshot of the current surface did not complete within 1000 ms.

assertElementTextAfterScrolling has budgeted 1000 ms per attempt since #1408, but until recently it was only reached from live-full-scenarios.ts. #1484 rewrote the smoke automation scenario to route automation-press and automation-longpress through it, replacing a fixed scroll down --pixels 120 plus is visible checks. That put the tight budget on the smoke lane, where a single capture of that surface exceeds it on a loaded runner.

The budget is also an outlier among its own neighbours — every other wait in that scenario allows far more:

wait budget
label="Settings" 10000 ms
alert wait 5000 ms
wait text "Automation lab" 2500 ms
scroll search (this helper) 1000 ms

Because the stall repeats on every attempt, the loop scrolls three times on no evidence and then fails — the element's actual visibility is never established either way.

The change

Two edits to assertElementTextAfterScrolling, both in test/integration/ios-simulator-e2e/live-assertions.ts:

  1. Per-attempt budget 1000 ms → 2500 ms, matching the nearest sibling wait, so one capture of the surface fits inside it.
  2. A stalled capture no longer consumes a scroll attempt. A stall means the surface was never read, so scrolling on it moves the surface for a reason unrelated to visibility. Two stall retries absorb a slow runner. A genuine absence is unaffected — it still spends attempts, still scrolls, and still fails.

The terminal assertion now includes the last wait's JSON, so the next failure distinguishes "stalled" from "genuinely never appeared" without needing the raw job log.

This is deliberately confined to the test harness. I found no product-side regression on the capture path between the last green commit and now — the only non-test changes there in that window are a dead-export deletion in src/snapshot/snapshot-processing.ts and Maestro-specific adapters that plain wait does not use.

Verification

pnpm lint ✅ · pnpm typecheck ✅ · pnpm exec oxfmt --check ✅ on the changed file · smoke-ios-simulator-coverage.test.ts 12/12 ✅ (the device-free contract guarding these scenarios).

I could not execute the live simulator lane — no macOS, no Xcode, and this fix is by nature only exercised against a real simulator. CI on this PR is the verification that matters; the iOS lane running green here is the pass criterion.

One caveat worth stating plainly: raising a timeout makes a lane that stalls occasionally pass, but if a capture of this surface routinely takes over ~2.5 s that is a product-side problem this PR would only be masking. The wait_capture_stalled diagnostic added to the final assertion exists so that case is visible rather than silent.


Generated by Claude Code

`assertElementTextAfterScrolling` waited 1000ms per attempt. The wait's whole
budget goes to its first capture, so that budget has to cover one snapshot of
the current surface; on a loaded simulator it does not, and every attempt fails
with `wait_capture_stalled` instead of reporting the element as off-screen.

This broke the iOS Smoke Tests lane on main. #1484 moved the smoke automation
scenario onto this helper for `automation-press` and `automation-longpress`;
before that it was only reached from the full lane, so the tight budget went
unnoticed. Every sibling wait in the same scenario already budgets 2500ms or
more (`label="Settings"` 10000, `alert wait` 5000, `wait text` 2500).

Two changes:

- Raise the per-attempt budget to 2500ms, matching the nearest sibling.
- Stop spending a scroll attempt on a stalled capture. A stall means the
  snapshot never came back, so the surface was never read — it is not evidence
  the element is off-screen, and scrolling on it moves the surface for an
  unrelated reason. Two stall retries absorb a slow runner; a genuine absence
  still consumes attempts and still fails.

The final assertion now carries the last wait's JSON, so a future failure says
whether it stalled or genuinely never found the element.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
@github-actions

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.91 MB 1.91 MB 0 B
JS gzip 612.3 kB 612.3 kB 0 B
npm tarball 729.8 kB 729.8 kB 0 B
npm unpacked 2.56 MB 2.56 MB 0 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.1 ms 29.8 ms +1.7 ms
CLI --help 58.7 ms 60.6 ms +1.9 ms

Top changed chunks: no changes in the largest emitted chunks.

@thymikee

Copy link
Copy Markdown
Member Author

Clean review at af2810cd8a9b9adb8cc146c850f3e966caa667e4. The confirmed main failure was a first-capture stall, and this test-harness-only change gives that capture a realistic 2.5s budget without consuming scroll attempts on an unread surface. Non-stalled absence still spends attempts, scrolls, and fails, while the terminal assertion now preserves the last structured wait error.

No code findings. All completed checks are green; the fixture-backed iOS simulator smoke—the meaningful regression validation for this change—is still in progress and should be confirmed before merge. Ready for human review pending that result.

Residual risk: no separately authorized cross-vendor review was performed.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 31, 2026
@thymikee

Copy link
Copy Markdown
Member Author

Validation follow-up: the fixture-backed iOS simulator Smoke Tests job completed successfully in 12m42s. All exact-head checks are now green, so the clean review is fully merge-ready.

@thymikee
thymikee merged commit e3cdbcd into main Jul 31, 2026
27 checks passed
@thymikee
thymikee deleted the fix/ios-smoke-wait-capture-stall branch July 31, 2026 07:10
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-31 07:11 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants