Skip to content

fix(test): repair Android replay fixtures against live device reality - #1538

Merged
thymikee merged 2 commits into
mainfrom
fix/android-live-fixture-drift
Aug 1, 2026
Merged

fix(test): repair Android replay fixtures against live device reality#1538
thymikee merged 2 commits into
mainfrom
fix/android-live-fixture-drift

Conversation

@thymikee

@thymikee thymikee commented Aug 1, 2026

Copy link
Copy Markdown
Member

Three Android fixture defects from the #1482/#1484 full-tier suite, none of which ever executed in CI (both nightlies since failed on adb infra before the suite ran). All three verified live during the #1525 evidence arc on a fresh API 36 emulator with pixel_7 geometry and a Release fixture APK:

  • 01-navigation-scroll.ad clicked label=Catalog, but the expo-router NativeTabs cart badge leaks '0 new notifications' into the tab's content description even while hidden, and unselected native tabs expose no child text node — exact match can never hit. Target the composed label the device actually exposes (deterministic at fixture start: cart is 0 after --relaunch). The badge does not leak on iOS, so the iOS twin keeps label="Catalog".
  • checkout-form-android.ad opened by iOS display name 'Agent Device Tester'; Android open resolves packages (the APK label is 'Agentdevicelab'), so APP_NOT_INSTALLED was guaranteed. Use the package id, matching gesture-lab-android.ad.
  • gesture-lab-android.ad aimed every gesture at y=700, above the gesture card (targets span y754–1329 on pixel_7 geometry; the home screen gained content above the card since authoring). Re-aim pans inside the exact-two-pointer zone, flings on the image clear of that zone, and pinch/rotate/transform at the card center. Verified: full suite passes 2/2 via the public test command (20 + 32 steps replayed).

Validation

Live evidence from the #1525 arc (2026-07-31, documented in its PR comment): consolidated suite green 2/2 on emulator-5556 (Pixel_7_CI, API 36, freezer disabled), checkout 20/20 + gesture-lab 32/32 steps replayed through the public test command, JUnit non-empty. These fixture fixes are also a prerequisite for the test-app:replay:android live-evidence requirement on #1536.

Refs #1478

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.92 MB 1.92 MB 0 B
JS gzip 615.5 kB 615.5 kB 0 B
npm tarball 733.5 kB 733.5 kB 0 B
npm unpacked 2.57 MB 2.57 MB 0 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 26.5 ms 28.2 ms +1.6 ms
CLI --help 57.2 ms 59.6 ms +2.4 ms

Top changed chunks: no changes in the largest emitted chunks.

@thymikee

thymikee commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Reviewed head 5db5831. The three fixture corrections match the cited live Android evidence, and all CI is green, but one portability gap remains:

  • P2 — constrain or document the gesture fixture's validated geometry. examples/test-app/replays/gesture-lab-android.ad now hard-codes every target around y=1040, based on the API 36 Pixel 7 run. However, pnpm test-app:replay:android selects only --platform android, the script declares only kind=emulator, and examples/test-app/README.md presents that command as the general local Android suite. Any booted emulator can therefore receive these Pixel-7-specific coordinates; on another size/density the gesture card can move and the test will hit the wrong view. Constrain the route to the validated geometry, make target positioning geometry-independent, or at minimum document the strict Pixel 7/API 36 prerequisite and a reproducible device-selecting invocation.

The package-id fix and Android composed Catalog label are otherwise sound. No ready-for-human label yet.

thymikee and others added 2 commits August 1, 2026 12:38
Three Android fixture defects from the #1482/#1484 full-tier suite, none of
which ever executed in CI (both nightlies since failed on adb infra before
the suite ran). All three verified live on a fresh API 36 emulator with a
pixel_7-geometry AVD and a Release fixture APK:

- 01-navigation-scroll.ad clicked label=Catalog, but the expo-router
  NativeTabs cart badge leaks '0 new notifications' into the tab's content
  description even while hidden, and unselected native tabs expose no child
  text node - exact match can never hit. Target the composed label the
  device actually exposes (deterministic at fixture start: cart is 0 after
  --relaunch). The badge does not leak on iOS, so the iOS twin keeps
  label="Catalog".

- checkout-form-android.ad opened by iOS display name 'Agent Device
  Tester'; Android open resolves packages (the APK label is
  'Agentdevicelab'), so APP_NOT_INSTALLED was guaranteed. Use the package
  id, matching gesture-lab-android.ad.

- gesture-lab-android.ad aimed every gesture at y=700, above the gesture
  card (its targets span y754-1329 on pixel_7 geometry; the home screen
  gained content above the card since authoring). Re-aim pans inside the
  exact-two-pointer zone, flings on the image clear of that zone, and
  pinch/rotate/transform at the card center. Verified: full suite passes
  2/2 via the public test command (20 + 32 steps replayed).

Refs #1478
…etry

The re-aimed coordinates are validated on CI's profile (pixel_7 1080x2400
@420); any booted emulator can receive them via test-app:replay:android, so
the fixture and README now say which geometry the numbers mean and what a
mismatch failure looks like. The checkout twin is selector-driven and
unconstrained.

Co-Authored-By: Claude <noreply@anthropic.com>
@thymikee

thymikee commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Addressed in 071326034 (rebased onto current main): took the document half of "constrain or document" — the gesture fixture's header now names the validated geometry (CI's pixel_7, 1080×2400 @ 420 dpi, with the measured gesture-card span) and states that a mismatched emulator fails honestly at the canary waits rather than silently passing; the README's test-app:replay:android section says the same and distinguishes the selector-driven checkout twin, which is size-independent. Parse check green for both files.

I stopped short of a hard constraint (e.g. asserting wm size at open) because the .ad grammar has no device-geometry predicate and inventing one for a fixture would be scope creep — but if you'd rather gate than document, a wait-able geometry canary rendered by the test app itself would be the clean mechanism and I can file that as a follow-up.

Generated by Claude Code

@thymikee
thymikee force-pushed the fix/android-live-fixture-drift branch from 5db5831 to 0713260 Compare August 1, 2026 10:39
@thymikee

thymikee commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Re-reviewed exact head 071326034. The prior portability finding is resolved: the Android gesture fixture and local-suite README now pin the validated pixel_7, 1080x2400 @ 420 dpi geometry, explain how mismatches surface through the positive gesture canaries, and distinguish the size-independent selector-driven checkout replay.

The corrected fixture bytes remain identical to the cited live Android run, and each regression edit is guarded by a positive state/count wait or deterministic launch/selector failure. All exact-head CI is green, including replay provenance, integration, coverage, Android smoke, and release builds. No blocking findings — ready for human merge review.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 1, 2026
@thymikee
thymikee merged commit 8a6ddbc into main Aug 1, 2026
27 checks passed
@thymikee
thymikee deleted the fix/android-live-fixture-drift branch August 1, 2026 11:50
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-01 11:50 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant