Skip to content

test: fix Android observability scenario to Android contracts - #1531

Merged
thymikee merged 1 commit into
mainfrom
claude/elastic-shirley-ca1e0d
Jul 31, 2026
Merged

test: fix Android observability scenario to Android contracts#1531
thymikee merged 1 commit into
mainfrom
claude/elastic-shirley-ca1e0d

Conversation

@thymikee

Copy link
Copy Markdown
Member

What

Fixes full:observability-artifacts (from #1482/#1484) so it actually passes on a live Android emulator. The scenario had never executed end-to-end: both nightly Android Full Emulator Suite runs since merge failed on adb infra before the suite ran, and a live run dies deterministically at its first perf assertion. Fixing that surfaced four more latent failures, each asserting iOS or remote-daemon behavior that Android live runs don't have:

  • perf metrics: assert totalPssKb > 0 — Android's required adb-shell-dumpsys-meminfo field (src/platforms/android/perf.ts) — instead of the Apple-only residentMemoryKb.
  • presses: Android snapshots contain only on-screen nodes (unlike iOS AX), and the Quick-actions card is below the fold, so press id="home-open-catalog" / home-open-settings never matched. Added scroll down 0.7 reveals (the sibling scenarios' pattern — scroll down 1 starts inside the bottom nav zone and moves nothing) and a scroll top restore before waiting on the home title, since home scroll position persists across tab switches.
  • batch get: on Android the home-title container resolves to a child's text (the subtitle), unlike iOS's container label. Retargeted to id="dismiss-notice", a node that owns its text.
  • events timeline: the assertion expects snapshot in the timeline, but a standalone run under AGENT_DEVICE_ANDROID_E2E_SCENARIOS never issues one (nightly relied on other scenarios in the shared session). The scenario now captures one itself.
  • artifact inventory: screen-recording entries only exist for remote clients — artifacts without a client localPath are never tracked (src/daemon/request-finalization.ts), and only trace stop has an ?? outPath fallback. This would have failed in CI too. The scenario now asserts the local-client contract (trace-log present, downloadable, download consumes the entry); the MP4 itself is still asserted playable from the recording block. Also fixed an eager await response.text() in an assert message that consumed the download body before arrayBuffer().

Why

The nightly full suite should go green on this scenario once the adb infra issue clears, instead of tripping over never-executed iOS-copied assertions one at a time.

Reviewer notes

  • Validated live: two consecutive green runs of the full tier (AGENT_DEVICE_ANDROID_E2E=1 … AGENT_DEVICE_ANDROID_E2E_SCENARIOS=full:observability-artifacts) on a dedicated Pixel 9 Pro XL API 37 emulator (~35s each), plus lint, typecheck, and the catalog coverage contract test.
  • Two product issues found during diagnosis are deliberately not addressed here (follow-ups flagged separately): downloading a local trace inventory entry deletes the user's trace file (trace/recording tracking asymmetry), and the Android helper's persistent-session disable is sticky for the daemon lifetime, degrading to collision-prone one-shot instrumentation.

The full:observability-artifacts scenario (#1482/#1484) had never executed
end-to-end: both nightly Android Full Emulator Suite runs since merge died
on adb infra before the suite ran, and a live run fails deterministically
at its first perf assertion. Fixing that revealed four more latent
failures, each written against iOS or remote-daemon behavior the Android
live run does not have. Validated with two consecutive green full-tier
runs on a dedicated Pixel 9 Pro XL emulator.

- perf metrics: assert totalPssKb (Android's required meminfo field)
  instead of the Apple-only residentMemoryKb.
- presses: reveal the Quick-actions card with scroll steps before
  pressing home-open-catalog/home-open-settings — Android snapshots only
  contain on-screen nodes — and restore scroll top before waiting on the
  home title, since scroll position persists across tab switches.
- batch get: target id="dismiss-notice" (a node that owns its text);
  Android resolves the home-title container to a child's text (the
  subtitle), unlike iOS's container label.
- events: run an explicit snapshot so the timeline assertion holds when
  the scenario runs standalone under AGENT_DEVICE_ANDROID_E2E_SCENARIOS.
- artifacts: assert the local-client contract — trace-log tracked,
  downloadable, consumed; screen-recording inventory entries only exist
  for remote clients (artifacts without a client localPath are never
  tracked). Also stop consuming the download response body in the assert
  message before arrayBuffer() reads it.
@github-actions

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.91 MB 1.91 MB -384 B
JS gzip 613.4 kB 613.3 kB -147 B
npm tarball 731.5 kB 731.3 kB -154 B
npm unpacked 2.57 MB 2.57 MB -384 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.3 ms 29.0 ms +0.7 ms
CLI --help 61.8 ms 60.5 ms -1.3 ms

Top changed chunks: no changes in the largest emitted chunks.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed exact head 560e9aeafd687ffcb29951d96665964e2605578f. Clean review: the scenario now asserts Android’s real contracts—PSS memory, visible-only snapshot navigation, selector-owned text, standalone event generation, and local-client artifact tracking—without changing production behavior or weakening the MP4/trace evidence. Each correction is revert-sensitive, and the PR includes two consecutive successful full-tier Pixel 9 Pro XL API 37 runs.

Completed CI checks are green; Android/iOS smoke, Coverage, and FreeRange are still running with no confirmed failures. Code review is clean and this is ready for human review under the CI-in-progress policy. Residual risk: I did not independently rerun the live device scenario, and no separately authorized cross-vendor review was performed.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 31, 2026
@thymikee
thymikee merged commit e669676 into main Jul 31, 2026
27 checks passed
@thymikee
thymikee deleted the claude/elastic-shirley-ca1e0d branch July 31, 2026 17:48
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-31 17:49 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant