Skip to content

feat: add WebView accessibility lab - #1397

Merged
thymikee merged 3 commits into
mainfrom
agent/webview-accessibility
Jul 25, 2026
Merged

feat: add WebView accessibility lab#1397
thymikee merged 3 commits into
mainfrom
agent/webview-accessibility

Conversation

@thymikee

@thymikee thymikee commented Jul 24, 2026

Copy link
Copy Markdown
Member

Summary

Add a react-native-webview accessibility lab to Agent Device Tester with a deterministic semantic fixture and exploratory targets for Callstack, Vercel, Hacker News, Reddit, W3C Forms, and Wikipedia.

Teach iOS interactive snapshot presentation to recognize typed and legacy WebView roots and safely project WebKit's direct Other -> StaticText wrapper pairs as text or headings. Preserve genuine Other landmarks when XCTest does not expose stronger semantics, keep raw snapshots unchanged, and document scoped WebKit subtrees as an XCTest limitation.

Before / after accessibility snapshots

These focused excerpts omit refs and geometry. The first replays the WebKit tree shape captured during iOS testing through main and this PR.

WebKit wrapper pattern

Before (main)After (this PR)
[application] "Example"
  [webview] "Example page"
    [other] "Example page"
      [other] "Welcome"
      [other] "Introduction"
      [link] "Read more"
[application] "Example"
  [webview] "Example page"
    [heading] "Welcome"
    [text] "Introduction"
    [link] "Read more"

The document-title wrapper is removed, while the heading, paragraph, and link remain in their semantic order.

Tester app fixture

The deterministic react-native-webview fixture reproduced the community report on iOS:

Before (main)After (this PR)
[other] "WebView semantic fixture"
[other] "Ordinary paragraph text should not
be categorized as other."
[other] "Account form"
[heading] "WebView semantic fixture"
[text] "Ordinary paragraph text should not
be categorized as other."
[heading] "Account form"

The conversion is scoped to direct Other -> StaticText pairs below a WebView. Adversarial fixtures confirm that structural landmarks such as [other] "main", same-label nested action wrappers, and equivalent pairs outside a WebView stay unchanged.

Validation

  • pnpm format:check
  • pnpm typecheck
  • pnpm lint
  • Focused Vitest suite: 60 tests passed
  • pnpm build:xcuitest
  • pnpm check:fallow --base origin/main
  • Full unit suite: 4,375 tests passed; one host-load slow gate passed in isolation
  • Provider integration contention cases passed in isolation
  • Live iOS verification on the fixture, Callstack, Vercel, Hacker News, Reddit, W3C Forms, and Wikipedia
  • Live Android verification of the deterministic fixture through the Android helper backend
  • Adversarial reviews completed, including claude -p

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.83 MB 1.82 MB -2.9 kB
JS gzip 583.8 kB 582.8 kB -1.1 kB
npm tarball 698.0 kB 697.0 kB -1.0 kB
npm unpacked 2.45 MB 2.44 MB -2.8 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.3 ms 27.8 ms +0.6 ms
CLI --help 57.3 ms 57.7 ms +0.4 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/runtime.js -1.1 kB -450 B
dist/src/session.js -1.3 kB -449 B
dist/src/internal/daemon.js +1.3 kB +397 B
dist/src/selector-runtime.js -464 B -120 B
dist/src/cli-help.js -316 B -116 B

@thymikee
thymikee marked this pull request as ready for review July 24, 2026 18:19
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed exact head f3ac9ef. P2: web.ts classifies any WebView-descended Other with a same-label direct StaticText as text, without proving it is a leaf WebKit wrapper. A legitimate labelled landmark/container such as Other "News" containing StaticText "News" plus a Link also matches; the presenter rewrites the container to text and duplicate suppression removes the actual text child. That contradicts the stated guarantee to retain genuine Other landmarks and can turn a large container into the apparent text target. Restrict the projection to a proven wrapper shape (for example, exclude nodes with semantic sibling/descendant content) and add this direct-child adversarial fixture. The current test covers only a nested same-label StaticText under a Button and misses this case. Current checks are green except one Smoke Tests job still in progress.

@thymikee

Copy link
Copy Markdown
Member Author

Addressed in 4c65a10. WebKit text/heading projection now requires a proven leaf wrapper shape: the Other must have exactly one direct same-label StaticText child, and that text node must itself be a leaf. Any additional or nested content keeps the container as Other. Added the adversarial Other "News" -> StaticText "News" + Link "Read more" fixture; focused presentation tests pass (33/33). The affected run’s unrelated host-load timeout files also pass in isolation (63/63).

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed exact head 4c65a10. The prior WebView container false-positive is fixed: text/heading projection now requires an exact leaf wrapper with one direct same-label StaticText child and no descendants. The added regression covers the previously missing Other "News" → StaticText "News" + Link "Read more" shape and confirms the landmark remains Other with its semantic child preserved. All current checks are green. No further findings.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 25, 2026
@thymikee
thymikee merged commit 11e0a1f into main Jul 25, 2026
26 checks passed
@thymikee
thymikee deleted the agent/webview-accessibility branch July 25, 2026 09:40
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-25 09:40 UTC

thymikee added a commit that referenced this pull request Jul 25, 2026
* origin/main:
  docs(adr): rules-first ADR restructure + ADR 0017 proposal (unified event journal) (#1399)
  feat: add first-class Vega VVD TV support (#1396)
  fix(replay): preserve cwd scope for opened sessions (#1401)
  docs: restructure AGENTS.md and CONTEXT.md for progressive disclosure (#1402)
  fix(cli): compact stale device status (#1388)
  feat: add WebView accessibility lab (#1397)
  feat: parameterize sensitive recorded inputs (#1369)
  fix(daemon): keep an active replay session's daemon alive over the CLI path (#1390)

# Conflicts:
#	docs/adr/0012-interactive-replay.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant