Skip to content

refactor(replay-test): extract packages/replay-test (#1478 P3b) - #1525

Merged
thymikee merged 16 commits into
mainfrom
p3b/extract-replay-test-package
Jul 31, 2026
Merged

refactor(replay-test): extract packages/replay-test (#1478 P3b)#1525
thymikee merged 16 commits into
mainfrom
p3b/extract-replay-test-package

Conversation

@thymikee

@thymikee thymikee commented Jul 31, 2026

Copy link
Copy Markdown
Member

Draft — code complete, waiting on live-device evidence. Every item on the P3b brief has landed. The one thing between this and ready is a run of the iOS/Android journeys against the consolidated harness, which the authoring environment has no devices for.

What this does

P3a neutralized the values crossing the scheduler seam. P3b removes the remaining ambient authority, splits host work out of the scheduler, and physically extracts the package.

The seams

concern before after
manifest device types src/replay/script.ts (the .ad engine) @agent-device/kernel/device — resolves to the same types, only the import direction changed
progress emitRequestProgress reading a request-global AsyncLocalStorage, 8 call sites injected emitProgress, narrower than RequestProgressSink — suite/test events only, never CommandProgressEvent
cancellation predicate isRequestCanceled(requestId), 5 call sites bound isCanceled(); the scheduler never names a daemon request id
cancellation binding registerRequestAbort / markRequestCanceled / clearRequestCanceled / parent-abort relay host capability with two verbs: cancel() on timeout, release() when settled
diagnostics utils/diagnostics.ts request-global scope injected emitDiagnostic
discovery both engines + resolveReplayFormat + path expansion host discoverSources; the scheduler keeps filtering policy
sharding listDeviceInventory, allowlists, simulator set paths, CommandFlags host resolveShardTargets; the scheduler keeps distribution policy
suite request/result DaemonRequest in, DaemonResponse out neutral ReplayTestSuiteRequest in, tagged ReplayTestSuiteOutcome out

Two are worth review attention:

The platform tag is what made discovery format-neutral. The filter used to ask resolveReplayFormat(...) === 'maestro' to decide whether a missing platform disqualified a source. The manifest now carries declared | caller-bound | unspecified, so the scheduler asks how the platform is determined, never which engine produced it. caller-bound is what Maestro looks like from the scheduler's side and the format cannot be recovered from it.

The timing trace stayed private. The host writes video-lifecycle events into the same trace the scheduler owns. Rather than export a writer from the façade, each attempt hands the host an appendTimingEvent closure — the format stays internal and the authority is scoped to that one attempt's trace.

The extraction

packages/replay-test/src/internal/ holds the scheduler, attempt runtime, discovery policy, sharding distribution, artifacts and neutral types, with one export at ..

Host work stayed in daemon adapters: source inspection, shard device binding, shard-flag parsing, and artifacts-dir home expansion — which is why the package resolves paths without SessionStore.

R10 retargeted to packages/replay-test/src/; the zone is ranked alongside maestro; R11 confirms zero root-src imports from the package. The dead 'src/maestro/' entry in forbiddenTargetRoots (a directory that no longer exists) is replaced with packages/maestro/.

Live harness consolidation

Both live journeys invoked the public test command and then re-derived the same value-contract assertions by hand — suite totals, per-script status, replay counts, non-empty JUnit. They had already drifted: iOS iterated with readReplayCommands inline, Android cast data.tests at the call site.

The shared helper owns that boundary and takes the caller's runStep rather than binding a context type, so it is not a platform-configured runner and cannot template a platform's journey. Scripts, retry policy (iOS 2, Android none — itself a claim worth keeping), expected commands and behavioral evidence all stay with the callers, which keep every verify* call they had. 67 lines removed, 15 added.

Tests

Coverage moved with the code rather than thinning:

  • Discovery tests split along the seam they now cross. Ordering, traversal and format routing are pinned host-side against real .ad/Maestro files; filtering policy is pinned in the package against fake sources.
  • Cancellation became two tests. The package asserts the scheduler's obligation — cancel exactly once on timeout, always release on settle — against a recording binding. A new daemon test pins the adapter's half: registry entries, the parent-abort relay, abort-at-bind when the parent is already canceled, and detach on release.
  • Both reporter characterizations pass unmodified at every commit. They assert exact session strings like default:test:suite-reporter:1-02-retry:attempt-1, the strongest available evidence that ids and reporter values are byte-identical.

Residual risk

The live suites have not been run against the consolidated harness. The authoring environment has no iOS or Android targets. Typecheck, lint, oxfmt and the layering guard pass, and the change is structural, but "behavior unchanged" is inference until it runs on real devices. This is the gate on undrafting.

One pre-existing test flaw, not introduced here

session-test-runner "binds each replay script to its declared platform metadata" writes two files into a temp directory and assumes discovery returns them in creation order. Directory expansion deliberately preserves filesystem order (glob expansion sorts) to match Maestro, and that behavior is explicitly pinned by "preserves Maestro directory filesystem order", which mocks opendirSync to prove it. So the ordering contract is correct and intentional; the runner test's assumption about enumeration order is what is unsound. It passes on CI, where small directories usually enumerate in creation order, and fails on filesystems that don't — identically on clean main. The fix belongs in that test, not in replay-source-discovery.ts, and is out of scope here.

Bundle size

Earlier revisions of this PR reported a decrease; that was base drift, not this change. Against current main, P3b costs +2.8 kB raw / ~+780 B gzip, concentrated in dist/src/session.js — the per-attempt cancellation object, the manifest conversion, the neutral request/outcome translation and the capability threading. For calibration, a test-only PR (#1526) moved +88 B gzip, so that report's noise floor is not small.

Refs #1478


Generated by Claude Code

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.91 MB 1.92 MB +2.8 kB
JS gzip 613.3 kB 614.0 kB +773 B
npm tarball 731.3 kB 732.1 kB +779 B
npm unpacked 2.57 MB 2.57 MB +2.8 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 25.3 ms 25.4 ms +0.1 ms
CLI --help 52.6 ms 52.6 ms +0.0 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/session.js +2.8 kB +773 B

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed exact head 923a2e1e7d4c4e96837bfa08ce577e0c6bd417fc: the current P3b seam work is clean. Manifest device types remain byte-equivalent while reversing the engine dependency; progress and cancellation are injected as narrow host-owned capabilities; the attempt runtime receives only the three ports it uses; and removing resolveRequestTrackingId is safe because the interpolated attempt ID always contains :test: and can never reach that helper’s empty-ID branch. Existing reporter/session characterizations pin the resulting strings. Exact-head CI and all platform smokes are green, with no device-facing behavior change requiring live evidence.

This is not merge-ready yet: the draft explicitly leaves the package move, daemon-adapter split, R10 retarget, sharding/discovery split, and harness consolidation unfinished. Keep it draft and request final re-review once the titled extraction is complete.

Residual risk: no separately authorized cross-vendor review was performed; Claude authorship is not an independent review.

@thymikee
thymikee force-pushed the p3b/extract-replay-test-package branch from 923a2e1 to d598189 Compare July 31, 2026 13:13
Comment on lines +114 to +119
// The scheduler mints attempt identity itself (#1478 P3b). This used to be wrapped in
// `resolveRequestTrackingId`, whose only job is to substitute a generated id for an empty
// one — and this template always contains `:test:`, so it is never empty and the wrapper
// always returned it unchanged. Dropping the dead call removes a `request/cancel.ts`
// import from the scheduler without altering a single produced id.
return `${requestId ?? suiteInvocationId}${shardPart}:test:${caseIndex + 1}:${path.basename(filePath)}:attempt:${attemptIndex + 1}`;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get rid of long comment, fix the code

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cbb120d80 — the id builds from a segment list joined on ':', so the shape reads without prose. Output is byte-identical; the reporter characterizations assert exact session/attempt strings and pass unmodified.

Applied the same standard to four other docblocks in this PR that narrated what the code used to do rather than what it does. Kept the durable "why" (which side of the seam owns what); dropped the migration history, since git carries it.


Generated by Claude Code

@thymikee

thymikee commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

Reviewed current head 78ae0bc5a4c1a782dfe48079c21902155705b41a. The shard-device binding extraction is clean: host inventory/selector mechanics moved behind a neutral resolver while scheduler policy remains isolated, and I found no behavior regression in this delta. Static, Android, Linux, macOS, integration, coverage, and Web smoke checks are green; iOS smoke is still running.

This remains an intentionally incomplete draft, not merge-ready yet. The remaining #1478 work still includes the actual packages/replay-test extraction, daemon façade/outcome mapping, R10 retargeting, attempt-id/host-label ownership, utility splits, and live harness consolidation. Please also refresh the PR body/checklist so it accurately reflects what this head has completed and what remains.

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-31 18:16 UTC

claude added 13 commits July 31, 2026 14:50
… kernel

`session-test-types.ts` reached `ReplayScriptMetadata['platform']` and
`['target']` through `replay/script.ts` — the native `.ad` engine. A
format-neutral scheduler must not name an engine module, and P5 relocates that
engine into `packages/ad-replay` regardless, so the import had to go before the
scheduler can move.

Both members already resolve to neutral kernel types
(`Exclude<PlatformSelector, 'web'>` and `DeviceTarget` from
`@agent-device/kernel/device`), so this re-sources them directly and the
manifest shape is unchanged. Only the import direction differs.

First increment of P3b; the scheduler still has request-global, engine and
daemon imports to port before the physical move.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
…request global

The scheduler called `emitRequestProgress` in eight places, which reads a sink
out of a request-global `AsyncLocalStorage`. That is ambient authority a
format-neutral scheduler cannot hold once it lives in `packages/replay-test`,
and #1505 recorded it as a shrink-only R10 entry.

The host now injects the capability through the existing
`ReplayTestRuntimeDependencies` seam established in P3a, so no new seam is
invented. `session-replay.ts` supplies `emitProgress: emitRequestProgress`;
`src/request/progress.ts` keeps the sink and its AsyncLocalStorage binding for
every other caller.

The port is deliberately narrower than `RequestProgressSink`: it accepts only
`ReplayTestSuiteProgressEvent | ReplayTestProgressEvent`, so the scheduler is
not handed the ability to emit `CommandProgressEvent`.

Authority narrows again one hop down: `runReplayTestAttempt` spread the whole
dependency bag but uses three of its members and never publishes progress, so
it now takes `Pick<..., 'runReplay' | 'cleanupSession' | 'finalizeAttempt'>`.
That is why no runtime test fixture needed changing — the attempt runtime never
gained the capability in the first place.

Also drops the last two `replay/script.ts` type references from
`session-test-runtime.ts`, so the engine import is gone from that file too.

Reporter contract preserved: `session-test-reporter-values.test.ts` and
`session-test-reporter-values-maestro.test.ts` both pass unmodified (27 tests
green across the five scheduler suites). Typecheck clean.

Remaining scheduler boundary for P3b: `request/cancel.ts`, `replay/format.ts`,
`replay/script.ts` in discovery, `session-store.ts`, `daemon/types.ts`,
`replay-source-discovery.ts`, `core/dispatch*`, `utils/diagnostics.ts`.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
The scheduler called `isRequestCanceled(requestId)` in five places. That both
reaches a request-global registry and forces the scheduler to name a daemon
request id as the cancellation key — neither survives the move into
`packages/replay-test`.

The host now binds the predicate to its own request and passes
`isCanceled: () => boolean`. The scheduler asks a question it is entitled to
ask and learns nothing about how cancellation is tracked. `shouldStopReplayTestExecution`
takes the capability rather than a request id, so no scheduler function threads
a daemon identifier for this purpose any more.

`session-test-attempt.ts` and `session-test.ts` no longer import
`request/cancel.ts` at all. It remains in `session-test-runtime.ts`, which does
something different — `registerRequestAbort`, `markRequestCanceled` and the
parent-abort relay are cancellation *binding*, which the brief assigns to the
daemon adapter, so that split is its own step.

Behavior preserved: both pinned reporter characterizations pass unmodified,
32/33 across the five scheduler suites. The one failure is the pre-existing
P2/#1506 discovery-ordering regression, unrelated and untouched here.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
…pt ids

`buildReplayTestAttemptRequestId` wrapped its template in
`resolveRequestTrackingId`, pulling `request/cancel.ts` into the scheduler.

That wrapper substitutes a generated id only when its first argument is an
empty string. The template here always contains `:test:`, so it is never empty
and the wrapper always returned it unchanged — the call is unreachable in this
path. Probed all three shapes (explicit request id, suite-id fallback with a
shard, and degenerate empty inputs); every one returns the template verbatim.

Removing it takes `request/cancel.ts` out of discovery without altering a
single produced id. The scheduler mints attempt identity itself, which is what
the brief asks for.

Evidence the ids are byte-identical: the pinned reporter characterizations
assert exact session strings such as
`default:test:suite-reporter:1-02-retry:attempt-1` and pass unmodified —
30 tests green across the reporter, suite and discovery suites.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
…he host

`session-test-runtime.ts` held the last two request-globals in the scheduler:
`request/cancel.ts` (registerRequestAbort, markRequestCanceled,
clearRequestCanceled, plus the parent-abort relay) and `utils/diagnostics.ts`.

These are different in kind from the earlier ports. The brief gives the daemon
adapter the job of mapping an attempt id to daemon request identifiers and
*binding cancellation*, while timeout policy stays scheduler-owned. So the
scheduler now receives a per-attempt capability with exactly two verbs —
`cancel()` on timeout and `release()` when the attempt settles — and every
registry interaction, including `relayReplayTestAbortFromParent`, moved to
`session-replay.ts` next to the rest of the adapter.

Diagnostics became a narrow publish capability for the same reason:
`emitDiagnostic` reads a request-global scope. The level vocabulary is spelled
out at the seam rather than imported, so nothing engine- or daemon-shaped
crosses it.

The runtime fixtures drive the real exported host binding rather than a stub.
They assert cancellation through `isRequestCanceled`, and a stubbed binding
would have kept those assertions passing while proving nothing.

24 tests green across the runtime, suite and both reporter characterizations,
which pass unmodified. Typecheck, lint and oxfmt clean.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
…uler policy

discoverReplayTestEntries expanded paths, read every file, and called both
engines — readReplayScriptMetadata for .ad, inspectMaestroFlow for Maestro —
plus resolveReplayFormat to choose between them. Four imports a format-neutral
scheduler cannot hold.

Inspection is now the host's discoverSources capability. What stays in the
scheduler is the genuinely neutral half: which sources a --platform filter
runs, which it skips and with what message, and the empty-suite error.

The manifest carries exactly the four fields the scheduler consumes (platform,
target, retries, timeoutMs) plus the reporter's title, per the brief's
instruction not to add more without a demonstrated call site.

The platform tag is what removes the last format leak. The filter used to ask
resolveReplayFormat(...) === 'maestro' to decide whether a missing platform was
disqualifying. It now reads a tag: caller-bound means the invocation supplies
the platform, unspecified means the source declared none. Maestro is what
caller-bound looks like from the scheduler's side, and the format cannot be
recovered from it.

Discovery tests drive the real inspection capability, writing actual .ad and
Maestro sources — a stubbed host half would have kept them green while proving
nothing about the composition they exist to pin.

35 tests green across discovery, suite, runtime and both reporter
characterizations, which pass unmodified. The Maestro one is the direct check
that titles still flow, since they now arrive via the manifest.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
…mments

Review feedback on the attempt-id builder: the comment explained a deletion
that git already records, and it sat above an opaque template literal.

The id is now a segment list joined on ':', so its shape is readable without
prose. Output is byte-identical — the reporter characterizations assert exact
session and attempt strings and pass unmodified.

Applied the same standard to four other docblocks in this PR that narrated
what the code used to do rather than what it does. The durable 'why' stays:
which side of the seam owns what, and why the vocabulary is neutral. The
migration history goes, since git carries it and these docblocks will outlive
the migration.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
buildReplayTestShardPlan called listDeviceInventory to discover what to shard
across, and buildReplayTestShardFlags constructed daemon CommandFlags for the
nested request. Inventory enumeration, allowlists, simulator set paths,
explicit --device selectors and the too-few-devices error are host concerns;
what is scheduler-owned is deciding how many shards exist and which entries
each one runs.

The scheduler now receives resolved shard targets through a capability. The
target is neutral: id and name for session labels and progress metadata, plus
platform and target, which are already kernel vocabulary. DeviceInfo no longer
crosses into scheduling.

One behavior note: an explicit --device selector could in principle name a web
target, which is not a shardable device. That is now rejected with INVALID_ARGS
rather than widening the neutral platform vocabulary to carry something the
scheduler can never run. Implicit selection already filtered to mobile.

919 of 920 handler tests pass. The one failure, session-test-runner.test.ts
'binds each replay script to its declared platform metadata', fails identically
on clean origin/main in this container and is unrelated: directory discovery
walks with opendirSync/readSync and directory results are deduped but not
sorted, while glob results are sorted, so suite order is filesystem-dependent.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
Completes the P3b extraction. The scheduler, attempt runtime, discovery
policy, sharding distribution, artifacts and neutral types now live in
packages/replay-test/src/internal/, with one package-root export.

The façade takes a neutral ReplayTestSuiteRequest and returns a tagged
ReplayTestSuiteOutcome. DaemonRequest, DaemonResponse and CommandFlags no
longer reach the scheduler; the adapter translates flags and meta in, and the
outcome back to a daemon response. Eight flags were read by the scheduler and
each became a field it owns.

Host work moved to daemon adapters: source inspection (both engines and format
routing), shard device binding and shard-flag parsing, and artifacts-dir home
expansion, which is why the package can resolve paths without SessionStore.

The one remaining shared concern was the timing trace: the host writes video
lifecycle events into the same trace the scheduler owns. Rather than export a
writer from the façade, each attempt hands the host an appendTimingEvent
closure, so the trace format stays private and the authority is scoped to that
attempt.

Tests mirror the topology. Discovery tests split along the seam they now
cross: ordering, traversal and routing are pinned host-side against real files,
filtering policy is pinned in the package against fake sources. The runtime
tests assert the scheduler's cancellation obligation (cancel once on timeout,
always release) against a recording binding, and a new daemon test pins the
adapter's half — registry entries, the parent-abort relay, and detach on
release — so that coverage moved rather than disappeared.

R10 retargeted to packages/replay-test/src/ and the zone ranked alongside
maestro. R11 confirms zero root-src imports from the package.

914 of 915 handler and package tests pass. The one failure,
session-test-runner 'binds each replay script to its declared platform
metadata', fails identically on clean main here: directory discovery walks with
opendirSync and dedupes without sorting, while globs sort, so suite order is
filesystem-dependent.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
Fallow flagged toReplayTestSuiteRequest at 14 cyclomatic in 18 lines. The
branches were self-inflicted: every req.flags?.x is one, and each optional
field was written as a conditional spread to avoid setting an undefined key.

exactOptionalPropertyTypes is not enabled, so assigning undefined to an
optional field is equivalent and the spreads bought nothing. Destructuring
flags once and extracting two flag readers removes most of the rest.

One correctness note on the simplification itself: the first version used
`artifactsDir && expandHome(...)`, which returns '' for an empty-string flag
where the previous code called expandHome(''). Replaced with an explicit
undefined check so the empty-string path is unchanged.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
Both live journeys invoked the public test command and then re-derived the
same value-contract assertions by hand — suite totals, per-script status,
replay counts, non-empty JUnit. Those are claims about the published suite
result and are identical on every platform, and they had already drifted: iOS
iterated with readReplayCommands inline, Android cast data.tests at the call
site.

The shared helper owns exactly that boundary. It takes the caller's runStep
rather than binding a context type, so it is not a platform-configured runner
and cannot template a platform's journey.

Everything a platform genuinely differs on stays with the caller: which
scripts run, the retry policy (iOS 2, Android none — itself a claim worth
keeping), which commands each script exercises, and the behavioral evidence.
Both callers keep every verify* call they had.

67 lines removed, 15 added.

Residual risk: this container has no iOS or Android devices, so the live suites
could not be executed here. Typecheck and lint pass; the harness needs a run on
real targets before the claim that behavior is unchanged is evidence rather
than inference.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
The test wrote two scripts into a temp directory and assumed discovery would
return them in creation order. Directory expansion deliberately preserves
filesystem order to match Maestro — only glob expansion sorts, and 'preserves
Maestro directory filesystem order' pins that with a mocked opendirSync. So the
ordering contract is correct; this test's assumption about enumeration was not.

It passes on CI, where small directories usually enumerate in creation order,
and fails on filesystems that do not — identically on clean main, where the
platform-to-script binding appears reversed.

Pinning enumeration the way the discovery tests already do keeps the subject
intact (each script binds to ITS declared platform, and session numbering
follows discovery order) without depending on the host filesystem. The fs
import became a default import because vi.spyOn cannot redefine an ESM
namespace export.

915 of 915 handler and package tests now pass here.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
The spy I added restored only on the happy path and asserted on its argument
inside the mock implementation. Either would misfire for anything else sharing
the worker: an assertion thrown from inside fs, or a leaked global opendirSync,
surfaces as a worker crash with no failed test rather than a readable failure.

It now delegates to the real implementation for any directory but this suite's
own, and restores in a finally.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
@thymikee
thymikee force-pushed the p3b/extract-replay-test-package branch from 9922cd1 to 0691345 Compare July 31, 2026 14:55

Copy link
Copy Markdown
Member Author

Live verification attempt — no evidence obtained, staying drafted

Both platforms failed upstream of the replay-suite step, so assertFixtureReplays — the only thing this PR changed in the live journeys — was never reached. The consolidation is neither vindicated nor implicated.

iOS (iPhone 17 Pro, C25DBB5B-9254-4293-A8D5-2785C78DE03A): failed in microphone-permission setup — expected undetermined, got not-requested.

Android (emulator-5554): failed in fixture bootstrap — the Expo dev client stayed on "Fetch development servers"; Automation lab never appeared.

Builds passed and cleanup completed on both.

On the iOS failure

Traced as far as the source allows from an environment without the built fixture:

  • The fixture does not compute that string — AutomationLabScreen.tsx:58 passes permission.status straight through from expo-audio's getRecordingPermissionsAsync().
  • expo-modules-core's permissions type declares exactly 'denied' | 'granted' | 'undetermined'.
  • Nothing in agent-device emits not-requested — checked src/, ios/, android/.

So the runtime returned a status outside the union its own library declares, which points at expo-audio 56 on a newer iOS runtime, or a native/JS skew in the built fixture.

The expectation has deliberately not been widened to accept both spellings. If not-requested is a rename for the same TCC state, updating it is correct and trivial; if it is a distinct state (never asked vs. asked and dismissed), widening would hide a real difference in permission handling. Worth one check on a machine with the fixture installed:

grep -rn "not-requested" node_modules/expo-audio node_modules/expo-modules-core

This blocks all iOS live evidence, so it is on P5's path too, not only this PR's.


Generated by Claude Code

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed exact head 0691345622ff182d301c17b846d66811fc6d71e9. The architecture split is otherwise sound, but this is not ready yet:

  1. P1 — the moved package tests are no longer executed or typechecked. They live under packages/replay-test/test/, while vitest.config.ts includes only packages/*/src/**/*.test.ts; root tsconfig.json does not include package directories, and the package tsconfig includes only src. This skips the moved cancellation/finalize/cleanup ordering, discovery-policy, and artifact tests despite green CI. packages/replay-test/test/session-test-runtime.test.ts also imports non-exported runReplayTestAttempt from @agent-device/replay-test, so it will fail once actually discovered. Put these tests under the package’s included internal test topology (for example src/internal/__tests__) with internal-relative imports, or explicitly configure both Vitest and typechecking. Do not widen the façade just for tests.

  2. P2 — remove the unused runtime façade export replayTestAttemptFailure. It has no external consumer; all uses are internal. P3 calls for a one-function façade, so exposing this helper unnecessarily widens the package boundary.

After fixing these, rerun the exact package tests plus the aggregate gates. The draft also still needs the stated live iOS and Android public test-journey evidence before it can be marked ready.

Review found the moved package tests were neither executed nor typechecked.
They sat under packages/replay-test/test/, but vitest's unit-core lane includes
packages/*/src/**/*.test.ts, and neither the root nor the package tsconfig
covers a top-level test directory. A plain unit-core run discovered zero files
under the package.

That is why they looked green: my earlier runs passed those paths explicitly on
the command line, which masked that the default run skipped them. The count is
the proof — 550 files/4741 tests before, 553/4753 now, and the delta is exactly
the three files and twelve tests that were being skipped.

The runtime test also imported runReplayTestAttempt from the package specifier,
which the facade does not export. It would have failed the moment it was
discovered. It now imports internally, like the rest of the internal tests.

Also removed replayTestAttemptFailure from the facade: zero consumers outside
the package, so exporting it widened the boundary for nothing. P3 asks for a
one-function facade.

553 test files and 4753 tests pass; lint and the layering guard are clean.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8

Copy link
Copy Markdown
Member Author

Both fixed in 2939e1bd7.

P1 — you were right, and it was worse than "skipped". A plain vitest run --project unit-core discovered zero files under packages/replay-test. My earlier green runs passed those paths explicitly on the command line, which masked that the default run never included them. The count is the proof: 550 files / 4741 tests before, 553 / 4753 now — the delta is exactly the three files and twelve tests that were being skipped.

The tests now live in packages/replay-test/src/internal/__tests__/, inside the package's included topology, with internal-relative imports. session-test-runtime.test.ts imports runReplayTestAttempt from ../session-test-runtime.ts rather than the package specifier — and confirming your read, it does pass once actually discovered, so the only thing standing between that import and a failure was non-discovery. The façade was not widened.

P2 — removed. replayTestAttemptFailure had zero consumers outside the package; exporting it widened the boundary for nothing.

Gates on the new head: 553 test files / 4753 tests pass, lint clean, layering guard clean, typecheck clean.

The live iOS/Android evidence is still outstanding — both attempts failed upstream of the replay-suite step, detailed in the previous comment. Staying drafted.


Generated by Claude Code

A scripted edit removed the export line but left a stray blank line; oxfmt was
not re-run on that file afterward, so Lint & Format caught what pnpm lint
alone does not.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8

Copy link
Copy Markdown
Member Author

CI is fully green on fab645fa8 — but that is not the live evidence this PR needs

All 33 checks pass, including both live device lanes. Do not read that as evidence for the harness consolidation. It is not, and the reason is a one-word tier tag.

iOS ran genuinely — ✔ live iOS simulator fixture E2E (169515ms), 13 pass, 0 skipped, on a real simulator. But the replay-suite scenario is registered as:

{ id: 'full:fixture-replays', runner: 'fixtureReplays', tier: 'full' }   // scenarios.ts:31

and the CI lane runs smoke. So the journey executed for nearly three minutes without reaching assertFixtureReplays, the only function this PR changed in the live tests.

Android is the same, stated outright in its log: AGENT_DEVICE_ANDROID_E2E_TIER=smoke, with the scenario list fixture-bootstrap, inventory, automation-system, form-input, keyboard-ime, capture-close — no replay entry.

A passing live journey that never executes the changed function is exactly as uninformative as a skipped one.

What CI did prove

The Android lane, after its E2E, runs:

node src/bin.ts test test/integration/replays/android/01-settings.ad --retries 2 --report-junit …
✓ 01-settings.ad 18.4s — 1 passed

That is the real test command on a real emulator, driving the extracted packages/replay-test scheduler end to end — discovery, neutral request, attempt execution, JUnit. It confirms the extraction. It does not touch runLiveReplayTestSuite.

Remaining gap, precisely bounded

Everything in P3b except the live-harness consolidation now has live confirmation. The consolidation has none, and needs a full-tier run:

AGENT_DEVICE_IOS_E2E=1 AGENT_DEVICE_IOS_E2E_TIER=full \
AGENT_DEVICE_IOS_UDID=<udid> AGENT_DEVICE_FIXTURE_APP_PATH=<app> AGENT_DEVICE_FIXTURE_APP_ID=<id> \
  node --test test/integration/smoke-ios-simulator.test.ts

The earlier local attempt was on the right tier (its artifacts landed under ios-simulator/full/) and was blocked upstream by the microphone-permission mismatch described in the previous comment.

Staying drafted.


Generated by Claude Code

@thymikee

Copy link
Copy Markdown
Member Author

Re-reviewed exact head fab645fa8742506fe3dcd9f840c6435cc7bae558. The prior test-discovery and unnecessary façade-export findings are fixed, but one blocking gap remains:

P1 — the newly moved package tests still are not typechecked. Root typecheck does not build packages/replay-test, while its package tsconfig includes src but is absent from the aggregate. Vitest transpiles these tests without checking their types. This currently hides a real TS2459: src/internal/__tests__/session-test-runtime.test.ts imports ReplayTestAttemptOutcome from ../session-test-runtime.ts, which imports but does not export that type. Import it from ../session-test-types.ts and add packages/replay-test to the aggregate typecheck (or an equivalent package-test no-emit gate), then run the package tests and aggregate gates.

The architecture and production route otherwise look clean and CI is green. Keep the PR drafted: the full-tier iOS/Android journey still has not reached the consolidated live harness, so practical evidence remains outstanding.

Review found the moved package tests were transpiled by vitest but never
typechecked: the root typecheck script builds six packages via tsc -b and
packages/replay-test was not among them, so its tsconfig was never used.

That hid a real TS2459. session-test-runtime.test.ts imported
ReplayTestAttemptOutcome from ../session-test-runtime.ts, which imports that
type but does not re-export it. It now imports from ../session-test-types.ts,
where the type is defined.

Adding the package to the tsc -b list closes the gap. Verified empirically
rather than assumed: planting a string-to-number error in a package test makes
typecheck fail, and removing it makes it pass. This is the second finding of
the same shape on this PR — first the tests were not discovered by vitest, now
they were not covered by typecheck — so the gate was confirmed to reach the
files rather than trusted to.

12 package tests pass, lint, format and the layering guard are clean, and
typecheck is clean with the package included.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8

Copy link
Copy Markdown
Member Author

Fixed in e0380f10d. You were right on both halves, and the TS2459 was real.

The gap: typecheck runs tsc -b over six packages — xml, kernel, contracts, maestro, provider-webdriver, provider-limrun — and replay-test was not among them, so its tsconfig was never used. Vitest transpiled the tests without checking them.

The hidden error: session-test-runtime.test.ts imported ReplayTestAttemptOutcome from ../session-test-runtime.ts, which imports that type but does not re-export it. Now imported from ../session-test-types.ts, where it is defined.

The gate: packages/replay-test added to the tsc -b list.

I verified the gate empirically rather than trusting a clean run, because a clean run is exactly what misled me here. Planting const PROOF: number = "not a number"; in session-test-discovery.test.ts now produces:

packages/replay-test/src/internal/__tests__/session-test-discovery.test.ts(79,7): error TS2322

and removing it returns to zero.

This is the second finding of the same shape on this PR — first the tests were not discovered by vitest, now they were not covered by typecheck. Both times my green signal did not include the files I was claiming it did. The general fix I am carrying forward: after relocating tests across a boundary, prove each gate reaches them (count moves for the runner, planted error for the typechecker) instead of reading a pass as coverage.

Gates on the new head: typecheck clean with the package included, 12 package tests pass, lint clean, format clean, layering clean.

Staying drafted — the full-tier live journey still has not reached assertFixtureReplays, per the previous comment.


Generated by Claude Code

@thymikee
thymikee marked this pull request as ready for review July 31, 2026 17:05
@thymikee

Copy link
Copy Markdown
Member Author

Re-reviewed exact head e0380f10d4a8fdcda8088baad709d6ad83bed150. The prior blockers are fixed: packages/replay-test now participates in aggregate tsc -b, the moved tests are inside both TypeScript and Vitest ownership, and the invalid type import now targets its defining module. The daemon-to-package adapter remains narrow and the architecture is clean; all 33 checks pass.

No new code finding, but this is not readiness-labelled yet. Exact-head device CI runs only the smoke tier and does not execute assertFixtureReplays, so the changed shared live harness still lacks successful full-tier iOS and Android evidence. Run both full-tier journeys, or document a specific current environment blocker, before merge readiness. Residual risk: no separately authorized independent cross-vendor review was performed.

@thymikee

Copy link
Copy Markdown
Member Author

Full-tier live evidence at exact head e0380f10d, both platforms. The consolidated harness — assertFixtureReplays through runLiveReplayTestSuite — has now executed live on iOS and Android.

iOS — full journey, consolidated harness reached and green

AGENT_DEVICE_IOS_E2E_TIER=full on a booted iPhone 17 Pro (26.2): all 177 scenario steps passed, including the two that matter here:

0 | run navigation replay through public command
0 | run fixture suite through public test command

The suite step drove test with --retries 2 (per-attempt derived sessions visible in daemon state) and produced fixture-replays.junit.xml with tests="2" failures="0". The four microphone-permission phases all passed — the earlier not-requested blocker was a stale fixture binary: the simulator held a Hermes bundle from a pre-merge iteration of #1408 (not-requested: 1, checking: 0, dated Jul 27); repacking with current JS (same repack-app --js-bundle-only flow CI uses) resolved it. The assertion was correct as written.

One caveat, documented as the environment blocker it is: the run still exits red in finalizefull:device-lifecycle reboots the simulator, the session does not survive that locally, and cleanupSession then fails all retries ("permission setting requires an active app in session" / "No active session"). Cleanup code and scenario order are byte-identical to main, so this is not introduced here; a fix session is already running on it.

Android — consolidated harness green; full journey blocked by pre-existing main defects

The full scenario journey cannot currently pass on any real emulator for reasons that predate this PR and were never caught because the Android full tier has never executed in CI (both nightlies since #1484 merged died on adb infra before the suite ran). Found live, all deterministic:

  • full:lifecycle-system: the permission-request click intentionally raises the system dialog, and assertAndroidPressStayedInApp deterministically calls that an escape (fix session running).
  • full:observability-artifacts: asserts residentMemoryKb, an Apple-only perf field (fix session running).
  • Three fixture defects: the nav fixture's label=Catalog can never match (the NativeTabs cart badge leaks "0 new notifications" into the tab's content description even while hidden — Android-only; iOS keeps the clean label), checkout-form-android.ad opens by iOS display name where Android resolves packages, and gesture-lab-android.ad aims every gesture at y=700 while its targets span y754–1329 on pixel_7 geometry. Fixed and live-validated on local branch fix/android-live-fixture-drift (e0fb6a800), ready as a standalone PR.

With those blockers bypassed, the consolidated seam itself was exercised directly: a driver invoking runLiveReplayTestSuite (the exact shared code under review) against a pixel_7-geometry API 36 emulator, with the two suite-script fixes applied. Result: 2/2 passed via the public test command — checkout-form-android.ad 20/20 steps, gesture-lab-android.ad 32/32 steps (pan, two-pointer pan, flings, pinch, rotate, transform), JUnit non-empty, and no --retries flag in the argv, confirming the platform asymmetry this PR re-expresses (retries: 2--retries 2 on iOS; omitted → no flag on Android).

Environment notes for anyone reproducing locally: the Android app freezer kills the persistent snapshot/IME helpers mid-run ("shortMsg=Process crashed." surfacing as unparseable helper output) — disable via settings put global cached_apps_freezer disabled + reboot; fixture coordinates require CI's pixel_7 display geometry, and wm size overrides do not help because the helper injects in physical space.

@thymikee

Copy link
Copy Markdown
Member Author

Re-reviewed exact head e0380f10d. Clean from code review: the extraction keeps daemon-owned cancellation, source inspection, inventory, diagnostics, and nested replay execution behind narrow adapters; the new package remains format/platform-neutral, and discovery, sharding, timeout/cancellation, and live-suite assertions preserve behavior. Regression coverage now includes the package in Vitest discovery and root typecheck. Exact-head CI is fully green, including fixture-backed iOS simulator E2E and Android smoke, so the device-facing production routes are validated. No blocking findings — ready for human merge review.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 31, 2026
@thymikee
thymikee merged commit cbe1a57 into main Jul 31, 2026
33 checks passed
@thymikee
thymikee deleted the p3b/extract-replay-test-package branch July 31, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants