Skip to content

test(ios): stop cold-host timing from failing the occupancy test - #2847

Open
thymikee wants to merge 1 commit into
mainfrom
test/ios-runner-xctest-cold-start
Open

thymikee wants to merge 1 commit into
mainfrom
test/ios-runner-xctest-cold-start

Conversation

@thymikee

@thymikee thymikee commented Sep 23, 2026

Copy link
Copy Markdown
Member

Summary

Fixes testAbandonedTreeCaptureSkipsQuerySweepAndHonorsWarmupExemption, which runs first in "Run targeted iOS runner XCTest regressions" and failed 8 times from 09-15 to 09-23 on cold or loaded hosts, once on main. Test-only; 1 file, +9/−1.

  • Line 106 (4 failures): a cold viewport read overran its 1 s cap before the tree capture the test targets, so the recorded reason was "preparing tree snapshot". The test now takes that read uncapped before the timed section. The production cap is unchanged.
  • Line 111 (2 failures): the blocking stub's own 20 s timeout released the main thread before the test read the abandonment count (Optional(0)). It is now a named 75 s leak guard, well past the plan deadline, so only the test's own release ends the block.

Dropped after review:

Not fixed here: 4 cold app.launch() timeouts.

Validation

  • ba936608c6, rebased on main: pnpm check:affected --run passed.
  • Unit-test build on a fresh iPhone 17 Pro / iOS 26.2 simulator: the occupancy test passed 5 of 5 iterations.
  • Mutation: pinning the stub timeout to the 8 s slice budget reproduces Optional(0).

@github-actions

github-actions Bot commented Sep 23, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.79 MB 4.79 MB 0 B
Package (unpacked) 4.79 MB 4.79 MB 0 B
Package (download) 1.43 MB 1.43 MB +8 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 22.4 ms 22.5 ms +0.1 ms
CLI --help 62.1 ms 61.1 ms -1.0 ms

@thymikee
thymikee force-pushed the test/ios-runner-xctest-cold-start branch from 8ab2b2c to 41249a0 Compare September 23, 2026 17:23
@thymikee thymikee changed the title test(ios): stop the xctest lane's cold start from failing the wrong test test(ios): stop cold-host timing from failing the occupancy and bare-type tests Sep 23, 2026
@thymikee

Copy link
Copy Markdown
Member Author

This does not fix the bare-type half of the flake, at commit 41249a0. executeTypeCommand builds a failure response as Response(ok:false, error:ErrorPayload(...)) with data == nil, and only sets textEntryRoute on the ok:true DataPayload (https://github.com/callstack/agent-device/blob/41249a0/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift). The test still asserts XCTAssertEqual(typeResponse.data?.textEntryRoute, "synthesized-first-responder") unconditionally (line 408), so when the runner returns TEXT_INPUT_COMMIT_NOT_OBSERVED, data is nil, the comparison is nil-vs-expected, and the test still goes red on a cold host, now with a route-mismatch message instead of the real commit-not-observed cause. The new prefix and code checks never gate the outcome, so a host that drops keystrokes still fails this test.

Can the route assertion move inside the if typeResponse.ok branch, and can the not-ok branch assert the route from evidence the failure path actually carries — either the AGENT_DEVICE_RUNNER_TEXT_ENTRY_ROUTE log line, or textEntryRoute attached to the failure response's data (the failure path's textResult already carries it through dispatchFailureResult)? A unit case that drives the commit-not-observed response through this assertion would prove the fix.

The rest of the diff (occupancy stub timeout) reads sound, so this is the one blocking change before merge. All ten checks on this commit are still queued or in progress, none have failed yet, and the diff only touches iOS runner XCTest source, so those checks are unlikely to speak to this issue either way.

testAbandonedTreeCaptureSkipsQuerySweepAndHonorsWarmupExemption runs first in
the targeted XCTest lane and absorbed a cold host twice over:

- A cold viewport read overran its 1 s cap before the tree capture the test
  targets, so the abandonment reason read "preparing tree snapshot" (4 CI
  failures at line 106). The test now pays that read uncapped before the timed
  section; the production cap is unchanged.
- The blocking stub's own 20 s timeout could release the main thread before
  the test read the abandoned-work count, reading Optional(0) (2 CI failures at
  line 111). It is now a named leak guard far past the plan deadline, so only
  the test's release ends the block.
@thymikee
thymikee force-pushed the test/ios-runner-xctest-cold-start branch from 41249a0 to ba93660 Compare September 23, 2026 18:31
@thymikee thymikee changed the title test(ios): stop cold-host timing from failing the occupancy and bare-type tests test(ios): stop cold-host timing from failing the occupancy test Sep 23, 2026
@thymikee

Copy link
Copy Markdown
Member Author

Addressed at ba93660. The bare-type change is gone: #2845 moved bare type to xctest-element, so the synthesized-route keystroke drop no longer applies to this test. Main's version is kept as is. Also applied a thermo-nuclear quality pass: the occupancy leak guard is now a named constant, and the narration comments are cut. The PR is now 1 file, +9/−1.

@thymikee

Copy link
Copy Markdown
Member Author

The unconditional textEntryRoute assertion on a nil-data failure response from the earlier review (41249a0) is gone at ba93660; this PR now only touches RunnerTests+SnapshotCapturePlanOccupancyTests.swift and is test-only, so it does not carry that risk.

18/18 checks are green on ba93660, and the diff touches only that occupancy test file, which is exactly what the previously failing 'Run targeted iOS runner XCTest regressions' job exercises for this test name, so the green run is direct evidence the fix runs. A single green run does not rule out the cold-host flake recurring, so a repeat run or two on that job would help confirm the fix holds.

I did not run the runner XCTest suite myself; this reads on the diff plus your stated validation (5/5 pass on iPhone 17 Pro/iOS 26.2, and a mutation test pinning the stub to the 8s slice budget reproduces Optional(0)) and the 18/18 green checks. I also didn't re-run pnpm check:affected --run independently.

This looks ready to merge from the code side.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant