Skip to content

test(replay-test): pin the retry-path reporter hint - #1516

Merged
thymikee merged 1 commit into
mainfrom
p3b-pre/pin-retry-hint
Jul 31, 2026
Merged

test(replay-test): pin the retry-path reporter hint#1516
thymikee merged 1 commit into
mainfrom
p3b-pre/pin-retry-hint

Conversation

@thymikee

Copy link
Copy Markdown
Member

A one-assertion ratchet that has to land before P3b relocates session-test-attempt.ts into packages/replay-test, so the move cannot drop a shipped reporter field silently.

The gap

hint reaches reporters from two emit sites in src/daemon/handlers/session-test-attempt.ts:

site when asserted before this PR
emitReplayTestRetryProgress (~:287) every retried failure no
final failure progress (~:374) terminal failure yes, via #1505

#1505 added hint coverage, but only for the final path. The retry emit site was reachable by no assertion at all — and could not have been caught even in principle, because the retry fixture's first-attempt error carried no hint for the scheduler to forward.

Change

runRetrySuite's first-attempt error now carries a hint, and the retry result value asserts it. Test-only; git diff on session-test-attempt.ts is empty.

Counterfactual — run, not assumed

Deleting hint: attempt.outcome.error.hint from the retry emit site:

 ✓ a reporter sees the shipped suite-start, skip, and test-start values
 × reporter step and result sessions track the running attempt, not the start value
 ✓ a failing suite reaches the reporter with the failure message, hint fields, and exit code
 ✓ sharded runs give the reporter shard-scoped sessions and device identity

-   "hint": "retry hint from the failed attempt",
+   "hint": undefined,

 Tests  1 failed | 3 passed (4)

Only the new pin fails. The existing final-path hint test keeps passing — which is precisely the gap being closed. Restored afterwards; production diff empty; both reporter suites green (6/6).

Why now

P3b moves this file. A field that no test asserts is a field a large mechanical move can drop without any gate objecting — the exact failure mode #1505's ratchet exists to prevent, left half-covered. Cheaper to pin it here than to detect it after a 3,000-line relocation.

Refs #1478


Generated by Claude Code

`emitReplayTestRetryProgress` publishes `hint` on every retried failure
(session-test-attempt.ts), and nothing asserted it. The existing hint coverage
coming out of #1505 pins only the final-attempt emit site, so deleting the
retry line left the entire suite green.

The retry fixture's first-attempt error carried no hint, so no assertion could
have caught it even in principle. It now carries one, and the retry result
value asserts it.

Counterfactual, run rather than assumed — with `hint: attempt.outcome.error.hint`
deleted from the retry emit site:

    ✓ a reporter sees the shipped suite-start, skip, and test-start values
    × reporter step and result sessions track the running attempt, not the start value
    ✓ a failing suite reaches the reporter with the failure message, hint fields, and exit code
    ✓ sharded runs give the reporter shard-scoped sessions and device identity
    -   "hint": "retry hint from the failed attempt",
    +   "hint": undefined,

Only the new pin fails; the final-path hint test keeps passing, which is the
gap. Production code is unchanged — `git diff` on session-test-attempt.ts is
empty.

This lands before P3b relocates session-test-attempt.ts into
packages/replay-test, so the move cannot drop the field silently.

Refs #1478

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8
@github-actions

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.91 MB 1.91 MB 0 B
JS gzip 612.5 kB 612.5 kB 0 B
npm tarball 730.1 kB 730.1 kB 0 B
npm unpacked 2.56 MB 2.56 MB 0 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.1 ms 26.3 ms -0.8 ms
CLI --help 59.1 ms 56.2 ms -2.9 ms

Top changed chunks: no changes in the largest emitted chunks.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 31, 2026
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 7cab271: clean readiness. This pins the retry-path reporter hint through the real handler → progress → reporter-registry route; deleting the retry emission drops the observed hook value and fails the new assertion, independently of the terminal-failure pin. Production behavior is unchanged, and CI is green.

@thymikee
thymikee merged commit ab80fb6 into main Jul 31, 2026
43 of 44 checks passed
@thymikee
thymikee deleted the p3b-pre/pin-retry-hint branch July 31, 2026 08:13
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-31 08:13 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants