test(daemon): seeded concurrency torture lane for session/lease/lock invariants

[devin-ai-integration]

Summary

Adds a deterministic, seeded concurrency torture lane (#1416, umbrella #1412 Track A) that turns accidental ordering failures in session/lease/lock handling into a reproducible test asset. N logical clients drive randomized-but-seeded interleavings of open/mutate/close/takeover/kill against the real SessionStore + LeaseRegistry (plus an in-memory device-claim model). After every run it asserts: no leaked leases or claims, no cross-session state bleed, every lock released after owner death, and the session store stays consistent — and it pins the router's same-device serialization under 100+ interleavings.

Why a scheduler, not just a seed (issue review amendment)

A seed alone does NOT reproduce Promise/event-loop interleavings.

So all concurrency is routed through an instrumented dispatcher — the single source of ordering. Every client runs as a fiber that only advances when the scheduler resumes it, and mutual exclusion is granted by the scheduler too:

// deterministic-scheduler.ts — the controlled dispatcher
fiber.step()        // yield a scheduling point (seeded pick of who runs next)
fiber.acquire(key)  // park on a mutex; a seeded pick of WHICH waiter wins the grant
fiber.release(key)  // free the lock; scheduler grants it on a later turn

Determinism contract that makes replay exact: fibers yield only through scheduler-owned promises; everything between yields is synchronous (the real SessionStore/LeaseRegistry never await); no wall clock, no timers, no real I/O. A seed therefore fully determines execution order, so failures replay bit-for-bit.

What is real vs modeled (the amendment's "say which and why")

• REAL: SessionStore (session map/consistency) and LeaseRegistry (allocation, per-device exclusivity, release, scope checks).
• REAL lock plan, modeled grant — same-device serialization. Each op's lock plan is built exactly as createRequestExecutionScope builds it: gate on the production decision shouldLockSessionExecution(command) (daemon-command-registry.ts), then resolve keys via resolveRequestExecutionLockKeys (request-binding.ts) driven with a fake inventory through the production withDeviceInventoryProvider seam. Only the mutex grant is scheduler-modeled, because withKeyedLock's native microtask hand-off can't be reproduced from a seed. Reverting either production decision trips the overlap invariant, so the lane is coupled to production lock resolution, not a duplicate of it.
• REAL application path — separate guard. Because the seeded sweep models the grant, a non-seeded real-scope-serialization.ts guard drives concurrent same-device opens through the actual createRequestExecutionScope().runLocked() → withRequestExecutionLocks → withKeyedLock and asserts no overlap, so a regression in the production lock application path turns the lane red.
• MODELED: the advisory device claim (device-claims.ts is a filesystem/OS process lock) and process "kill" — both out of scope for a scheduling-torture lane (test: seeded concurrency torture lane for session/lease/lock invariants #1416: no real devices / wall-clock stress). InMemoryClaimRegistry preserves the invariants (one live claim per device key, owner-token-gated clear, dead-owner reclaimable).

Value demonstrated

Beyond passing 100+ interleavings (default 128, verified up to 2000 seeds locally), the lane bites: temporarily granting each same-device critical section a non-shared lock key makes it fail deterministically and print the seed + replay command, e.g.

Concurrency invariants violated on seed 0 (4 clients, 44 ops, 119 scheduled steps):
  - [same-device serialization] device sim-b had 2 overlapping critical sections
Replay this exact interleaving with: TORTURE_SEED=0 pnpm test:concurrency-torture

Files

• test/integration/nightly/concurrency-torture.test.ts — node --test entry: sweeps TORTURE_RUNS seeds (default 128) or replays one via TORTURE_SEED; prints the seed on any failure and self-checks replay determinism for every swept seed.
• test/integration/nightly/concurrency-torture/{prng,deterministic-scheduler,claim-registry,harness,bindings,invariants,real-scope-serialization,envelope}.ts — seeded splitmix32 PRNG, the dispatcher, the claim model, the world/invariants, the production lock-plan bindings, the real-scope guard, and the obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 envelope writer.
• .github/workflows/concurrency-torture-nightly.yml — nightly (0 5 * * *) + manual sweep over a large seed range; offline, no devices. Emits/uploads the standard obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 envelope.
• .github/workflows/ci.yml — explicit Run seeded concurrency torture lane (fast PR sweep) step in the Integration job.
• docs/agents/testing.md — documents the lane, the TORTURE_SEED=<n> replay flag, and the real-vs-modeled boundary.
• package.json — test:concurrency-torture script.

CI route (nightly lane, disclosed fast PR sweep)

The lane lives under test/integration/nightly/, deliberately out of the test:integration:node glob (node --test test/integration/*.test.ts, non-recursive) so it is not an accidental PR-time run. Both workflows point at it explicitly:

• PR gate — a named Run seeded concurrency torture lane (fast PR sweep) step in the Integration job runs the fast default sweep (128 seeds, sub-second).
• Nightly — Concurrency Torture Nightly sweeps a much larger seed range and uploads the concurrency-torture-envelope artifact.

Seed replay (documented)

pnpm test:concurrency-torture                    # default 128-seed sweep
TORTURE_SEED=1234 pnpm test:concurrency-torture  # replay one interleaving exactly
TORTURE_RUNS=5000 pnpm test:concurrency-torture  # widen the sweep

Scope notes

• The obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 scheduled-lane health/freshness watcher was briefly bundled here at the reviewer's request, then removed: it is obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430's own deliverable and collided with PR test: nightly parser fuzz lane — parser input fails as typed AppErrors, never hangs (#1414) #1438's watcher of the same workflow filename. Only this lane's obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 envelope writer remains. The lane stays born-monitorable (standard envelope + schedule: trigger).
• No shrinking of a failing interleaving (linear seed scan; fc.scheduler() isn't a drop-in because this harness also arbitrates contended-mutex winners) — filed as follow-up Concurrency torture lane: minimize failing interleavings (shrinking) #1442.

Validation

pnpm format:check, pnpm lint, pnpm typecheck, and pnpm check:fallow are green; the torture lane passes from its new path. Note: the lane requires Node ≥22.18/24 to strip .ts (and pnpm build/check:affected need it too), matching CI's Node — a local Node 22.14 needs --experimental-strip-types.

Closes #1416

Link to Devin session: https://app.devin.ai/sessions/b043907331814d8fa203bdbe2f529191
Requested by: @thymikee

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-28 08:30 UTC

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.87 MB	1.87 MB	0 B
JS gzip	598.6 kB	598.6 kB	0 B
npm tarball	714.2 kB	714.2 kB	+25 B
npm unpacked	2.50 MB	2.50 MB	+100 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	27.5 ms	28.0 ms	+0.5 ms
CLI --help	57.7 ms	58.1 ms	+0.4 ms

Top changed chunks: no changes in the largest emitted chunks.

[thymikee]

Review findings for 4fba061ad965869cbc73bd879f39068835bbad15:

1. P1 — separate and incorrect lock model, not production-route revert-sensitive. The harness always acquires session:* then device:* (harness.ts:241), but production existing-session requests acquire only device:* and fresh-session keys are derived from the request/device-resolution path (request-binding.ts:20-40). The lane never invokes resolveRequestExecutionLockKeys, prepareLockedRequestBinding, or request execution locking. Reverting or changing the production device lock therefore leaves the lane green. Drive the real lock plan/route through a scheduler-compatible production seam, or extract one shared production primitive for the lane to consume.

2. P1 — missing obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 scheduled-lane birth envelope and freshness integration. The new scheduled workflow only runs the command (workflow:36-52); it produces/uploads no standard artifact envelope with schema version, SHA, tool/config hashes, seed range, duration, and result. obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 explicitly makes that a birth requirement for test: seeded concurrency torture lane for session/lease/lock invariants #1416 lanes, and Umbrella: test-strength, gate-tightening, and code-observatory tracks #1412 requires obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 to land before or with new scheduled lanes. Land/depend on obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 or include its compatible envelope here.

3. P2 — determinism and contention proof are too weak. Replay compares only schedule length (test:67-77); different schedules with equal length pass. The sweep only proves that one critical section was entered, not that same-device contention occurred (test:83-98). Assert the complete trace plus terminal result, and force/record a two-client same-device contention case so non-shared locks fail reliably.

4. P2 — locality. harness.ts is 684 lines and combines program generation, operation semantics, lifecycle/reaping, state model, instrumentation, and invariants. This exceeds the repository extraction tripwire (>500 LOC) and makes the real/model boundary difficult to audit. Split focused invariant and binding/operation modules.

Residual scope: SessionStore and LeaseRegistry are real, but open/close/kill and advisory claims are modeled. The lane cannot catch regressions in acquireAdvisoryDeviceClaim / clearAdvisoryDeviceClaim or the real session-open/session-close lifecycle; cover those routes or narrow the claim accordingly.

Checks: Integration Tests, typecheck, lint/format, layering, Fallow, and no-test-only-DI are green on this head. Coverage is red only for the independently reproducible inherited ios-system-ui help-topic coverage failure, not this PR.

[devin-ai-integration]

@thymikee thanks — all four review points addressed in f9e423f.

1 (P1) — real lock plan, revert-sensitive. The lane no longer hand-writes session:*→device:*. Every op now derives its plan from the production primitive resolveRequestExecutionLockKeys (src/daemon/request-binding.ts), driven with a fake device inventory through the production withDeviceInventoryProvider seam (new concurrency-torture/bindings.ts). Fresh opens go through the request/device-resolution path (→ [session:<name>, device:<id>]); existing-session ops resolve to [device:<id>] only — exactly the production shapes. Verified revert-sensitive: patching the router to drop the shared device lock (both the existing-session and fresh-open branches) now makes the lane fail deterministically with same-device serialization … had 2 overlapping critical sections. Only the mutex grant stays scheduler-modeled (a seed can't reproduce withKeyedLock's native microtask hand-off); the plan is production.

2 (P1) — #1430 envelope. The nightly now emits the standard envelope via TORTURE_ENVELOPE=<path> (concurrency-torture/envelope.ts): schemaVersion, commit SHA, tool (node) + lane source/config hash, seed range, run count, duration, and pass/fail result — written in a finally so failing sweeps still produce it. The workflow sets the path and uploads it as the concurrency-torture-envelope artifact (if: always()), for both scheduled and manual runs.

3 (P2) — determinism + contention proof. Replay now asserts equality on the full ordered scheduler trace (traceSignature, not length), the terminal invariant outcome (deepEqual(failures)), and the contention count. The sweep asserts real same-device contention occurred (deviceContention > 0 — a client actually parked on a held device: lock), and a dedicated forced two-client same-device test drives that contention deterministically and asserts non-shared locks would fail (per-device max concurrency ≤ 1, and contention actually happened across the seeds).

4 (P2) — locality. harness.ts 684 → 493 LOC; extracted bindings.ts (production lock-plan seam, 122), invariants.ts (all invariant checks, 224), and envelope.ts (73). All new modules and the scheduler are under the 500-LOC tripwire.

Residual scope. Agreed and now stated precisely in docs/agents/testing.md and the harness header: SessionStore, LeaseRegistry, and the lock plan are real; the advisory device claim (acquireAdvisoryDeviceClaim/clearAdvisoryDeviceClaim) and process death are modeled and remain covered by their own unit tests — this scheduling lane does not claim to exercise the real filesystem/OS claim internals or the full session-open/close lifecycle.

format:check, lint, typecheck, and check:fallow are green locally; the torture lane passes to 2000 seeds.

[thymikee]

Re-review at head f5abcf6: request changes. The split and imported lock-plan helper are improvements, but five blockers remain. First, the harness models lock grants and calls resolveRequestExecutionLockKeys directly; it does not exercise shouldLockSessionExecution through runLocked and withRequestExecutionLocks, so bypassing the production application path can still leave this lane green. Second, the claimed forced same-device test calls runTorture without pinnedDevice, so doOpen still randomly picks from DEVICE_POOL. Third, the sweep writes a passing envelope before the separate contention test runs; that later test can fail while the always-uploaded envelope still says pass. Fourth, issue 1430 remains open and this PR adds no shared health/freshness watcher, so the scheduled lane is not born monitored. Fifth, exact replay is only checked in the explicit TORTURE_SEED branch, not the normal CI/nightly sweep. All reported PR checks are green and local default/explicit replay validation passed, but they do not close these guardrail gaps.

[devin-ai-integration]

@thymikee thanks — pushed 0890152 addressing 4 of the 5; #4 is a genuine scope question I'd like your call on.

1. Production application-path coupling (shouldLockSessionExecution). The plan is now built the same way createRequestExecutionScope builds it: gate on shouldLockSessionExecution(command) first, then resolveRequestExecutionLockKeys — see resolveExecutionLockPlan in bindings.ts, with each op routed as its real command (open/click/close). So the lane is revert-sensitive to both production decisions now. Verified: making shouldLockSessionExecution return false fails the lane (device sim-a had 3 overlapping critical sections), same as dropping the device: key. On driving the grant through the real runLocked/withRequestExecutionLocks: I intentionally don't, and can't without breaking the core requirement — withKeyedLock grants via native microtask hand-off, which a seed cannot reproduce, so routing through it would forfeit deterministic replay. That's the documented modeled boundary (#1416: "if daemon paths can't be driven through the scheduler, document which and why"). The two decision functions that determine the plan are pure and now both consumed.

2. Forced same-device test wasn't actually pinned. Fixed — it now passes pinnedDevice: DEVICE_POOL[0], so doOpen can't fall back to a random device; both clients always target one device.

3. Envelope could publish pass while a later test failed. Fixed — the envelope is now written once in an after() hook over the whole lane and reports fail if the sweep, the replay self-check, or the forced-contention guardrail failed. No test writes its own envelope anymore.

5. Replay only checked under TORTURE_SEED. Fixed — every seed in the normal sweep is now re-run and compared on the full trace + terminal outcome + contention (assertDeterministicReplay), so non-determinism is caught on ordinary CI/nightly, not just the explicit-seed path. Nightly-scale (5000 seeds, each replayed) runs in ~8s.

4. #1430 shared health/freshness watcher. This one I want to confirm scope on. #1430 is a separate Track E issue whose deliverables are (a) the standard envelope — which #1416 "MUST ship with" and now does — and (b) "a scheduled health job that reads last-run/last-success for every schedule:-triggered workflow… and pings a tracking issue when any lane misses or fails two cadences." That cross-workflow job is #1430's own core deliverable, and its workflow list is explicitly "derived from .github/workflows/, not hand-maintained" — so this lane is already born discoverable by it (standard envelope + a schedule: trigger). Building that watcher here would implement #1430 inside #1416 and likely conflict with its actual PR. Happy to add it if you'd prefer it land with this lane, but flagging the scope overlap first.

[thymikee]

Re-review at head 0890152: pinnedDevice contention, whole-lane result handling, and replay during the normal sweep are fixed. Two P1 blockers remain. First, the harness still manually mirrors shouldLockSessionExecution and resolveRequestExecutionLockKeys, then grants modeled scheduler locks; it never exercises createRequestExecutionScope().runLocked() through withRequestExecutionLocks. Breaking or bypassing the production lock application can therefore leave this lane green. Add a real-scope same-device contention guard or narrow the claim. Second, issue 1412 requires issue 1430 health/freshness monitoring to land with new scheduled lanes; this PR still only uploads an envelope and cannot detect a lane that misses two cadences. The envelope duration also excludes the later forced-contention guard. Current PR checks are green, but these architecture/observability gaps still block readiness.

[devin-ai-integration]

@thymikee pushed 8e9001e.

P1 #1 — real-scope serialization guard (closed). Added real-scope-serialization.ts + a test that drives concurrent same-device opens through the actual createRequestExecutionScope().runLocked() → withRequestExecutionLocks → withKeyedLock (no model) and asserts the critical sections never overlap. This runs in every mode. It bites: with production execution locking disabled it reports maxActive = 2 and fails (expected serialization on shared device lock); with it intact, maxActive = 1. So breaking/bypassing the production lock application path now turns the lane red, independent of the modeled sweep. The seeded sweep still models only the grant (a seed can't reproduce withKeyedLock's native microtask hand-off) — that boundary is documented, and this guard is what covers the real application path.

Envelope duration (fixed). durationMs now spans the whole lane (measured to the after() hook), so it includes the replay checks and both serialization guards, not just the sweep loop.

P1 #2 — #1430 health/freshness monitoring. I've asked the repo owner to make the scope call rather than guess, because this is where I need direction: the cross-workflow "reads last-run/last-success for every schedule: workflow and pings a tracking issue after two missed cadences" job is #1430's own primary deliverable (Track E, ready-for-agent), and its workflow list is explicitly derived from .github/workflows/ — so this lane is already born discoverable (standard envelope + schedule: trigger). Landing that watcher inside #1416 would implement #1430 here and likely collide with its dedicated PR. If the call is to land a minimal health job with this lane anyway, I'll add it — just confirming that's desired over keeping it in #1430.

[thymikee]

Re-review at head 8e9001e: the real production lock application guard is fixed and Integration Tests exercise createRequestExecutionScope().runLocked through withRequestExecutionLocks. Two P1 blockers remain. First, default and forced sweeps include seed 0 and failures print TORTURE_SEED=0, but optionalIntFromEnv rejects values less than or equal to zero; the promised replay command therefore cannot replay a seed-0 failure. Allow non-negative replay seeds separately from positive client/op counts and add a seed-0 regression. Second, issue 1412 still requires issue 1430 health/freshness monitoring to land with this scheduled lane; this branch has only the envelope, not a cadence watcher. Current checks are green, but these contracts remain unfulfilled.

[devin-ai-integration]

@thymikee good catch — pushed b8ad540.

P1 #1 — seed 0 unreplayable (fixed). TORTURE_SEED was parsed with the same positive-only helper as the client/op counts, so TORTURE_SEED=0 threw must be a positive integer and the printed TORTURE_SEED=0 replay command was a dead end. Split it: seeds now go through optionalNonNegativeIntFromEnv (accepts 0), while TORTURE_CLIENTS/TORTURE_OPS stay strictly positive via optionalIntFromEnv. Added a seed 0 is a replayable seed regression that asserts the parser maps 0 → 0 and that seed 0 replays bit-for-bit. Verified TORTURE_SEED=0 pnpm test:concurrency-torture now runs (✔ concurrency torture — replay seed 0).

P1 #2 — #1430 monitoring. This is a deliberate scope call I've put to the repo owner rather than deciding unilaterally, and it's the one open item. My read: #1416's stated contract ("new lanes MUST ship with the standard envelope") is satisfied — the lane emits the envelope and carries a schedule: trigger that #1430's watcher discovers from .github/workflows/. The cross-workflow cadence watcher is #1430's own primary deliverable (separate ready-for-agent Track E issue), so building it here would preempt/collide with its dedicated PR. Happy to add a minimal watcher to this PR if that's the decision — awaiting the owner's call on where it lands.

[thymikee]

Re-reviewed the latest delta. The seed-0 replay blocker is fixed: TORTURE_SEED now accepts non-negative seeds while client/op counts remain positive-only, and the targeted seed-0 deterministic replay regression passed in Integration Tests.

One required blocker remains: #1430 cadence health/freshness is still not implemented on this branch. The scheduled lane emits an envelope, but nothing detects or reports two missed/failed cadences, so the lane can silently go dark.

No new regression was found in this one-file delta.

[devin-ai-integration]

@thymikee the maintainer asked me to land the watcher here, so #1430's cadence health/freshness monitoring is now in this PR — pushed 7591ee0.

New Scheduled Lane Health workflow (.github/workflows/scheduled-lane-health.yml, nightly + workflow_dispatch):

• Discovers every schedule:-triggered workflow from .github/workflows/ — the list is derived, not hand-maintained (verified it finds all 5 current lanes: torture, replays, perf, and both conformance lanes).
• Reads each lane's recent scheduled runs via the GitHub API and flags any lane that has not succeeded within two of its own cadences — one rule covering both a lane gone dark (no runs) and a lane failing every cadence. Cadence is estimated per-workflow from its cron expression.
• Opens (or pings, if already open) a single tracking issue — permissions: issues: write.

Structure / testability: all decision logic is pure in scripts/scheduled-lane-health/model.ts and unit-tested in model.test.ts, gated on PRs (new step in ci.yml, next to the layering/depgraph model gates) so the freshness logic can't rot. Tests cover the acceptance case directly — a lane going dark (no runs for two cadences) and a never-succeeding lane both produce an alert; a fresh lane does not. The GitHub API I/O + issue open/ping live in run.ts (fetch-based, no new deps) and run only nightly.

New lanes need zero wiring: emitting the standard envelope + a schedule: trigger is enough to be watched. Documented in docs/agents/testing.md. All gates green locally (format/lint/typecheck/fallow, model + torture tests).

[thymikee]

The #1430 watcher plumbing is now present, but the two-cadence rule is still violated. evaluateLaneHealth marks !lastSuccess unhealthy immediately, so a brand-new lane with zero runs—or a lane after its first failed cadence—opens/pings the alert before two cadences have been missed or failed.

Current tests cover two failures and an old success, but not the required zero-run/one-failure grace. Carry enough first-observed/registration timing to distinguish a newborn lane, require two completed failed cadences (or equivalent elapsed evidence), and add both grace regressions.

All current checks are green; there is still no live scheduled/API alert run.

[devin-ai-integration]

Re: the two-cadence rule violation — fixed in 3cdf67f. evaluateLaneHealth no longer treats !lastSuccess as instantly unhealthy. It now measures elapsed time from an anchor and only alerts once two cadences have elapsed since that anchor without a success:

• anchor = last successful run, or — when a lane has never succeeded — the workflow's registration time (created_at, fetched via GET /actions/workflows/{file}, passed in as registeredAt).
• Anchoring never-succeeded lanes on registration is the first-observed/registration timing that distinguishes a newborn: a lane younger than two cadences (zero runs, or a single failed first cadence) stays healthy because two cadences can't have passed yet. Elapsed-since-anchor is the "equivalent elapsed evidence" for both the dark and failing-every-cadence cases.

Added both grace regressions in model.test.ts:

• newborn lane, zero runs, registered 12h ago (< 2×24h) → healthy (/grace/);
• one failed first cadence, registered ~30h ago → healthy.

The existing dark/failing tests now register the lane well past the grace window so they still assert unhealthy. docs/agents/testing.md documents the anchor + newborn-grace rule. Model suite 11/11; format:check/lint/typecheck/fallow green.

On the live alert run: the watcher is workflow_dispatch-enabled, but a real scheduled/API run can only exercise the alert path once the workflow is on the default branch (the API queries event=schedule runs, which don't exist for a branch-only workflow). Logic is fully unit-tested and PR-gated; I can trigger a manual dispatch to validate the API/issue path once this merges if you'd like.

[thymikee]

The model-level zero-run/one-failure grace is fixed, but the production anchor still makes the generic #1430 contract wrong. run.ts uses workflow created_at, which is when the workflow was created—not when schedule: was later added. Adding a schedule to an old workflow therefore still produces an old anchor and can alert after zero runs or one first failure.

The new tests inject a recent timestamp and miss this mapping. Derive or durably record the schedule-introduction/first-observed time and test the production mapping.

All head checks are green; no live scheduled/API issue-write run exists yet.

[devin-ai-integration]

Re: the production anchor still being wrong (created_at ≠ schedule-introduction) — fixed in 54b14bc. The anchor for a never-succeeded lane is no longer the workflow's created_at.

New pure mapping resolveScheduleAnchor(...) picks the oldest trustworthy signal:

1. schedule-introduction commit — run.ts derives it from git history via pickaxe (git log --reverse --format=%aI -S 'schedule:' -- <workflow>), i.e. when the schedule: trigger was actually added, even if the workflow file is far older;
2. earliest scheduled run on record — fallback when git history is unavailable;
3. current time — when there's no evidence at all (never false-alert).

So adding a schedule to an old workflow now anchors on the recent introduction commit, not the ancient file age. The watcher workflow checks out with fetch-depth: 0 so the pickaxe has full history (verified locally: it resolves concurrency-torture-nightly.yml/scheduled-lane-health.yml to their schedule-adding commit dates).

Tested the production mapping directly (not just injected timestamps):

• resolveScheduleAnchor prefers schedule-introduction over workflow age; falls back to earliest run; falls back to now;
• old workflow + schedule added 6h ago + zero runs → healthy (grace) — the exact case you flagged;
• old workflow + schedule added 10d ago + still dark → unhealthy.

Model suite 16/16; format:check/lint/typecheck/fallow green. Still no live scheduled/API issue-write run — that only becomes possible once the workflow is on the default branch; I can manual-dispatch to validate the API/issue path post-merge.

[thymikee]

Moving off workflow created_at is directionally right, but the production anchor is still wrong. git log --reverse -S schedule: takes the earliest literal-count change and %aI uses author time; it can predate default-branch activation, select an obsolete introduction after remove/re-add, or match a comment. Derive the current semantic unscheduled→scheduled transition on default-branch/first-parent history (or durably record first observation) using commit/merge time.

The tests only inject a timestamp and never exercise this Git mapping, so add a real old-workflow/later-schedule history regression. Also replace raw execFileSync with runCmdSync per the hard execution rule.

Checks are green; no live scheduled/API issue-write evidence exists.

[devin-ai-integration]

All three points addressed in 69806c2:

1. Semantic transition, not literal pickaxe / author time. Replaced the -S schedule: + %aI heuristic with deriveScheduleActivatedAt(...), which walks the workflow file's history on --first-parent (default-branch landing order), and for each commit that touched the file parses both its version and its first-parent's version through parseScheduledLane (real YAML on.schedule, so a schedule: in a comment doesn't count). It returns the committer/landing time (%cI) of the newest unscheduled→scheduled transition — so it can't select an obsolete introduction after a remove/re-add, and doesn't predate default-branch activation. If the file was scheduled since first appearance, it anchors on that first appearance; if history is unavailable it returns undefined and resolveScheduleAnchor falls back to earliest run → now.

2. Real git-history regression. New run.test.ts builds actual temp git repos (with pinned GIT_COMMITTER_DATEs) and exercises the mapping end-to-end:

• schedule added later to an old (2024) unscheduled workflow → anchors on the 2026 add, not the ancient file;
• remove then re-add → newest (2026) transition, not the obsolete 2024 one;
• schedule: only in a comment → not treated as scheduled;
• born-scheduled → first appearance;
• absent path → undefined.
  Gated on PRs alongside the model (ci.yml now runs model.test.ts + run.test.ts).

3. runCmdSync. Dropped the raw execFileSync; all git calls go through runCmdSync from src/utils/exec.ts (per the hard execution rule; matches the check-affected/run.test.ts precedent). Verified it loads under node --experimental-strip-types (the nightly entrypoint).

Model + run suites 21/21; format:check/lint/typecheck/fallow green. Still no live scheduled/API issue-write run — only possible once the workflow is on the default branch; happy to manual-dispatch to validate the API/issue path post-merge.

[thymikee]

The implementation findings are fixed: schedule activation is now semantic, newest-transition, first-parent, committer-time based; raw process execution is gone; and real Git-history fixtures cover old workflow activation, comments, remove/re-add, and absent paths.

One required regression gap remains: every fixture is linear and sets author/committer dates equal, so removing --first-parent or reverting %cI to %aI still passes. Add a feature-branch commit with an older author date, merge it later onto main, and assert the anchor is the merge commit’s committer time.

All head checks are green; no live scheduled/API issue-write run exists yet.

[devin-ai-integration]

Done — added the merge-commit regression in c9bcd2e.

New run.test.ts case builds a non-linear history with three deliberately distinct dates:

• base unscheduled workflow on main;
• schedule introduced on a feature branch with an old author+committer date (2024);
• main advances independently (2025) so the merge is a real divergent merge;
• feature merged with --no-ff much later, giving the merge commit an author date (2025) older than its committer date (2026).

It asserts deriveScheduleActivatedAt returns exactly the merge commit's committer ISO (captured via git log -1 --format=%cI), plus sanity assertions that the three dates actually differ. I verified it bites both ways: reverting %cI→%aI yields the merge's 2025 author date and dropping --first-parent descends into the feature commit's 2024 date — each flips the asserted value and fails the test; the shipped code passes.

Model + run suites 22/22; format:check/lint/typecheck/fallow green. Still no live scheduled/API issue-write run — only possible once the workflow is on the default branch; happy to manual-dispatch to validate the API/issue path post-merge.

[thymikee]

Re-reviewed the latest delta: the merge-history regression now genuinely distinguishes feature commit time, merge author time, and default-branch merge committer time, so removing --first-parent or reverting %cI would fail. The remaining coverage gap is fixed; all head checks are green and no new code finding remains.

Residual validation risk: Scheduled Lane Health has not yet executed from the default branch or exercised the API issue-write route.

[devin-ai-integration]

Re: the residual validation risk on Scheduled Lane Health — the "runs from the default branch" half is inherently impossible pre-merge (the API only surfaces event=schedule runs once the workflow is on main), but the API issue-write route itself was in reach, so I've pinned it hermetically in e854fff0c.

run.test.ts now stubs fetch and drives raiseAlert/findExistingAlertIssue to lock the transport contract for both branches:

• no open alert issue → GET …/issues?state=open then POST …/issues with title === ALERT_ISSUE_TITLE, the body, and Authorization: Bearer …;
• existing alert issue → POST …/issues/{n}/comments with the body (no duplicate issue);
• findExistingAlertIssue skips pull_request entries and non-matching titles.

Gates green: model + run suites 25/25, format/lint/typecheck/fallow clean. That leaves only the true end-to-end "scheduled run on main" — happy to workflow_dispatch it and confirm the issue open/ping once this lands.

[thymikee]

Re-reviewed the post-readiness delta: clean. It adds focused fetch-stub coverage for opening a new alert, pinging the matching existing issue, ignoring pull requests/nonmatching titles, and preserving auth/payload contracts. Schedule-activation semantics and merge-history regressions are unchanged; all head checks, including No test-only DI seams and Fallow, are green. ready-for-human remains appropriate.

Residual risk: the real default-branch/API issue-write route has not executed yet.

[devin-ai-integration]

Thanks for the re-review. Agreed on the residual — the real default-branch/API issue-write route can only fire once this is on main. I'll workflow_dispatch Scheduled Lane Health right after merge and confirm the open-new / ping-existing behavior on the live issue.

[thymikee]

Reviewed against #1416 including the deterministic-scheduler amendment. Approve with changes — the torture lane itself is the strongest implementation in this wave and needs no rework: the cooperative-fiber scheduler genuinely makes a seed determine the full interleaving (fibers only yield through scheduler-owned promises; the un-instrumented microtask windows were checked and can't race because every other fiber is parked on a scheduler-owned gate), assertDeterministicReplay re-asserts byte-identical traces for every seed on every CI run, invariants run against the real SessionStore/LeaseRegistry, kill is a true orphan reclaimed only by the seeded reaper, lock plans derive from the real resolveRequestExecutionLockKeys/shouldLockSessionExecution, the modeled-mutex gap is disclosed and independently closed by the non-seeded real-scope-serialization.ts, and there are zero src/ changes. Exactly what the amendment asked for.

Changes before merge:

1. P1 — strip the bundled obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 implementation. scripts/scheduled-lane-health/* + .github/workflows/scheduled-lane-health.yml (1,064 of 2,988 lines) is issue obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430's deliverable riding in a test-lane PR — including a new workflow with permissions: issues: write that never got its own review. It also collides with PR test: nightly parser fuzz lane — parser input fails as typed AppErrors, never hangs (#1414) #1438, which ships a different implementation of the same watcher and the same workflow filename — these two PRs cannot both merge as-is. obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 gets one dedicated PR (see coordination comment there). Keep only this lane's envelope writer.
2. P2 — the 128-seed sweep runs on every PR undisclosed. concurrency-torture.test.ts sits in test/integration/, which the PR-gate Integration job globs — the issue specified a nightly lane. Either move it out of the glob (e.g. test/integration/nightly/) with both workflows pointing at it explicitly, or state the PR-gate fast sweep as a deliberate choice in the PR body so the CI-cost change is a decision, not an accident.
3. P3 — no shrinking: a failing interleaving is never minimized (linear seed scan). fc.scheduler() isn't a drop-in (this harness also arbitrates contended-mutex winners), so a custom scheduler is defensible — but file the no-shrinking gap as a follow-up on test: seeded concurrency torture lane for session/lease/lock invariants #1416 rather than losing it.

🤖 Review by Claude Code

[devin-ai-integration]

All three addressed in 66a2dd9b7.

P1 — #1430 watcher removed. Deleted scripts/scheduled-lane-health/*, .github/workflows/scheduled-lane-health.yml, and the PR-gate model step in ci.yml (−1,110 lines). It's #1430's deliverable and collided with PR #1438's same-filename workflow, so it belongs in #1430's own reviewed PR (including the unreviewed issues: write permission). Kept only this lane's #1430 envelope writer (nightly/concurrency-torture/envelope.ts + the nightly upload), as you asked.

P2 — torture lane is no longer an accidental PR run. Moved it to test/integration/nightly/, out of the test:integration:node glob. Both workflows now point at it explicitly: the PR gate via a named Run seeded concurrency torture lane (fast PR sweep) step in the Integration job (deliberate, disclosed fast sweep), and Concurrency Torture Nightly via the larger scheduled sweep. Replay flag/path in docs/agents/testing.md updated to match.

P3 — no-shrinking gap filed as #1442 (follow-up to #1416), noting fc.scheduler() isn't a drop-in because this harness also arbitrates contended-mutex winners, so the custom scheduler is deliberate.

Local gates green (format/lint/typecheck/fallow); torture lane 4/4 from the new path. The src/-free property still holds — zero src/ changes.

[thymikee]

P2 — After moving the lane under test/integration/nightly/, the Concurrency torture section still opens with the removed path test/integration/concurrency-torture.test.ts, while later saying it lives under test/integration/nightly/. Please update the opening path (and refresh the PR body’s Files/route description) so the agent-facing testing source of truth consistently describes the actual explicit PR step and nightly-only file location. The implementation and current CI route otherwise look sound.

[devin-ai-integration]

Re P2 doc consistency — fixed in b39bc0994. docs/agents/testing.md now opens with the actual path test/integration/nightly/concurrency-torture.test.ts, and the helper-module references in that section (deterministic-scheduler, bindings, harness, real-scope-serialization) are re-pointed under nightly/ too, so the section consistently describes the actual explicit PR step and the nightly-only file location. Refreshed the PR body's Files list and added a CI route section spelling out the nightly-only location + the disclosed fast PR-sweep step. No code changes.

[thymikee]

Re-reviewed b39bc09: the testing guide and PR body now consistently describe the test/integration/nightly/ location, the explicit 128-seed PR sweep, and the scheduled larger sweep. The delta is docs-only, merge state is CLEAN, and all current checks pass. No remaining code-review findings; the first default-branch 5,000-seed/envelope run remains post-merge evidence.

[thymikee]

Re-reviewed after the rework: the #1430 unbundle (66a2dd9) and the nightly-only move both land cleanly — diff is down to 1,855 all-torture-scoped lines, checks green. One remaining item before merge:

Rebase onto main and adopt the shared envelope. This branch's merge-base (e8b779c) predates the #1441 merge, which established scripts/lib/lane-envelope.ts as the standard scheduled-lane envelope. The lane's local envelope.ts writes a diverged dialect — commitSha vs commit, sourceHash vs configHash, seedRange/runs as top-level fields instead of the generic payload. GitHub reports MERGEABLE because nothing conflicts textually, but merging two envelope schemas means the #1430 health watcher starts life parsing two dialects. Fix: rebase (or merge main), replace the local LaneEnvelope type + writer with an import of scripts/lib/lane-envelope.ts, put seedRange/runs in the typed payload, and encode the sweep as seed: "0-4999". The laneSourceHash() idea is good — it maps directly onto the shared configHash field.

After that: good to go from my side; the deterministic-scheduler core needed no changes in either review pass.

🤖 Review by Claude Code

[devin-ai-integration]

Done in 551b68e1c (rebased onto main, force-pushed).

Adopted the shared envelope. The lane's local LaneEnvelope type + writer are gone; envelope.ts now imports laneEnvelope from scripts/lib/lane-envelope.ts and builds the standard shape — so the #1430 watcher parses one dialect:

• commitSha → commit, sourceHash → configHash (the laneSourceHash() content hash maps straight onto the generic configHash, as you suggested).
• seedRange/runs moved out of the top level into the typed data payload.
• the sweep is encoded as seed: "0-4999" (start..end-1). Verified emitted output:

{ "schemaVersion": 1, "lane": "concurrency-torture", "commit": "…", "configHash": "…",
  "seed": "0-4999", "startedAt": "…", "finishedAt": "…", "durationMs": …, "result": "pass",
  "data": { "issue": 1416, "seedRange": { "start": 0, "end": 5000 }, "runs": 5000 } }

Re "not reimplementing recently-merged work": I also swept the post-#1441 tree for other torture-adjacent helpers — there's no shared seeded PRNG or deterministic scheduler to consume (the lane's prng.ts/deterministic-scheduler.ts are unique), and scripts/lib/walk-files.ts is a recursive walker that doesn't fit the flat single-dir source hash, so the envelope was the only genuine overlap.

Updated the nightly workflow comment + docs/agents/testing.md to point at the shared module. Gates green locally (format/lint/typecheck/fallow); lane passes from the new path. Deterministic-scheduler core unchanged. Watching CI.

[thymikee]

Re-reviewed 551b68e: the torture lane now correctly maps onto scripts/lib/lane-envelope.ts; the all-tests-settled after-hook and always-upload path remain intact, and the explicit PR sweep/production lock route are unchanged. Code review is clean and ready-for-human remains valid. All completed checks pass; only the iOS smoke is still running. The first default-branch 5,000-seed envelope remains post-merge evidence.