test: ratchet mutation score over enumerated decision kernels - #1441
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Size Report
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. |
|
Review: changes requested — three P1 blockers.
All current checks are green, but the PR-side mutation job only ran its self-tests, reported no affected decision kernels, and ran no mutants. That is not readiness evidence for the new lane. |
ff14f27 to
a19412b
Compare
|
All three P1s fixed in a19412b (branch also rebased onto latest 1. 2. Kernel tests now own their module.
3. Envelope. Both lanes write and upload On readiness: agreed that the PR job's "no affected kernels" run proved nothing. This push touches the kernel registry, so the job runs real mutants on it. Locally |
|
Readiness evidence from this push's PR job (all 23 checks green): the affected lane selected That also gives a real per-shard data point for the weekly budget: 183 mutants ≈ 7m16s here, and the largest shard ( |
|
Re-review at head a19412b: the bench:help-conformance script is fixed and checks are green, but two P1 blockers remain. First, affected test ownership is still a hand-picked list rather than derived/comprehensive: for example src/tests/daemon-error.test.ts directly exercises normalizeError from src/kernel/errors.ts but is absent from both the kernel-errors test ownership and workflow paths. Weakening that test skips the affected mutation run, and the current parity tests cannot detect omissions. Second, issue 1412 requires issue 1430 health/freshness monitoring to land with this new scheduled lane. The envelope is only written after report parsing and ratchet evaluation, so setup/Stryker/report failures can still produce no envelope, and there is no derived cadence monitor to detect a dark lane. Fix ownership derivation/completeness and land the shared health path before readiness. |
Adds a Stryker (vitest runner) mutation lane scoped to the decision kernels, a per-module baseline with tool/config provenance, and a ratchet that only lets scores rise. Non-gating until two consecutive stable weekly sweeps. Refs #1415 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…lysis Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ne envelope - restore bench:help-conformance, broken by a formatting-path edit - kernel test files select their module on PRs (registry `tests` + workflow paths), asserted to reach the kernel through the import graph - every mutation run writes the standard scheduled-lane artifact envelope - move src/utils/__tests__/errors.test.ts beside its source per the mirror rule Refs #1415, #1430 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… monitor Ownership of a kernel's tests is now computed from the static import graph (scripts/mutation/ownership.ts) instead of a hand-listed set, so a test that reaches a kernel indirectly -- src/__tests__/daemon-error.test.ts through src/daemon.ts -- selects that kernel on a PR. The PR lane triggers on every src test and shards the derived modules, keeping wall clock at one module. The lane envelope (#1430) is now written on every exit path with the stage it reached, so a crash before any mutant runs is distinguishable from a lane that never ran. Adds the derived cadence monitor (scripts/lane-health, daily workflow): scheduled lanes are enumerated from .github/workflows/ and reported dark, failing, or never-run against their own cron cadence.
a19412b to
8c1dbe7
Compare
|
Both remaining blockers addressed at 1. Ownership is derived, not hand-picked. The registry's and The honest consequence: reachability is a superset of kill-power, so the derived owner sets are large (kernel-errors 469 test files, the others ~320 each). Any src test change therefore selects several kernels. I took that over any narrowing heuristic, since a narrowing is exactly the false negative you flagged. To keep it affordable, the PR lane is now 2. Envelope on every exit path + a cadence monitor. For the dark-lane half I landed the shared path rather than deferring to #1430: Run against this repo just now, it immediately found a real one: Gates green locally: |
The shard artifacts now carry the lane envelope beside mutation.json, and the merge globbed every .json under the download path, so the ratchet job fed the envelope to the report parser and died after the mutants had already run.
thymikee
left a comment
There was a problem hiding this comment.
Re-reviewed at da4bf16. The derived ownership change closes the prior hand-picked-test omission, including daemon-error.test.ts, and the current GitHub checks plus the focused ownership/workflow/envelope/health tests are green.
Two blockers remain:
-
[P1] Reject incomplete shard sets before writing the aggregate envelope. readShardedReports only requires at least one mutation.json, while summarizeReport creates a zero score for every requested module absent from the downloaded reports. While gating is false, that becomes a non-failing regression, so the ratchet can write a complete/pass aggregate envelope even though a matrix shard failed. Validate that every requested module has a real report/nonzero mutant coverage, or otherwise propagate the shard result, and cover a partial-shard failure.
-
[P1] Extend the envelope boundary to genuine early failures. Argument parsing and provenance/baseline reads occur before the try/finally, so malformed or missing configuration still exits without an envelope. The weekly workflow also runs mutation:test before invoking the runner, so a self-test failure produces no aggregate lane envelope. Ensure these pre-state and pre-run failures create and upload a fail envelope, with tests that exercise them.
Residual risk, not a blocker: issue #1415 requires the weekly lane to complete in under 30 minutes, but the first real weekly run has not happened and shard timeouts remain 45 minutes. Please capture the first scheduled or manual weekly timing as follow-up evidence.
…lures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Both P1s fixed in [P1] Incomplete shard sets no longer pass. The aggregate now asserts, after scoring, that every requested module actually reported mutants — a module scored from no report is a dead shard, not a 0% score: That throws before [P1] The envelope boundary now covers pre-state and pre-run failures. Argument parsing, For steps that run outside the runner — the weekly Gates: On the 30-minute budget — agreed, still unproven. The 45-minute shard timeout is deliberate headroom so a slow runner reports instead of being killed; I'll dispatch the weekly lane after merge and post real per-shard wall clock as follow-up evidence. |
thymikee
left a comment
There was a problem hiding this comment.
Re-review confirms the missing-shard and early setup-envelope blockers are fixed and tested. One lane-result bug remains: the weekly ratchet can write a pass envelope, then cp/git checkout can fail; the if: failure() fallback calls --fail-envelope, but onlyIfAbsent preserves any existing envelope, so the failed scheduled job is published as passing. Preserve an existing failure, but upgrade an existing pass to fail for a later workflow failure, and add a pass-then-post-step-failure regression.
All head checks are green and the focused suite passes; the kernel-errors shard took ~10m38, but no full Mutation Weekly run exists, so the <30m budget remains unproven residual risk.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Good catch on the pass-then-post-step-failure hole — fixed in
The regression test does the real sequence: a passing Agreed on the residual risk: |
thymikee
left a comment
There was a problem hiding this comment.
Re-reviewed the latest delta: the later-step failure bug is fixed. Recovery preserves an existing real failure, but upgrades an existing pass to fail; targeted pass-then-failure coverage passes, and all head checks are green. No new code finding.
Residual evidence: the kernel-errors shard ran ~9m04, but Mutation Weekly has never run, so the full-lane <30m budget is not yet empirically proven.
|
Thanks — nothing further to change then. On the residual: |
|
Reviewed against #1415 including the 2026-07-27 amendments. Approve with changes — this is real StrykerJS ( Three things before merge:
P3: the deliberately-missing 🤖 Review by Claude Code |
…ndled watcher Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
All four addressed in 1. Watcher stripped. 2. Sharding is now registry data, and it slices modules rather than just splitting the matrix. A module may declare Exercised end to end locally: 3. The PR lane no longer spends anything before graduation. 4. P3 documented. The One honest side effect: that comment changes the config content hash, so the first weekly sweep reports |
|
P1 — The pre-graduation tooling exception is currently vacuous for a tooling-only diff. |
…mutants Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Re: the P1 on the vacuous tooling exception — correct, and thanks; it was decorative exactly as you describe. Fixed in
if (!gating && !touchesLane) return [];
const modules = new Set(derivedAffectedModules(changed, root));
if (touchesLane) modules.add(LANE_CANARY); // lane sources own no kernelSo a tooling-only diff proves the gate on real mutants at one shard's cost, not on an empty matrix. The regression is end to end rather than a path-filter assertion:
45 mutation self-tests, plus format/lint/typecheck/fallow green. |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Re-reviewed |
|
Thanks — agreed on the residual. The <30-min full-lane figure stays a projection until the workflow is on the default branch ( |
Rebase onto main (post-#1441) and replace the lane-local LaneEnvelope dialect with the shared scripts/lib/lane-envelope.ts builder, so the #1430 health watcher parses one schema: commitSha->commit, sourceHash-> configHash, seedRange/runs moved into the typed data payload, and the sweep encoded as seed "<start>-<end>". Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…invariants (#1439) * test(daemon): seeded concurrency torture lane for session/lease/lock invariants Refs #1416 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): drive torture lane through real lock plan + review fixes - derive each op's lock plan from production resolveRequestExecutionLockKeys via a fake device-inventory provider, so reverting the router's same-device serialization trips the overlap invariant (verified) - assert exact replay: full scheduler trace, terminal outcome, contention - assert real same-device lock contention in the sweep + a forced 2-client case - split harness into bindings/invariants/envelope modules (all <500 LOC) - emit #1430 scheduled-lane envelope (schema/SHA/hash/seed range/duration/result) and upload it from the nightly workflow Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): pass claim data as plain view accessors (fallow) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): gate lock plan on shouldLockSessionExecution; sweep replay + forced-device contention; whole-lane envelope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): add real-scope runLocked serialization guard; whole-lane envelope duration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): allow seed 0 replay; add seed-0 regression (TORTURE_SEED must accept 0) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * obs(#1430): add scheduled-lane freshness/cadence health watcher Discovers schedule: workflows from .github/workflows/, reads recent scheduled runs via the GitHub API, and opens/pings a tracking issue when a lane misses or fails two consecutive cadences. Pure model unit-tested and gated on PRs; API I/O + issue open/ping run nightly. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): give scheduled-lane watcher a two-cadence newborn grace Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): anchor lane grace on schedule-introduction, not workflow age Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): derive schedule-activation semantically via git, through runCmdSync Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): add merge-commit regression pinning first-parent + committer time Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): pin scheduled-lane-health issue-write route via stubbed fetch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): unbundle #1430 watcher; make torture lane nightly-only Strip the scheduled-lane-health watcher (scripts + workflow + PR gate) — it is #1430's deliverable and collides with PR #1438's workflow of the same filename; keep only this lane's #1430 envelope writer. Move the torture lane under test/integration/nightly/ so it is out of the test:integration:node glob, and run it via an explicit, disclosed PR step plus the nightly sweep. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: fix stale torture-lane paths after nightly/ move Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(daemon): adopt shared lane-envelope for torture lane Rebase onto main (post-#1441) and replace the lane-local LaneEnvelope dialect with the shared scripts/lib/lane-envelope.ts builder, so the #1430 health watcher parses one schema: commitSha->commit, sourceHash-> configHash, seedRange/runs moved into the typed data payload, and the sweep encoded as seed "<start>-<end>". Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Summary
Adds a mutation-score ratchet over an enumerated set of decision kernels, so "are these tests
load-bearing?" gets a mechanical answer instead of a coverage percentage. Closes #1415 (umbrella
#1412, Track A).
Scope is a registry, not a config file:
scripts/mutation/modules.tsowns the five modules(
src/kernel/errors.ts,src/daemon/ref-frame.ts,src/commands/interaction/runtime/settle.ts,src/utils/scroll-edge-state.ts,src/selectors/), and everything else derives from it —stryker.config.json'smutateglobs are asserted equal tomutateGlobs(), PR-affected selectionmaps changed files through the import graph, and both workflow matrices are asserted equal to
shardMatrix(). Modules that spawn subprocesses or wait real time stay out by construction.The ratchet, not Stryker, owns pass/fail (
thresholds.break: null), and it is deliberately dormant:Each baseline entry carries the
strykerVersionand a content hash ofstryker.config.json, which iswhat makes drift separable from test strength. Committed starting point (full sweep, Stryker 9.6.1):
daemon-ref-frame100%,selectors70.56%,interaction-settle68.95%,kernel-errors55.19%,scroll-edge-state28.75% — 2,054 mutants total. A failing check prints the surviving mutants(file:line, mutator, replacement) plus the command to reproduce.
Things the diff won't explain on its own:
shards: n;--shard i/npacks its expanded sources intonsize-balanced bins and mutates one.selectorsis ~1,280 mutants — at the rate this PR's own CI job measured (183 mutants / 9m04 on a2-core runner) one job would be ~45–60 min, past both the acceptance budget and its timeout — so it
runs as four jobs. Every job is capped at
timeout-minutes: 30and a test asserts no job declaresmore. The aggregate requires the whole set (
--expect-shards), so a dead shard fails the laneinstead of scoring its module 0.
--list-affectedreturns[]and no mutants run: a report nobody acts on is not worth the minutes. The exception is a diff that
touches the lane's own tooling (
scripts/mutation/,scripts/lib/,stryker.config.json,mutation-baselines/) — that is the one pre-graduation run that buys something, since the gate hasto be proven before it starts failing PRs.
--affectedapplies the same rule, so the ratchet jobcan't spend what
selectdeclined to.scripts/mutation/ownership.ts): a test owns everykernel its imports reach, so
src/__tests__/daemon-error.test.tsselectskernel-errorsthroughsrc/daemon.tswithout naming it. Reaching a kernel is a superset of killing its mutants, so thelane over-selects on purpose: a false positive costs runner minutes, a false negative costs the gate.
scripts/mutation/test-scope.tsasks Vitest for therelated tests (the same delegation
pnpm check:affecteduses) and drops three groups it cannot run:subprocess-stub tests (real time), non-
src/conformance gates, and thread-hostile tests (Stryker'svitest runner hard-codes
pool: 'threads', whereprocess.chdirthrows and anode:worker_threadsworker inside a worker kills the runner). Neither exclusion can hide a survivor — a mutant only an
excluded test could kill is reported as surviving.
scripts/lib/lane-envelope.ts, obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430): every run writes.tmp/mutation/lane-envelope.json—schemaVersion, commit,tool.stryker,configHash,seed(explicit
null: enumerated input, not randomized), duration, stage,result, per-module scores.Written on every exit path, including a crash before any mutant and a step that fails after a
passing ratchet (
--fail-envelopedowngrades apass), so a lane going dark is never mistaken fora lane that passed. It lives in
scripts/lib/for the other scheduled lanes to adopt.Also extracts the duplicated
parseArgs+--helpboilerplate shared by this script andscripts/check-affected/run.tsintoscripts/lib/cli-args.ts.Validation
pnpm format:check,pnpm lint,pnpm typecheck,pnpm fallowclean;pnpm mutation:test(42cases) covers the acceptance criteria directly: a lowered score fails once gating is on, the failure
lists surviving mutants, the same lowering only reports before graduation, regressions keep the
high-water baseline, graduation needs two consecutive full sweeps, affected PR runs never advance it,
provenance drift is not a test-strength regression, an incomplete shard set is not a sweep, and every
failure path still writes an envelope. Real Stryker runs produced the committed baseline;
--shardwas exercised end to end locally (
selectors 1/4→ 2 sources, 393 mutants, 74.81%).Not verified here:
pnpm buildand the Node integration smokes fail on this machine on pristineorigin/maintoo (tsdowncannot importunrun; Node 22.14 has no default.tsstripping). Theweekly lane's wall clock stays a projection until it can be
workflow_dispatched from the defaultbranch. Note the tsconfig comment added in review changes the config hash, so the first weekly sweep
records
provenance-driftand re-bases the baseline before the graduation clock starts.Link to Devin session: https://app.devin.ai/sessions/97c343f089bb49cf820c2569653f3b35
Requested by: @thymikee