Skip to content

test: ratchet mutation score over enumerated decision kernels - #1441

Merged
thymikee merged 11 commits into
mainfrom
devin/1785158289-mutation-ratchet
Jul 28, 2026
Merged

test: ratchet mutation score over enumerated decision kernels#1441
thymikee merged 11 commits into
mainfrom
devin/1785158289-mutation-ratchet

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a mutation-score ratchet over an enumerated set of decision kernels, so "are these tests
load-bearing?" gets a mechanical answer instead of a coverage percentage. Closes #1415 (umbrella
#1412, Track A).

Scope is a registry, not a config file: scripts/mutation/modules.ts owns the five modules
(src/kernel/errors.ts, src/daemon/ref-frame.ts, src/commands/interaction/runtime/settle.ts,
src/utils/scroll-edge-state.ts, src/selectors/), and everything else derives from it —
stryker.config.json's mutate globs are asserted equal to mutateGlobs(), PR-affected selection
maps changed files through the import graph, and both workflow matrices are asserted equal to
shardMatrix(). Modules that spawn subprocesses or wait real time stay out by construction.

The ratchet, not Stryker, owns pass/fail (thresholds.break: null), and it is deliberately dormant:

evaluateRatchet(scores, baseline, provenance)
  → verdict per module: new | improved | held | regressed | provenance-drift
  → failed = baseline.gating && anyRegressed        // reports, but does not fail, before graduation

applyRun(...)                                       // --update only
  → keeps the high-water score on regression        // a bad week cannot lower the bar
  → rebases (never "regresses") on provenance drift // tool/config change ≠ weaker tests
  → stableRuns++ only for a comparable FULL sweep; gating = stableRuns >= requiredStableRuns (2)

Each baseline entry carries the strykerVersion and a content hash of stryker.config.json, which is
what makes drift separable from test strength. Committed starting point (full sweep, Stryker 9.6.1):
daemon-ref-frame 100%, selectors 70.56%, interaction-settle 68.95%, kernel-errors 55.19%,
scroll-edge-state 28.75% — 2,054 mutants total. A failing check prints the surviving mutants
(file:line, mutator, replacement) plus the command to reproduce.

Things the diff won't explain on its own:

  • Sharding is registry data, and slices modules, not just splits them. A module may declare
    shards: n; --shard i/n packs its expanded sources into n size-balanced bins and mutates one.
    selectors is ~1,280 mutants — at the rate this PR's own CI job measured (183 mutants / 9m04 on a
    2-core runner) one job would be ~45–60 min, past both the acceptance budget and its timeout — so it
    runs as four jobs. Every job is capped at timeout-minutes: 30 and a test asserts no job declares
    more. The aggregate requires the whole set (--expect-shards), so a dead shard fails the lane
    instead of scoring its module 0.
  • The PR lane costs nothing until it can bite. Before graduation --list-affected returns []
    and no mutants run: a report nobody acts on is not worth the minutes. The exception is a diff that
    touches the lane's own tooling (scripts/mutation/, scripts/lib/, stryker.config.json,
    mutation-baselines/) — that is the one pre-graduation run that buys something, since the gate has
    to be proven before it starts failing PRs. --affected applies the same rule, so the ratchet job
    can't spend what select declined to.
  • Test ownership is derived, never listed (scripts/mutation/ownership.ts): a test owns every
    kernel its imports reach, so src/__tests__/daemon-error.test.ts selects kernel-errors through
    src/daemon.ts without naming it. Reaching a kernel is a superset of killing its mutants, so the
    lane over-selects on purpose: a false positive costs runner minutes, a false negative costs the gate.
  • The suite Stryker replays is derived too. scripts/mutation/test-scope.ts asks Vitest for the
    related tests (the same delegation pnpm check:affected uses) and drops three groups it cannot run:
    subprocess-stub tests (real time), non-src/ conformance gates, and thread-hostile tests (Stryker's
    vitest runner hard-codes pool: 'threads', where process.chdir throws and a node:worker_threads
    worker inside a worker kills the runner). Neither exclusion can hide a survivor — a mutant only an
    excluded test could kill is reported as surviving.
  • Lane envelope (scripts/lib/lane-envelope.ts, obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430): every run writes
    .tmp/mutation/lane-envelope.jsonschemaVersion, commit, tool.stryker, configHash, seed
    (explicit null: enumerated input, not randomized), duration, stage, result, per-module scores.
    Written on every exit path, including a crash before any mutant and a step that fails after a
    passing ratchet (--fail-envelope downgrades a pass), so a lane going dark is never mistaken for
    a lane that passed. It lives in scripts/lib/ for the other scheduled lanes to adopt.

Also extracts the duplicated parseArgs + --help boilerplate shared by this script and
scripts/check-affected/run.ts into scripts/lib/cli-args.ts.

Validation

pnpm format:check, pnpm lint, pnpm typecheck, pnpm fallow clean; pnpm mutation:test (42
cases) covers the acceptance criteria directly: a lowered score fails once gating is on, the failure
lists surviving mutants, the same lowering only reports before graduation, regressions keep the
high-water baseline, graduation needs two consecutive full sweeps, affected PR runs never advance it,
provenance drift is not a test-strength regression, an incomplete shard set is not a sweep, and every
failure path still writes an envelope. Real Stryker runs produced the committed baseline; --shard
was exercised end to end locally (selectors 1/4 → 2 sources, 393 mutants, 74.81%).

Not verified here: pnpm build and the Node integration smokes fail on this machine on pristine
origin/main too (tsdown cannot import unrun; Node 22.14 has no default .ts stripping). The
weekly lane's wall clock stays a projection until it can be workflow_dispatched from the default
branch. Note the tsconfig comment added in review changes the config hash, so the first weekly sweep
records provenance-drift and re-bases the baseline before the graduation clock starts.

Link to Devin session: https://app.devin.ai/sessions/97c343f089bb49cf820c2569653f3b35
Requested by: @thymikee

@thymikee thymikee self-assigned this Jul 27, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-28 08:07 UTC

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.87 MB 1.87 MB 0 B
JS gzip 598.6 kB 598.6 kB 0 B
npm tarball 714.1 kB 714.2 kB +82 B
npm unpacked 2.50 MB 2.50 MB +642 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 21.8 ms 23.3 ms +1.5 ms
CLI --help 45.2 ms 47.7 ms +2.6 ms

Top changed chunks: no changes in the largest emitted chunks.

@thymikee

Copy link
Copy Markdown
Member

Review: changes requested — three P1 blockers.

  1. Restore package.json bench:help-conformance to node scripts/help-conformance-bench.mjs. The new node scripts/mutation scripts/help-conformance-bench.mjs invokes a directory instead of the benchmark entrypoint.
  2. Make affected mutation selection cover tests for all four exact-file kernels, both in moduleForFile() and the PR workflow paths. Current tests live outside those exact source paths (for example src/utils/__tests__/errors.test.ts, src/daemon/__tests__/ref-frame.test.ts, and src/commands/interaction/runtime/settle.test.ts), so weakening them skips the intended PR mutation ratchet.
  3. Add obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430’s standard artifact envelope and freshness integration for this new weekly lane. The weekly workflow currently uploads raw reports/proposed baseline only; it lacks the required schema version, commit SHA, tool/config provenance, duration, and result envelope.

All current checks are green, but the PR-side mutation job only ran its self-tests, reported no affected decision kernels, and ran no mutants. That is not readiness evidence for the new lane.

@devin-ai-integration
devin-ai-integration Bot force-pushed the devin/1785158289-mutation-ratchet branch from ff14f27 to a19412b Compare July 27, 2026 15:21
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

All three P1s fixed in a19412b (branch also rebased onto latest main).

1. bench:help-conformance is back to node scripts/help-conformance-bench.mjs. A sed that added scripts/mutation to the format paths also hit that script — good catch.

2. Kernel tests now own their module. KERNEL_MODULES gained a tests field, moduleForFile() matches it, and the PR path filter lists the same files (workflow.test.ts asserts the parity). Two supporting changes:

  • src/utils/__tests__/errors.test.tssrc/kernel/__tests__/errors.test.ts, so it mirrors src/kernel/errors.ts per the topology rule. A new test asserts that any mirrored test file of a kernel source selects that module, so a future mirror cannot be silently unlisted.
  • Listed tests are verified, not asserted: modules.test.ts walks the import graph and fails if a listed test never reaches the kernel's sources. That is what validated daemon/handlers/__tests__/interaction-settle.test.ts (reaches settle.ts transitively), and it is why scroll-edge-state lists its gesture/dispatch callers — it has no mirrored test at all, which is also why it scores lowest (28.75%).

affectedModules(['src/kernel/__tests__/errors.test.ts', 'src/commands/interaction/runtime/settle.test.ts']) now returns ['kernel-errors', 'interaction-settle'].

3. Envelope. Both lanes write and upload .tmp/mutation/lane-envelope.json via scripts/lib/lane-envelope.ts: schemaVersion, commit SHA, tool.stryker, configHash, seed (explicit null — the input is enumerated, not randomized), startedAt/finishedAt/durationMs, result, plus per-module score/status and the graduation counters. The weekly job also folds it into the step summary. It lives in scripts/lib/ rather than the mutation tree so #1430 can adopt it for the other scheduled lanes instead of re-deriving field names.

On readiness: agreed that the PR job's "no affected kernels" run proved nothing. This push touches the kernel registry, so the job runs real mutants on it. Locally pnpm mutation:run --modules daemon-ref-frame kills 55/55 in 1m05s, and the merged five-shard sweep (2,054 mutants) is the committed baseline. The <30-minute weekly claim is still a projection from local shard timings — I can workflow_dispatch the weekly lane once this is on main and post the real wall clock.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Readiness evidence from this push's PR job (all 23 checks green): the affected lane selected kernel-errors from the registry diff and ran real mutants on an ubuntu runner — 183 mutants, 101 killed, 55.19%, held against the committed baseline, Stryker 7m16s / 557s lane total. So the ubuntu-runner score reproduces the locally-recorded baseline exactly, which is the cross-machine determinism the ratchet depends on.

That also gives a real per-shard data point for the weekly budget: 183 mutants ≈ 7m16s here, and the largest shard (selectors, 1,277 mutants) is the one the sharding exists for. I'll dispatch the weekly lane after merge and post the actual per-shard wall clock rather than leaving the <30-min claim as a projection.

@thymikee

Copy link
Copy Markdown
Member

Re-review at head a19412b: the bench:help-conformance script is fixed and checks are green, but two P1 blockers remain. First, affected test ownership is still a hand-picked list rather than derived/comprehensive: for example src/tests/daemon-error.test.ts directly exercises normalizeError from src/kernel/errors.ts but is absent from both the kernel-errors test ownership and workflow paths. Weakening that test skips the affected mutation run, and the current parity tests cannot detect omissions. Second, issue 1412 requires issue 1430 health/freshness monitoring to land with this new scheduled lane. The envelope is only written after report parsing and ratchet evaluation, so setup/Stryker/report failures can still produce no envelope, and there is no derived cadence monitor to detect a dark lane. Fix ownership derivation/completeness and land the shared health path before readiness.

thymikee and others added 4 commits July 27, 2026 16:30
Adds a Stryker (vitest runner) mutation lane scoped to the decision kernels,
a per-module baseline with tool/config provenance, and a ratchet that only
lets scores rise. Non-gating until two consecutive stable weekly sweeps.

Refs #1415

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…lysis

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ne envelope

- restore bench:help-conformance, broken by a formatting-path edit
- kernel test files select their module on PRs (registry `tests` + workflow paths),
  asserted to reach the kernel through the import graph
- every mutation run writes the standard scheduled-lane artifact envelope
- move src/utils/__tests__/errors.test.ts beside its source per the mirror rule

Refs #1415, #1430

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… monitor

Ownership of a kernel's tests is now computed from the static import graph
(scripts/mutation/ownership.ts) instead of a hand-listed set, so a test that
reaches a kernel indirectly -- src/__tests__/daemon-error.test.ts through
src/daemon.ts -- selects that kernel on a PR. The PR lane triggers on every src
test and shards the derived modules, keeping wall clock at one module.

The lane envelope (#1430) is now written on every exit path with the stage it
reached, so a crash before any mutant runs is distinguishable from a lane that
never ran. Adds the derived cadence monitor (scripts/lane-health, daily
workflow): scheduled lanes are enumerated from .github/workflows/ and reported
dark, failing, or never-run against their own cron cadence.
@devin-ai-integration
devin-ai-integration Bot force-pushed the devin/1785158289-mutation-ratchet branch from a19412b to 8c1dbe7 Compare July 27, 2026 16:30
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Both remaining blockers addressed at 8c1dbe794.

1. Ownership is derived, not hand-picked. The registry's tests field is gone. scripts/mutation/ownership.ts computes ownership from the static import graph: a test owns every kernel its imports reach, transitively. Your example lands automatically —

ownersOf('src/__tests__/daemon-error.test.ts') -> ['kernel-errors']   # via src/daemon.ts

and ownership.test.ts asserts completeness rather than parity with a list: it enumerates the whole derived map (ownedTestFiles), recomputes reachability independently for every owning file, and requires each module to have owners. There is no list left to omit an entry from.

The honest consequence: reachability is a superset of kill-power, so the derived owner sets are large (kernel-errors 469 test files, the others ~320 each). Any src test change therefore selects several kernels. I took that over any narrowing heuristic, since a narrowing is exactly the false negative you flagged. To keep it affordable, the PR lane is now select → per-module shard matrix (--list-affected) → merged ratchet, so wall clock is one module rather than their sum, and the path filter is deliberately wide (src/**/*.test.ts) with the decision left to select. workflow.test.ts now asserts that width instead of enumerating test paths.

2. Envelope on every exit path + a cadence monitor. run.ts reads provenance/baseline before doing any work and writes the envelope from a finally, carrying the stage it died in (select / stryker / report / ratchet / complete) and the error message. Covered by an integration test that runs the real CLI against a missing report and asserts result: fail, stage: report, plus intact tool/config provenance.

For the dark-lane half I landed the shared path rather than deferring to #1430: scripts/lane-health/ + a daily lane-health.yml. The lane list is derived from .github/workflows/ (so this PR's own weekly lane is watched without being listed), cadence comes from each lane's own cron, and a lane is reported dark after two missed cadences, failing after two consecutive non-success scheduled runs (cancellations excluded), or never-run once it has been declared longer than two cadences without firing. Unhealthy lanes ping one tracking issue; the job never fails, because a second red lane nobody watches isn't an alert.

Run against this repo just now, it immediately found a real one:

| `replays-nightly.yml`         | failing | two consecutive scheduled runs concluded failure, failure |
| `mutation-weekly.yml`         | fresh   | declared recently, awaiting its first run                 |
| `conformance-differential.yml`| fresh   | last scheduled run 2026-07-27T06:12:51Z                  |

Gates green locally: format:check, lint, typecheck, fallow, check:production-exports, pnpm mutation:test (33 cases), pnpm lane-health:test (7 cases). Rebased on latest main. Still outstanding by construction: the <30-min weekly figure remains a projection from per-shard timings until the first real scheduled sweep, which I'll dispatch and report after merge.

The shard artifacts now carry the lane envelope beside mutation.json, and the
merge globbed every .json under the download path, so the ratchet job fed the
envelope to the report parser and died after the mutants had already run.

@thymikee thymikee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at da4bf16. The derived ownership change closes the prior hand-picked-test omission, including daemon-error.test.ts, and the current GitHub checks plus the focused ownership/workflow/envelope/health tests are green.

Two blockers remain:

  • [P1] Reject incomplete shard sets before writing the aggregate envelope. readShardedReports only requires at least one mutation.json, while summarizeReport creates a zero score for every requested module absent from the downloaded reports. While gating is false, that becomes a non-failing regression, so the ratchet can write a complete/pass aggregate envelope even though a matrix shard failed. Validate that every requested module has a real report/nonzero mutant coverage, or otherwise propagate the shard result, and cover a partial-shard failure.

  • [P1] Extend the envelope boundary to genuine early failures. Argument parsing and provenance/baseline reads occur before the try/finally, so malformed or missing configuration still exits without an envelope. The weekly workflow also runs mutation:test before invoking the runner, so a self-test failure produces no aggregate lane envelope. Ensure these pre-state and pre-run failures create and upload a fail envelope, with tests that exercise them.

Residual risk, not a blocker: issue #1415 requires the weekly lane to complete in under 30 minutes, but the first real weekly run has not happened and shard timeouts remain 45 minutes. Please capture the first scheduled or manual weekly timing as follow-up evidence.

…lures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Both P1s fixed in 0218d2174.

[P1] Incomplete shard sets no longer pass. The aggregate now asserts, after scoring, that every requested module actually reported mutants — a module scored from no report is a dead shard, not a 0% score:

Incomplete shard set from .tmp/mutation/shards: no mutants for daemon-ref-frame.
A shard job failed or its artifact is absent; the aggregate is not a sweep.

That throws before evaluateRatchet, so nothing can write a complete/pass envelope or advance stableRuns off a partial matrix, independent of gating. Covered by a partial-shard test that writes one shard, requests two modules, and asserts exit≠0 plus a fail envelope at stage ratchet.

[P1] The envelope boundary now covers pre-state and pre-run failures. Argument parsing, readProvenance(), and readBaseline() moved inside the try/finally (initial stage is setup, provenance defaults to unknown instead of the envelope being skipped, and the commit read tolerates failure), so a bad --modules, an unreadable stryker.config.json, or a malformed baseline all emit a fail envelope naming the error.

For steps that run outside the runner — the weekly pnpm mutation:test, the affected select job, the artifact download — there's now --fail-envelope <reason>, invoked from if: failure() in both workflows and uploaded like any other envelope. It writes only when no envelope exists, so a genuine ratchet verdict is never clobbered by the generic recovery step. Tests cover the malformed invocation, the --fail-envelope path, and the no-clobber rule; a workflow assertion pins both lanes to having a recovery step.

Gates: format:check, lint, typecheck, check:fallow, 39 mutation self-tests green.

On the 30-minute budget — agreed, still unproven. The 45-minute shard timeout is deliberate headroom so a slow runner reports instead of being killed; I'll dispatch the weekly lane after merge and post real per-shard wall clock as follow-up evidence.

@thymikee thymikee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review confirms the missing-shard and early setup-envelope blockers are fixed and tested. One lane-result bug remains: the weekly ratchet can write a pass envelope, then cp/git checkout can fail; the if: failure() fallback calls --fail-envelope, but onlyIfAbsent preserves any existing envelope, so the failed scheduled job is published as passing. Preserve an existing failure, but upgrade an existing pass to fail for a later workflow failure, and add a pass-then-post-step-failure regression.

All head checks are green and the focused suite passes; the kernel-errors shard took ~10m38, but no full Mutation Weekly run exists, so the <30m budget remains unproven residual risk.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Good catch on the pass-then-post-step-failure hole — fixed in f6fbba1c7.

--fail-envelope is now asymmetric: it preserves an existing envelope only when that envelope already reports fail (the specific reason beats the generic recovery one), and it overwrites an existing pass, because a pass written before a later step failed is exactly the lie the envelope exists to prevent. Absent or unreadable envelopes are written as before.

The regression test does the real sequence: a passing --report-dir ratchet writes a pass envelope, then --fail-envelope "the baseline copy step failed" runs, and the envelope must end up result: fail carrying that reason — paired with the existing preserve-the-first-failure test on the other side. 40 mutation self-tests, format:check/lint/typecheck/check:fallow green.

Agreed on the residual risk: <30m stays unproven until a real Mutation Weekly run exists. The ~10m38 kernel-errors shard is the only datapoint; selectors is the long pole (~1,280 mutants). I'll dispatch the lane after merge and post per-shard wall clock.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 27, 2026

@thymikee thymikee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the latest delta: the later-step failure bug is fixed. Recovery preserves an existing real failure, but upgrades an existing pass to fail; targeted pass-then-failure coverage passes, and all head checks are green. No new code finding.

Residual evidence: the kernel-errors shard ran ~9m04, but Mutation Weekly has never run, so the full-lane <30m budget is not yet empirically proven.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Thanks — nothing further to change then.

On the residual: workflow_dispatch only exists once the workflow is on the default branch, so the full lane genuinely cannot be timed before merge. As soon as it lands I'll dispatch Mutation Weekly and post per-shard wall clock here; selectors (~1,280 mutants) is the long pole, and the 45-minute shard timeout is deliberate headroom so a slow runner reports instead of being killed mid-sweep. If the real numbers land anywhere near the 30-minute budget, the fix is more shards (the registry already drives the matrix), not a looser budget.

@thymikee

Copy link
Copy Markdown
Member

Reviewed against #1415 including the 2026-07-27 amendments. Approve with changes — this is real StrykerJS (@stryker-mutator/core + vitest runner; the 1,404-line lockfile diff is the genuine transitive tree, not a custom engine), scoped to exactly the five kernels with the scope asserted at three independent points, non-gating with the ratchet owning verdicts, and both amendments implemented precisely (strykerVersion + configHash per baseline entry; provenance drift classified separately from regression; graduation criteria written down AND mechanically enforced via requiredStableRuns). Test hygiene is clean — CLI-as-black-box, no DI seams, no suppressions, 2 lines of src/ churn with zero behavior change.

Three things before merge:

  1. P2 — strip or explicitly disclose the bundled obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 watcher. scripts/lane-health/* + .github/workflows/lane-health.yml (472 lines, with permissions: issues: write) is issue obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430's health-job deliverable. This is now the THIRD parallel implementation of that watcher across this PR wave (test: nightly parser fuzz lane — parser input fails as typed AppErrors, never hangs (#1414) #1438: scripts/scheduled-lane/, test(daemon): seeded concurrency torture lane for session/lease/lock invariants #1439: scripts/scheduled-lane-health/). One dedicated obs: scheduled-lane health, freshness telemetry, and standard artifact envelope #1430 PR must consolidate; strip the watcher here. Keep scripts/lib/lane-envelope.ts — it's the cleanest of the four envelope writers in this wave and the natural shared module for the others to rebase onto.
  2. P2 — the selectors shard will likely blow its budget. The only real CI datapoint — Mutants (kernel-errors): 9m04s for 183 mutants on a 2-core runner — extrapolates to ~45–55 min for the 1,277-mutant selectors shard, past both the <30-min acceptance criterion and the shard's own 45-min timeout. Run one manual workflow_dispatch of mutation-weekly.yml before relying on the Sunday schedule, and re-shard or trim mutate globs if the projection holds.
  3. P2 — the PR-triggered mutation pipeline (mutation-affected.yml) is live today, dormant-by-baseline but already costing real minutes (this PR's own 9m04s run) before graduation provides any value. The issue's out-of-scope line was "gating PRs on mutation score"; shipping the pipeline early is defensible prove-the-gate engineering, but it's a spend decision — needs an explicit maintainer yes, or a path filter that keeps it off until stableRuns >= 2.

P3: the deliberately-missing tsconfig.stryker-absent.json trick relies on undocumented Stryker preprocessor behavior — add a comment linking the Stryker source/issue that makes it intentional, for the next person and the next version bump.

🤖 Review by Claude Code

…ndled watcher

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

All four addressed in 390cc39c2 + c60890abc.

1. Watcher stripped. scripts/lane-health/* and .github/workflows/lane-health.yml are gone (with their package scripts, format paths, and the docs section), so nothing in this PR claims #1430's health job. scripts/lib/lane-envelope.ts stays as the shared writer for #1438/#1439 to rebase onto.

2. Sharding is now registry data, and it slices modules rather than just splitting the matrix. A module may declare shards: n; --shard i/n packs its expanded sources into n size-balanced bins and mutates one bin. selectors declares 4, so the weekly matrix is 8 jobs, and timeout-minutes drops 45 → 30 everywhere with a test asserting no mutation job declares more — the acceptance criterion is now mechanical rather than a note. The aggregate also requires the whole set (--expect-shards 8), because sub-shards make "every module has some mutants" too weak on its own.

Exercised end to end locally: pnpm mutation:run --modules selectors --shard 1/4 → 2 sources, 393 mutants, 74.81%, 4m04 on 8 cores (~2m of that is the shared dry run). Against your 9m04/183 datapoint the 2-core projection per selectors shard is ~15–20 min including the dry run — inside the budget with room, but still a projection: workflow_dispatch needs the workflow on the default branch, so the first real full-sweep timing lands after merge and I'll post per-shard wall clock then. If a shard does run hot, raising shards is a one-line registry edit that both workflows and the runner pick up.

3. The PR lane no longer spends anything before graduation. --list-affected returns [] while the baseline says gating: false, so no mutant job starts; --affected applies the same rule so the ratchet job can't spend what select declined. The one exception is a diff touching the lane's own tooling (scripts/mutation/, scripts/lib/, stryker.config.json, mutation-baselines/) — that's the case where the pre-graduation run buys something, since the gate has to be proven before it starts failing PRs, and it's exactly this PR. No maintainer spend decision needed: after graduation the lane switches itself on.

4. P3 documented. The tsconfigFile_comment now names the upstream file the behaviour comes from (packages/core/src/sandbox/ts-config-preprocessor.ts) plus the config-option docs, and says what to do if a future Stryker errors on a missing tsconfig instead of no-opping (point it at a real, extends-free tsconfig).

One honest side effect: that comment changes the config content hash, so the first weekly sweep reports provenance-drift and re-bases the baseline before the graduation clock starts — which is the mechanism doing its job, not something to paper over by hand-editing the recorded hash.

@thymikee thymikee removed the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 28, 2026
@thymikee

Copy link
Copy Markdown
Member

P1 — The pre-graduation tooling exception is currently vacuous for a tooling-only diff. affectedMatrix() detects touchesLane, but then returns shardMatrix(derivedAffectedModules(changed, ...)); files under scripts/mutation/**, scripts/lib/**, stryker.config.json, or mutation-baselines/** own no decision-kernel module, so the matrix is [] and no real mutants run. The current PR only selected kernel-errors because its broader diff also touches an owning test. When touchesLane, return an intentional real-mutant matrix and add an end-to-end --list-affected regression for a tooling-only git diff. Current workflow path-filter tests do not prove that behavior.

…mutants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Re: the P1 on the vacuous tooling exception — correct, and thanks; it was decorative exactly as you describe. Fixed in 0ce6b338f.

affectedMatrixFor() now adds LANE_CANARY (kernel-errors — the registry's cheapest real sweep, one file / ~183 mutants) whenever touchesLane, on top of whatever the diff derives:

if (!gating && !touchesLane) return [];
const modules = new Set(derivedAffectedModules(changed, root));
if (touchesLane) modules.add(LANE_CANARY);   // lane sources own no kernel

So a tooling-only diff proves the gate on real mutants at one shard's cost, not on an empty matrix.

The regression is end to end rather than a path-filter assertion: scripts/mutation/selection.test.ts creates a throwaway git worktree, commits a change to exactly one file, and runs the real run.ts --list-affected --base HEAD~1 in it —

  • scripts/mutation/ratchet.ts only → [{ name: 'kernel-errors', module: 'kernel-errors' }] (the case that returned [] before);
  • src/utils/scroll-edge-state.ts only → [] while gating: false, and shardMatrix(['scroll-edge-state']) once gating is on;
  • docs-only → [] even when gating.

45 mutation self-tests, plus format/lint/typecheck/fallow green.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 28, 2026
@thymikee

Copy link
Copy Markdown
Member

Re-reviewed 7a8ea6d: tooling-only diffs now select the registry-declared kernel-errors canary, and the new throwaway-worktree test exercises the actual --list-affected CLI path non-vacuously. The affected canary shard and all current checks pass; merge state is CLEAN and no code-review findings remain. Residual: the first full Mutation Weekly run still needs to demonstrate the <30-minute full-lane target.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Thanks — agreed on the residual. The <30-min full-lane figure stays a projection until the workflow is on the default branch (workflow_dispatch can't reach it from a PR branch); once this merges I'll dispatch Mutation Weekly and post per-shard wall clock, and raise shards in the registry if any shard runs hot.

@thymikee
thymikee merged commit 8cce0ef into main Jul 28, 2026
28 checks passed
@thymikee
thymikee deleted the devin/1785158289-mutation-ratchet branch July 28, 2026 08:07
devin-ai-integration Bot added a commit that referenced this pull request Jul 28, 2026
Rebase onto main (post-#1441) and replace the lane-local LaneEnvelope
dialect with the shared scripts/lib/lane-envelope.ts builder, so the
#1430 health watcher parses one schema: commitSha->commit, sourceHash->
configHash, seedRange/runs moved into the typed data payload, and the
sweep encoded as seed "<start>-<end>".

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
thymikee added a commit that referenced this pull request Jul 28, 2026
…invariants (#1439)

* test(daemon): seeded concurrency torture lane for session/lease/lock invariants

Refs #1416

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): drive torture lane through real lock plan + review fixes

- derive each op's lock plan from production resolveRequestExecutionLockKeys
  via a fake device-inventory provider, so reverting the router's same-device
  serialization trips the overlap invariant (verified)
- assert exact replay: full scheduler trace, terminal outcome, contention
- assert real same-device lock contention in the sweep + a forced 2-client case
- split harness into bindings/invariants/envelope modules (all <500 LOC)
- emit #1430 scheduled-lane envelope (schema/SHA/hash/seed range/duration/result)
  and upload it from the nightly workflow

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): pass claim data as plain view accessors (fallow)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): gate lock plan on shouldLockSessionExecution; sweep replay + forced-device contention; whole-lane envelope

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): add real-scope runLocked serialization guard; whole-lane envelope duration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): allow seed 0 replay; add seed-0 regression (TORTURE_SEED must accept 0)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* obs(#1430): add scheduled-lane freshness/cadence health watcher

Discovers schedule: workflows from .github/workflows/, reads recent
scheduled runs via the GitHub API, and opens/pings a tracking issue when
a lane misses or fails two consecutive cadences. Pure model unit-tested
and gated on PRs; API I/O + issue open/ping run nightly.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): give scheduled-lane watcher a two-cadence newborn grace

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): anchor lane grace on schedule-introduction, not workflow age

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): derive schedule-activation semantically via git, through runCmdSync

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): add merge-commit regression pinning first-parent + committer time

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): pin scheduled-lane-health issue-write route via stubbed fetch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): unbundle #1430 watcher; make torture lane nightly-only

Strip the scheduled-lane-health watcher (scripts + workflow + PR gate) —
it is #1430's deliverable and collides with PR #1438's workflow of the
same filename; keep only this lane's #1430 envelope writer.

Move the torture lane under test/integration/nightly/ so it is out of the
test:integration:node glob, and run it via an explicit, disclosed PR step
plus the nightly sweep.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs: fix stale torture-lane paths after nightly/ move

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(daemon): adopt shared lane-envelope for torture lane

Rebase onto main (post-#1441) and replace the lane-local LaneEnvelope
dialect with the shared scripts/lib/lane-envelope.ts builder, so the
#1430 health watcher parses one schema: commitSha->commit, sourceHash->
configHash, seedRange/runs moved into the typed data payload, and the
sweep encoded as seed "<start>-<end>".

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: weekly mutation-score ratchet over decision kernels (Stryker, non-gating first)

1 participant