emrg: keep the Python test count in one doc, and guard that it stays there - #1177
emrg: keep the Python test count in one doc, and guard that it stays there#1177argszero wants to merge 1 commit into
Conversation
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260913-110421 (1/3)
Disclosure: I authored this PR, so this is not an independent review; treat it as the author's account of evidence measured this cycle on this head (7013d30b, base master 6456a98b, FRESH, CI double-green).
The defect, measured before the fix: DEVELOPMENT.md said uv run pytest tests/ -v # run tests (currently 681 items); the tree collected 1590 — off by 2.3x. Nothing reads that file. Both tests/test_doc_counts.py and scripts/check-doc-count.py parse Agent.md only, and a scan of all 16 tracked markdown files for a count claim found it in exactly two places: the guarded line, and that copy.
Why "one site" and not "guard the second site too": a second guarded copy is not only a second drift site, it is a second conflict site on the same line for every PR that adds a test. That cost is already measured in issue #1158: this cycle I read back, per PR, which file actually conflicts — all 10 CONFLICTING/DIRTY PRs conflict in Agent.md and nowhere else, and for #1136/#1153/#1170 the base/master/head values on that single line are three different numbers. So the change removes the copy and makes "exactly one doc states it" a rule, with Agent.md re-measured by the tool rather than by hand (1590 -> 1593, one token).
The guard is mutation-verified in both directions, and one mutant taught something worth writing down:
| Mutant | Result |
|---|---|
restore the stale (currently 681 items) line |
RED — test_the_python_test_count_is_stated_in_exactly_one_doc |
drop the pattern that sees Agent.md's line |
RED — test_the_single_count_source_really_states_the_count (the vacuity tripwire: without it, a rule that matches nothing reports "exactly one place" forever) |
drop the parenthesised count form |
RED — the synthetic positive control reports the wrong form |
The third mutant survived on its first cut, and that was an artifact, not a property of the guard: I cut it with sed '/parenthesised count/d', which also deleted the assertion line containing the same phrase, so the guard appeared to survive because its own check had been removed. Re-cut against the tuple alone and with the diff inspected before reading the result, it dies. A surviving mutant has to be confirmed as a mutant first.
Verification on this head: full suite 1592 passed, 1 skipped = 1593 collected == Agent.md (1593); check-doc-count.py → OK; from emrg.client.app import run_client and python -m emrg --help green; CI run 34734984555 double-green (test + test-windows).
What this does not do: it does not reduce the per-landing conflict on the count line — that stays exactly one line, and the queue still drains roughly one PR per generation. It only stops the number from spreading into places where nothing can keep it true.
|
Superseded by #1181 — closing rather than merging. This PR moved the Python test count into one document ("keep it in one place, and The measurement that decided it (2026-09-13): of 14 open PRs, 11 were Worth recording, because it is what this PR got right and #1181 keeps: the |
The finding
DEVELOPMENT.mdadvertiseduv run pytest tests/ -v # run tests (currently 681 items). Measured on master6456a98: the tree collects 1590 — the claim is off by 2.3x, and nothing reads that file.tests/test_doc_counts.pyandscripts/check-doc-count.pyboth parseAgent.mdonly. Scanning all 16 tracked markdown files for a count claim found the number in exactly two places: one guarded, one not.A derived number written into prose with no parser is a number nobody can keep true. It is also the same shape as what issue #1158 measures, one level down: the count is a global derived fact, so every copy of it is also a copy of the conflict.
The change
DEVELOPMENT.md— keep the command, drop the number, point at the guarded site:# run tests (the count lives in Agent.md; CI keeps it true).tests/test_doc_counts.py— new guard: no tracked markdown other thanAgent.mdmay state the Python test count. Four written forms are recognised (parenthesised count,currently N, bareN items, a count after apytestcommand). Measured against all 16 tracked docs: zero hits outside the canonical line, so the rule is precise rather than noise-limited.Agent.mdmust still be matched by the rule. Without it the scan is vacuous in the quiet direction: a rule that matches nothing reports "exactly one place" forever, and the dayAgent.md's count line changes shape the guard is green while nothing states the number.Agent.mdre-measured by the tool, not by hand:scripts/check-doc-count.py --write→1590→1593(three new tests), single-token replacement.Why "one site" rather than "guard the second site too": a second guarded copy would add a second conflict site on the same line for every PR that adds a test. The measured cost of one such line is already in issue #1158 — 9 of 10 open PRs conflicted on it, and resolving it is what the recent cycles' pushes have all been.
README.md/README.cn.mdalready took this route (Tests badge, rant 2026-08-11T19:50:37).Mutation verification
Each mutant was run on this branch and the branch was green again after restore:
(currently 681 items)linetest_the_python_test_count_is_stated_in_exactly_one_docAgent.md's linetest_the_single_count_source_really_states_the_count(the vacuity tripwire)parenthesised countpatternNote on the third: my first cut of that mutant was
sed '/parenthesised count/d', which also deleted the assertion containing the same phrase — the guard then appeared to survive because its own check had been removed. The mutant was re-cut against the tuple alone and itsdiffinspected before the result was read: a surviving mutant must be confirmed as a mutant first, or it is an artifact of how it was cut.Verification
1592 passed, 1 skipped= 1593 collected == Agent.md (1593);check-doc-count.py→OKfrom emrg.client.app import run_clientandpython -m emrg --helpgreengit ls-filesis used for the file list (index, not a walk) with explicit UTF-8 and separator normalisation, following thewindows-2025lessons recorded intest_conflict_markers.py