Skip to content

emrg: keep the Python test count in one doc, and guard that it stays there - #1177

Closed
argszero wants to merge 1 commit into
masterfrom
feature/single-count-source
Closed

emrg: keep the Python test count in one doc, and guard that it stays there#1177
argszero wants to merge 1 commit into
masterfrom
feature/single-count-source

Conversation

@argszero

Copy link
Copy Markdown
Owner

The finding

DEVELOPMENT.md advertised uv run pytest tests/ -v # run tests (currently 681 items). Measured on master 6456a98: the tree collects 1590 — the claim is off by 2.3x, and nothing reads that file. tests/test_doc_counts.py and scripts/check-doc-count.py both parse Agent.md only. Scanning all 16 tracked markdown files for a count claim found the number in exactly two places: one guarded, one not.

A derived number written into prose with no parser is a number nobody can keep true. It is also the same shape as what issue #1158 measures, one level down: the count is a global derived fact, so every copy of it is also a copy of the conflict.

The change

  1. DEVELOPMENT.md — keep the command, drop the number, point at the guarded site: # run tests (the count lives in Agent.md; CI keeps it true).
  2. tests/test_doc_counts.py — new guard: no tracked markdown other than Agent.md may state the Python test count. Four written forms are recognised (parenthesised count, currently N, bare N items, a count after a pytest command). Measured against all 16 tracked docs: zero hits outside the canonical line, so the rule is precise rather than noise-limited.
  3. The guard has a device test of itselfAgent.md must still be matched by the rule. Without it the scan is vacuous in the quiet direction: a rule that matches nothing reports "exactly one place" forever, and the day Agent.md's count line changes shape the guard is green while nothing states the number.
  4. Agent.md re-measured by the tool, not by hand: scripts/check-doc-count.py --write15901593 (three new tests), single-token replacement.

Why "one site" rather than "guard the second site too": a second guarded copy would add a second conflict site on the same line for every PR that adds a test. The measured cost of one such line is already in issue #1158 — 9 of 10 open PRs conflicted on it, and resolving it is what the recent cycles' pushes have all been. README.md/README.cn.md already took this route (Tests badge, rant 2026-08-11T19:50:37).

Mutation verification

Each mutant was run on this branch and the branch was green again after restore:

Mutant Result
restore the stale (currently 681 items) line RED — test_the_python_test_count_is_stated_in_exactly_one_doc
drop the pattern that sees Agent.md's line RED — test_the_single_count_source_really_states_the_count (the vacuity tripwire)
drop the parenthesised count pattern RED — the synthetic positive control reports the wrong form

Note on the third: my first cut of that mutant was sed '/parenthesised count/d', which also deleted the assertion containing the same phrase — the guard then appeared to survive because its own check had been removed. The mutant was re-cut against the tuple alone and its diff inspected before the result was read: a surviving mutant must be confirmed as a mutant first, or it is an artifact of how it was cut.

Verification

  • full suite: 1592 passed, 1 skipped = 1593 collected == Agent.md (1593); check-doc-count.pyOK
  • from emrg.client.app import run_client and python -m emrg --help green
  • git ls-files is used for the file list (index, not a walk) with explicit UTF-8 and separator normalisation, following the windows-2025 lessons recorded in test_conflict_markers.py

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260913-110421 (1/3)

Disclosure: I authored this PR, so this is not an independent review; treat it as the author's account of evidence measured this cycle on this head (7013d30b, base master 6456a98b, FRESH, CI double-green).

The defect, measured before the fix: DEVELOPMENT.md said uv run pytest tests/ -v # run tests (currently 681 items); the tree collected 1590 — off by 2.3x. Nothing reads that file. Both tests/test_doc_counts.py and scripts/check-doc-count.py parse Agent.md only, and a scan of all 16 tracked markdown files for a count claim found it in exactly two places: the guarded line, and that copy.

Why "one site" and not "guard the second site too": a second guarded copy is not only a second drift site, it is a second conflict site on the same line for every PR that adds a test. That cost is already measured in issue #1158: this cycle I read back, per PR, which file actually conflicts — all 10 CONFLICTING/DIRTY PRs conflict in Agent.md and nowhere else, and for #1136/#1153/#1170 the base/master/head values on that single line are three different numbers. So the change removes the copy and makes "exactly one doc states it" a rule, with Agent.md re-measured by the tool rather than by hand (1590 -> 1593, one token).

The guard is mutation-verified in both directions, and one mutant taught something worth writing down:

Mutant Result
restore the stale (currently 681 items) line RED — test_the_python_test_count_is_stated_in_exactly_one_doc
drop the pattern that sees Agent.md's line RED — test_the_single_count_source_really_states_the_count (the vacuity tripwire: without it, a rule that matches nothing reports "exactly one place" forever)
drop the parenthesised count form RED — the synthetic positive control reports the wrong form

The third mutant survived on its first cut, and that was an artifact, not a property of the guard: I cut it with sed '/parenthesised count/d', which also deleted the assertion line containing the same phrase, so the guard appeared to survive because its own check had been removed. Re-cut against the tuple alone and with the diff inspected before reading the result, it dies. A surviving mutant has to be confirmed as a mutant first.

Verification on this head: full suite 1592 passed, 1 skipped = 1593 collected == Agent.md (1593); check-doc-count.pyOK; from emrg.client.app import run_client and python -m emrg --help green; CI run 34734984555 double-green (test + test-windows).

What this does not do: it does not reduce the per-landing conflict on the count line — that stays exactly one line, and the queue still drains roughly one PR per generation. It only stops the number from spreading into places where nothing can keep it true.

@argszero

Copy link
Copy Markdown
Owner Author

Superseded by #1181 — closing rather than merging.

This PR moved the Python test count into one document ("keep it in one place, and
guard that it stays there"). #1181 removes the last copy instead, so there is no
place to keep it in: the count is a measurement, and a stored derived number is
both a conflict magnet and a silent-corruption site.

The measurement that decided it (2026-09-13): of 14 open PRs, 11 were
conflicting, and all 11 conflicted on that single line — every PR that adds a
test had to rewrite the same number. Two of them writing the same value merged
cleanly and left a merged tree that failed the guard with no conflict marker
anywhere (#1179 -> #1180: both wrote 1601, the merged tree collected 1603). A
better guard cannot fix that; not storing the number does.

Worth recording, because it is what this PR got right and #1181 keeps: the
DEVELOPMENT.md line you measured ("currently 681 items" against 1590 collected,
off by 2.3x, in a file no check read) is exactly the second copy #1181's rule
looks for, and its prose shape is now one of the rule's clauses — a rule keyed on
the number could never have found a claim that was wrong.

@argszero argszero closed this Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant