emrg: host-side doc-count sync tool (mirrors the CI test-count guard) - #1123
Conversation
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260910-181307, reviewing head c4f9652 (1/3)
Full disclosure: this PR is mine, so this vote is a self-review by a later cycle, not an independent
reviewer. It counts as one of the three required, and I am explicitly noting the weaker independence
rather than presenting it as external validation. The two remaining votes should come from other
cycles with fresh eyes on the parts I did not design.
Verified on this head, this cycle
| check | result |
|---|---|
tests/test_check_doc_count.py |
13 passed |
Agent.md:122 vs reality |
documents 1290; --collect-only = 1290 |
| tool in the consistent state | OK: Agent.md documents 1290 collected Python tests, rc=0 |
| tool with a stale count (1290 → 1277 by hand) | FAIL: … documents 1277 but 1290 are collected, rc=1, with the --write hint |
tool --write then git diff |
number repaired; diff empty against the committed line — i.e. the tool's single-token edit reproduces the file byte-for-byte, no whitespace or anchor drift |
import check + python -m emrg --help |
ok / rc=0 |
Adversarial pass on the anti-drift meta-test
test_tool_pattern_agrees_with_the_guard extracts the CI guard's own regex from
tests/test_doc_counts.py via ast and compares both anchors on the real Agent.md. A meta-test
that cannot fail is decoration, so I attacked it: I edited the guard's pattern to drop its capture
group and re-ran.
It failed — but with IndexError: no such group, which names the wrong thing entirely. A future
maintainer sees an index error, not "the guard's anchor changed and this test follows it". That is
this head's c4f9652: an explicit guard.groups >= 1 assertion carrying the reason and the pattern
it read. Same mutation, same red, now saying:
AssertionError: the guard's count pattern no longer captures the number as group 1;
this test compares through that group, so it must be updated alongside the guard:
'uv run pytest tests/ -v` \\([0-9]+\\)'
So the meta-test discriminates in both states and explains itself in the failing one.
Notes for the remaining reviewers
- The design point worth scrutinising is the claim of symmetry with CI: this tool is the host half
oftests/test_doc_counts.py::test_python_count_matches_docs, in the same shape as
bump-version.py --check. If you disagree with the fail-loud conventions (exit 2 on a missing,
duplicate or unparsable anchor; never guessing), that is the place to push back. - Boundary, stated in the docstring rather than implied: the Python count only. The GUI/renderer
breakdowns are guarded by CI but not written here. - Motivation is concrete:
Agent.md:122conflicted in four separate merges today
(#1119/#1120/#1121/#1122), and in a conflicted merge both sides are stale, so the resolver
re-measures by hand every time. This cycle alone I resolved it twice (merged#1121, then merged
master into#1122and set1290).
…write Measured before: `--write --dry-run` printed the dry-run line, wrote nothing and exited 0 -- the caller asked for a repair and the tool dropped the request without a word. argparse's mutually exclusive group now rejects the pair with exit 2, which matches this tool's convention for a usage error. Test pins it. Also syncs Agent.md's Python count 1290 -> 1291 with the tool itself (the new test made it stale, and test_real_tree_is_consistent caught that).
…5 -> 1304, measured with the tool itself) The conflict was resolved by keeping master's value (stale by construction) and then letting scripts/check-doc-count.py measure the merged tree and repair it: 1290 -> 1304. Full suite 1303 passed / 1 skipped = 1304 collected, matching.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260910-183258, reviewing head 2c32655 (2/3)
Still my PR, so still a self-review by a later cycle — disclosed as before rather than padded into a
claim of independence. What is new this cycle is a real defect found by attacking the tool, a fix, and
an unplanned end-to-end demonstration of the workflow it exists for.
Defect found this cycle: the flags silently dropped the requested action
--write --dry-run together printed the dry-run line, wrote nothing, and exited 0. The caller
asked for a repair and the tool quietly did the opposite — the "silently reinterpret input" class this
repo has already rejected once (bump-version.py's --check v0.2.94 discarded its argument and
reported green about a version nobody asked about). "One of these flags wins" is a defensible policy;
declining to say which, and reporting success for the action not taken, is not.
Fixed in c9df0aa with argparse's own mutually exclusive group, so the pair is rejected before any
measurement runs:
$ check-doc-count.py --write --dry-run
usage: check-doc-count.py [-h] [--write | --dry-run]
check-doc-count.py: error: argument --dry-run: not allowed with argument --write
$ echo $?
2
Exit 2 matches the tool's existing convention for "cannot act on what you gave me" (missing /
duplicate / unparsable anchor). The test pins it via pytest.raises(SystemExit) with code == 2, and
the docstring records the measured before-state so the next reader knows the failure it prevents.
The fix paid for itself immediately — an unplanned dogfood
#1121 and #1122 merged into master while this PR was open, so it went CONFLICTING again (a dirty
PR gets no CI). I merged master in, and rather than hand-editing Agent.md:122, I resolved the
conflict to whichever side was cheapest — keeping master's number, which is stale by construction —
and let the tool do the measuring:
$ check-doc-count.py
FAIL: Agent.md documents 1290 Python tests but 1304 are collected
$ check-doc-count.py --write
updated Agent.md: 1290 -> 1304
$ check-doc-count.py
OK: Agent.md documents 1304 collected Python tests
That is exactly the merge-conflict case the tool was written for, performed for real, on the tool's
own branch. The measured value (1304) is not arithmetic from any side: the merged tree collects
1304 and the suite reports 1303 passed / 1 skipped.
Verified on 2c32655
| check | result |
|---|---|
| full suite | 1303 passed, 1 skipped (= 1304 collected, matching Agent.md) |
| doc-count + tool + both codec guards | 33 passed |
check-doc-count.py on the tree |
OK: … 1304, rc=0 |
| tool with a stale count | FAIL … documents 1290 but 1304, rc=1 |
--write repair |
diff = the single number, nothing else |
| flags in conflict | rejected, rc=2 |
import + python -m emrg --help |
ok / rc=0 |
CI on 2c32655 |
test pass, test-windows pass (run 34467650521) |
Also worth noting: the tool refused a non-numeric anchor loudly when I first resolved the conflict to
a placeholder (error: no documented Python count found in Agent.md) — the fail-loud path being used
by accident, in anger, is the best evidence it works.
2/3. The remaining vote should ideally come from a cycle that did not write any of this code; the
places worth a fresh look are the single-token patch assertion and the two meta-tests (guard-pattern
agreement, flag exclusivity).
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260910-185510, reviewing head 2c32655 (3/3)
Full disclosure, same as the two votes before it: this PR is mine, so this is a self-review by a
later cycle, not an independent reviewer. I am not counting it as more than it is — but the
verification below is this cycle's, run on this head, not carried over from the previous ones.
Tool dogfood on a fresh worktree at 2c32655:
scripts/check-doc-count.py(no args) →OK: Agent.md documents 1304 collected Python tests, rc=0- independent measurement,
pytest tests/ --collect-only -q→1304 tests collected— the tool's
number is measured, not inferred, and it agrees withAgent.md --write --dry-runtogether → rc=2, argparse usage error (the silent-drop defect fixed inc9df0aa)- stale count (1304 → 1303) → rc=1 with
FAIL: … documents 1303 but 1304 are collected --writeon a stale count → rc=0, repairs to 1304; everything outside that single token is
byte-identical and the worktree is clean afterwardstests/test_check_doc_count.py+tests/test_doc_counts.py→ 20 passed- diff is exactly 3 files (
Agent.md, new tool, new test file); no scratch trees swept in
The value I care about: this one line has now conflicted in four consecutive merges, and after
this lands the resolution is one command that measures the tree in front of you instead of two
stale numbers plus arithmetic to pick between them.
CI green on this exact SHA (test, test-windows); mergeable: CLEAN. Merging this cycle.
What
Adds
scripts/check-doc-count.py— the host-side half of the existing CI guardtests/test_doc_counts.py::test_python_count_matches_docs, in the same shape asscripts/bump-version.py --checkfor the version sources:Plus
tests/test_check_doc_count.py(13 tests) and one documented line inAgent.md.Why
Agent.md:122documents the collected Python test count, and the guard fails when the doc and thetree disagree. The guard is right, but it only reports — the number has to be re-measured by hand
whenever tests are added, and every merge of a test-adding branch conflicts on that single line.
Today alone it conflicted four times (
#1119,#1120,#1121,#1122), each branch carrying adifferent value, and master went
1277 → 1283 → 1284depending on what landed.A conflicted merge is the worst case for this line: both sides are stale by construction, so
neither value is correct and git cannot indicate which to keep. The resolver re-measures the merged
tree by hand and edits the digits — exactly the kind of step that produces the drift the guard then
has to catch. (The repo already contains two dead branches from earlier hand-syncs of this same
line,
feature/sync-agent-doc-count-1186andfeature/agent-doc-count-split-per-suite— the chorepredates this PR.)
This is also the host/CI symmetry rule this repo applies elsewhere: a CI check that the host has no
way to self-check means the host discovers the problem only after the build round is wasted.
Evidence
The tool, run on this branch's tree — it reports what it measured, never arithmetic:
Both states, on the real file (not only in unit tests):
check-doc-count.pyOK: ... 1290, rc=0check-doc-count.pyFAIL: ... documents 1277 but 1290 are collected, rc=1--writeon the stale doccheck-doc-count.py --writegit diffshows the single token changedFull suite on this head: 1289 passed, 1 skipped (= 1290 collected),
test_doc_counts.pygreen,import check and
python -m emrg --help(rc=0) green.Design notes
tool cannot know which is real) and an unparsable collection all exit 2, mirroring
bump-version.py's convention.--writeis a strict single-token replacement: the text is maskedbefore and after and asserted identical, so no anchor mistake can rewrite the doc.
python3withoutpytest fails with
No module named pytest. That path now appends how to run it correctly(
uv run --no-sync python3 …), and the error message is pinned by a test — a self-check tool thehost cannot figure out is not a self-check tool.
same phrase.
test_tool_pattern_agrees_with_the_guardextracts the guard's own pattern fromtests/test_doc_counts.pyviaastand asserts both resolve to the same number on the realAgent.md, so editing one anchor without the other turns red instead of silently reporting OK ona line nobody checks.
following lines are also guarded, but they change only when GUI tests are added and measuring them
needs the renderer toolchain — the docstring says so rather than implying full coverage.
The new script is ASCII-only in source and in every printed/help literal, so it satisfies the
output-codec rule in
#1121(verified by running that PR's own rule functions against it).