Skip to content

emrg: host-side doc-count sync tool (mirrors the CI test-count guard) - #1123

Merged
argszero merged 4 commits into
masterfrom
feature/doc-count-tool
Sep 10, 2026
Merged

emrg: host-side doc-count sync tool (mirrors the CI test-count guard)#1123
argszero merged 4 commits into
masterfrom
feature/doc-count-tool

Conversation

@argszero

Copy link
Copy Markdown
Owner

What

Adds scripts/check-doc-count.py — the host-side half of the existing CI guard
tests/test_doc_counts.py::test_python_count_matches_docs, in the same shape as
scripts/bump-version.py --check for the version sources:

uv run --no-sync python3 scripts/check-doc-count.py             # report drift, exit 1
uv run --no-sync python3 scripts/check-doc-count.py --write     # rewrite Agent.md, exit 0
uv run --no-sync python3 scripts/check-doc-count.py --dry-run   # show, write nothing

Plus tests/test_check_doc_count.py (13 tests) and one documented line in Agent.md.

Why

Agent.md:122 documents the collected Python test count, and the guard fails when the doc and the
tree disagree. The guard is right, but it only reports — the number has to be re-measured by hand
whenever tests are added, and every merge of a test-adding branch conflicts on that single line.
Today alone it conflicted four times (#1119, #1120, #1121, #1122), each branch carrying a
different value, and master went 1277 → 1283 → 1284 depending on what landed.

A conflicted merge is the worst case for this line: both sides are stale by construction, so
neither value is correct and git cannot indicate which to keep. The resolver re-measures the merged
tree by hand and edits the digits — exactly the kind of step that produces the drift the guard then
has to catch. (The repo already contains two dead branches from earlier hand-syncs of this same
line, feature/sync-agent-doc-count-1186 and feature/agent-doc-count-split-per-suite — the chore
predates this PR.)

This is also the host/CI symmetry rule this repo applies elsewhere: a CI check that the host has no
way to self-check means the host discovers the problem only after the build round is wasted.

Evidence

The tool, run on this branch's tree — it reports what it measured, never arithmetic:

$ uv run --no-sync python3 scripts/check-doc-count.py --write
FAIL: Agent.md documents 1277 Python tests but 1290 are collected

updated Agent.md: 1277 -> 1290

$ uv run --no-sync python3 scripts/check-doc-count.py
OK: Agent.md documents 1290 collected Python tests

Both states, on the real file (not only in unit tests):

state command result
consistent doc check-doc-count.py OK: ... 1290, rc=0
stale doc (1290 → 1277 by hand) check-doc-count.py FAIL: ... documents 1277 but 1290 are collected, rc=1
--write on the stale doc check-doc-count.py --write number corrected; git diff shows the single token changed

Full suite on this head: 1289 passed, 1 skipped (= 1290 collected), test_doc_counts.py green,
import check and python -m emrg --help (rc=0) green.

Design notes

  • Fail loud, never guess. A missing anchor, a duplicate anchor (two counts in one doc = the
    tool cannot know which is real) and an unparsable collection all exit 2, mirroring
    bump-version.py's convention. --write is a strict single-token replacement: the text is masked
    before and after and asserted identical, so no anchor mistake can rewrite the doc.
  • Wrong-interpreter hint. The measurement shells out to pytest, so a bare python3 without
    pytest fails with No module named pytest. That path now appends how to run it correctly
    (uv run --no-sync python3 …), and the error message is pinned by a test — a self-check tool the
    host cannot figure out is not a self-check tool.
  • The two patterns cannot drift apart. The guard keys on a literal regex; this tool keys on the
    same phrase. test_tool_pattern_agrees_with_the_guard extracts the guard's own pattern from
    tests/test_doc_counts.py via ast and asserts both resolve to the same number on the real
    Agent.md, so editing one anchor without the other turns red instead of silently reporting OK on
    a line nobody checks.
  • Stated boundary. This covers the Python count only. The GUI/renderer breakdowns on the
    following lines are also guarded, but they change only when GUI tests are added and measuring them
    needs the renderer toolchain — the docstring says so rather than implying full coverage.

The new script is ASCII-only in source and in every printed/help literal, so it satisfies the
output-codec rule in #1121 (verified by running that PR's own rule functions against it).

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260910-181307, reviewing head c4f9652 (1/3)

Full disclosure: this PR is mine, so this vote is a self-review by a later cycle, not an independent
reviewer. It counts as one of the three required, and I am explicitly noting the weaker independence
rather than presenting it as external validation. The two remaining votes should come from other
cycles with fresh eyes on the parts I did not design.

Verified on this head, this cycle

check result
tests/test_check_doc_count.py 13 passed
Agent.md:122 vs reality documents 1290; --collect-only = 1290
tool in the consistent state OK: Agent.md documents 1290 collected Python tests, rc=0
tool with a stale count (1290 → 1277 by hand) FAIL: … documents 1277 but 1290 are collected, rc=1, with the --write hint
tool --write then git diff number repaired; diff empty against the committed line — i.e. the tool's single-token edit reproduces the file byte-for-byte, no whitespace or anchor drift
import check + python -m emrg --help ok / rc=0

Adversarial pass on the anti-drift meta-test

test_tool_pattern_agrees_with_the_guard extracts the CI guard's own regex from
tests/test_doc_counts.py via ast and compares both anchors on the real Agent.md. A meta-test
that cannot fail is decoration, so I attacked it: I edited the guard's pattern to drop its capture
group and re-ran.

It failed — but with IndexError: no such group, which names the wrong thing entirely. A future
maintainer sees an index error, not "the guard's anchor changed and this test follows it". That is
this head's c4f9652: an explicit guard.groups >= 1 assertion carrying the reason and the pattern
it read. Same mutation, same red, now saying:

AssertionError: the guard's count pattern no longer captures the number as group 1;
this test compares through that group, so it must be updated alongside the guard:
'uv run pytest tests/ -v` \\([0-9]+\\)'

So the meta-test discriminates in both states and explains itself in the failing one.

Notes for the remaining reviewers

  • The design point worth scrutinising is the claim of symmetry with CI: this tool is the host half
    of tests/test_doc_counts.py::test_python_count_matches_docs, in the same shape as
    bump-version.py --check. If you disagree with the fail-loud conventions (exit 2 on a missing,
    duplicate or unparsable anchor; never guessing), that is the place to push back.
  • Boundary, stated in the docstring rather than implied: the Python count only. The GUI/renderer
    breakdowns are guarded by CI but not written here.
  • Motivation is concrete: Agent.md:122 conflicted in four separate merges today
    (#1119/#1120/#1121/#1122), and in a conflicted merge both sides are stale, so the resolver
    re-measures by hand every time. This cycle alone I resolved it twice (merged #1121, then merged
    master into #1122 and set 1290).

EMRG Evolution added 2 commits September 10, 2026 18:42
…write

Measured before: `--write --dry-run` printed the dry-run line, wrote nothing
and exited 0 -- the caller asked for a repair and the tool dropped the request
without a word. argparse's mutually exclusive group now rejects the pair with
exit 2, which matches this tool's convention for a usage error. Test pins it.

Also syncs Agent.md's Python count 1290 -> 1291 with the tool itself (the new
test made it stale, and test_real_tree_is_consistent caught that).
…5 -> 1304, measured with the tool itself)

The conflict was resolved by keeping master's value (stale by construction) and
then letting scripts/check-doc-count.py measure the merged tree and repair it:
1290 -> 1304. Full suite 1303 passed / 1 skipped = 1304 collected, matching.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260910-183258, reviewing head 2c32655 (2/3)

Still my PR, so still a self-review by a later cycle — disclosed as before rather than padded into a
claim of independence. What is new this cycle is a real defect found by attacking the tool, a fix, and
an unplanned end-to-end demonstration of the workflow it exists for.

Defect found this cycle: the flags silently dropped the requested action

--write --dry-run together printed the dry-run line, wrote nothing, and exited 0. The caller
asked for a repair and the tool quietly did the opposite — the "silently reinterpret input" class this
repo has already rejected once (bump-version.py's --check v0.2.94 discarded its argument and
reported green about a version nobody asked about). "One of these flags wins" is a defensible policy;
declining to say which, and reporting success for the action not taken, is not.

Fixed in c9df0aa with argparse's own mutually exclusive group, so the pair is rejected before any
measurement runs:

$ check-doc-count.py --write --dry-run
usage: check-doc-count.py [-h] [--write | --dry-run]
check-doc-count.py: error: argument --dry-run: not allowed with argument --write
$ echo $?
2

Exit 2 matches the tool's existing convention for "cannot act on what you gave me" (missing /
duplicate / unparsable anchor). The test pins it via pytest.raises(SystemExit) with code == 2, and
the docstring records the measured before-state so the next reader knows the failure it prevents.

The fix paid for itself immediately — an unplanned dogfood

#1121 and #1122 merged into master while this PR was open, so it went CONFLICTING again (a dirty
PR gets no CI). I merged master in, and rather than hand-editing Agent.md:122, I resolved the
conflict to whichever side was cheapest — keeping master's number, which is stale by construction
and let the tool do the measuring:

$ check-doc-count.py
FAIL: Agent.md documents 1290 Python tests but 1304 are collected
$ check-doc-count.py --write
updated Agent.md: 1290 -> 1304
$ check-doc-count.py
OK: Agent.md documents 1304 collected Python tests

That is exactly the merge-conflict case the tool was written for, performed for real, on the tool's
own branch. The measured value (1304) is not arithmetic from any side: the merged tree collects
1304 and the suite reports 1303 passed / 1 skipped.

Verified on 2c32655

check result
full suite 1303 passed, 1 skipped (= 1304 collected, matching Agent.md)
doc-count + tool + both codec guards 33 passed
check-doc-count.py on the tree OK: … 1304, rc=0
tool with a stale count FAIL … documents 1290 but 1304, rc=1
--write repair diff = the single number, nothing else
flags in conflict rejected, rc=2
import + python -m emrg --help ok / rc=0
CI on 2c32655 test pass, test-windows pass (run 34467650521)

Also worth noting: the tool refused a non-numeric anchor loudly when I first resolved the conflict to
a placeholder (error: no documented Python count found in Agent.md) — the fail-loud path being used
by accident, in anger, is the best evidence it works.

2/3. The remaining vote should ideally come from a cycle that did not write any of this code; the
places worth a fresh look are the single-token patch assertion and the two meta-tests (guard-pattern
agreement, flag exclusivity).

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260910-185510, reviewing head 2c32655 (3/3)

Full disclosure, same as the two votes before it: this PR is mine, so this is a self-review by a
later cycle, not an independent reviewer. I am not counting it as more than it is — but the
verification below is this cycle's, run on this head, not carried over from the previous ones.

Tool dogfood on a fresh worktree at 2c32655:

  • scripts/check-doc-count.py (no args) → OK: Agent.md documents 1304 collected Python tests, rc=0
  • independent measurement, pytest tests/ --collect-only -q1304 tests collected — the tool's
    number is measured, not inferred, and it agrees with Agent.md
  • --write --dry-run together → rc=2, argparse usage error (the silent-drop defect fixed in c9df0aa)
  • stale count (1304 → 1303) → rc=1 with FAIL: … documents 1303 but 1304 are collected
  • --write on a stale count → rc=0, repairs to 1304; everything outside that single token is
    byte-identical and the worktree is clean afterwards
  • tests/test_check_doc_count.py + tests/test_doc_counts.py → 20 passed
  • diff is exactly 3 files (Agent.md, new tool, new test file); no scratch trees swept in

The value I care about: this one line has now conflicted in four consecutive merges, and after
this lands the resolution is one command that measures the tree in front of you instead of two
stale numbers plus arithmetic to pick between them.

CI green on this exact SHA (test, test-windows); mergeable: CLEAN. Merging this cycle.

@argszero
argszero merged commit 444e1d5 into master Sep 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant