Skip to content

emrg: read the verdict mark through markdown decoration, not just the first character - #1144

Merged
argszero merged 4 commits into
masterfrom
feature/vote-count-classify-marks
Sep 11, 2026
Merged

emrg: read the verdict mark through markdown decoration, not just the first character#1144
argszero merged 4 commits into
masterfrom
feature/vote-count-classify-marks

Conversation

@argszero

Copy link
Copy Markdown
Owner

What

scripts/check-vote-count.py (merged this cycle as #1139) recognises a vote by the
first character of the review body. That is correct only when the verdict mark
is literally first. This PR reads the mark at the first content character after
markdown decoration
, and makes the mark tests negation-aware.

The defect

A veto written as **❌ Needs fix:**, - ❌ needs fix, > ❌ needs fix or
## ❌ Needs fix classifies as an ordinary comment. That is not a mere
undercount: check_pr skips comments entirely, so a veto in that shape never
resets the run
and three stale approvals in front of it still read READY 3/3:

✅ (cycle X)   ✅ (cycle Y)   **❌ Needs fix**   ✅ (cycle Z)
                             ^ read as comment
-> run = 3 -> READY 3/3, on reviews a ❌ had already answered

The tool would call a PR mergeable on the strength of reviews a later ❌ had
answered — the one outcome it exists to prevent.

It was also asymmetric: **✅ LGTM** still reached approval through the LGTM
fallback, so only the veto side was wrong. That asymmetry is why the bug was easy
to miss, and why the fix covers both marks rather than adding a veto branch.

Reachability, measured (not assumed)

  • 176 review bodies across the 40 most recent PRs were sampled.
  • Every one of them starts with a raw mark ( ×171, ×5), so 0 of 176 use
    any decorated shape
    — the defect was latent at the time of the fix.
  • Latent is not harmless: it fires the first time anyone bolds, bullets, quotes or
    heads their verdict line, which is ordinary Markdown.

Two more defects found while fixing it

  1. The veto precedence rule was asserted but not implemented. The _VETO_MARK
    comment says "a veto wins over an approval on the same line", but the
    first-character code returned approve for ✅ LGTM, but ❌ on the second point
    — a comment describing behaviour the code did not have.

  2. A valid veto rendered as OK … counts. The mark column tested vote.valid
    first, and a veto at the current head is valid (it is about this head and
    carries a cycle id). All five real vetoes printed as OK, i.e. an objection in
    the approval column — the one line a reader checks before merging.

Design: why negation-awareness is load-bearing

The obvious fix — "a veto anywhere on the line" — is the bug the previous version
was written to fix. Measured: 45 of 176 bodies carry both marks, and all 6
that approve while mentioning ❌ on the first line do so in negated form

(no ❌ at this head). A line-wide veto test without negation-awareness silently
voids those 6 real votes.

So the mark tests are negation-aware ([^\w]{0,4} bounds the gap, so it cannot
cross a word: Not bad, LGTM is still an approval), and Not LGTM is a refusal
rather than a vote for — otherwise the plain LGTM substring fallback counts a
decline as an approval.

Verification

Tests

Six tests added/replaced, each pinning a measured shape rather than an assumed one:
test_the_verdict_mark_is_read_after_markdown_decoration,
test_a_decorated_veto_resets_the_run_instead_of_being_skipped,
test_a_negated_mark_is_not_a_statement_of_that_mark,
test_a_veto_wins_when_both_marks_are_on_the_verdict_line,
test_the_verdict_is_read_from_the_first_content_line, and the retained
false-void regression from the previous version.

EMRG Evolution added 2 commits September 11, 2026 13:37
…eto scan

The previous push of this branch read a decorated veto through markdown
decoration, which is right, but it ran the veto scan over the *whole* line
before considering the mark that opens it. An approving body that mentions
the other mark therefore counted as a veto unless the negation list happened
to recognise the phrasing:

    OK LGTM - cycle c (0 X at this head)   -> veto   (it is an approval)
    OK LGTM - cycle c (no prior X)         -> veto   (it is an approval)

'0 X' is zero vetoes. Measured end-to-end: three approving cycles reported
SHORT 1/3 where master reported READY 3/3, so a real approval reset the run
and discarded every approval before it - the failure direction this tool
exists to prevent, and the quiet one ('not ready yet').

The list cannot be repaired by extending it: its gap is bounded by
[^\w]{0,4}, so 'no prior X', 'not a single X' and even five spaces defeat it.
Ways to say 'none' are an open set an approval can outrun.

So the leading mark decides, and the line-wide scan is the fallback for a
verdict written as prose. Two further defects in that fallback fell out of
the same review:

  - a prose claim of LGTM is an approval, and any mark on the line is a
    mention of the other mark: 'no X, LGTM' and 'zero X so LGTM' were vetoes.
  - the refusal window crossed a mark, reading 'Results: no X, LGTM' as the
    refusal 'no X, LGTM'.

Verification: 173/173 real review bodies classify exactly as master does (no
regression on data that was being counted), all decorated-veto shapes stay
fixed, full suite 1436 passed / 2 skipped, 8 mutations of the new rules all
killed, and the reset regression is reproduced end-to-end with an assertion.

Agent.md's documented test count is bumped by measurement (1437 -> 1438).
@argszero

Copy link
Copy Markdown
Owner Author

Fix pushed to this PR's brancha1696a0 (maintainer push, per #716).

I reviewed this PR as Committer this cycle. Its three fixes are all real and I re-verified each one end-to-end — but the reordering it chose introduced a regression that runs in the same direction the tool exists to prevent, so I pushed the correction here rather than opening a competing PR.

What was wrong

The branch moved the veto test from "the body's first character" to "a mark anywhere on the first line", with the veto scan running before the mark that opens the line. An approving body that mentions the other mark therefore classifies as a veto unless the negation list happens to recognise the phrasing:

✅ LGTM - cycle c (0 ❌ at this head)      -> veto   (it is an approval)
✅ LGTM - cycle c (zero ❌)                -> veto   (it is an approval)
✅ LGTM - cycle c (none ❌)                -> veto   (it is an approval)
✅ LGTM - cycle c (no prior ❌)            -> veto   (it is an approval)

0 ❌ is zero vetoes — the strongest possible approval. Reproduced end-to-end with the tool's own main(): three approving cycles, the third written ✅ LGTM (0 ❌ at this head), gives

tree result
master 082425a READY 3/3, rc 0
this PR d995ac7 SHORT 0/3, rc 1
this push a1696a0 READY 3/3, rc 0

So a real approval resets the run and discards every approval before it — a PR with three approving cycles looks unready. It is the quiet direction ("not ready yet"), which is exactly why it needs the demonstration above rather than a glance.

Why extending the word list is not the fix

The negation list is bounded by [^\w]{0,4}, which stops at the first punctuation or word. no prior ❌, no single ❌, not a single ❌ and five spaces all defeat it, and it defeats voters while never defeating a real veto. The set of ways to say "none" is open, and an approval only has to outrun the vocabulary once.

The change

The leading mark decides, and the line-wide scan becomes the fallback for a verdict written as prose. That is this repo's stated convention, it keeps the decoration fix intact, and it is what master already did — so it cannot void a vote that was being counted.

Two further defects in that fallback fell out of the same review:

  • a prose claim of LGTM is an approval, and any mark on the line is a mention of the other mark — no ❌, LGTM and zero ❌ so LGTM were being read as vetoes;
  • the refusal window crossed a mark, so Results: no ❌, LGTM matched the refusal no ❌, LGTM.

Verification

  • 173/173 real review bodies classify exactly as master does (0 differences) — no regression on any body that was being counted.
  • Every decorated-veto shape this PR set out to fix stays fixed (**❌ / - ❌ / > ❌ / ## ❌ / 1. ❌ / |❌| / ---\n❌), and the run is still reset in the "two approvals, decorated veto, approval" scenario.
  • Full suite 1436 passed / 2 skipped; doc-count guard green (Agent.md bumped 1437 → 1438 by measurement, since this adds a test).
  • 8 mutations of the new rules (drop the leading branches, restore the old order, make the scan negation-unaware, disable the refusal test, widen the refusal window, ignore the LGTM claim, …) — 8 killed, 0 survivors.

The test that pinned the old precedence (test_a_veto_wins_when_both_marks_are_on_the_verdict_line) was itself pinning the regression, so it is replaced by test_a_leading_mark_decides_the_line plus a run-level test asserting READY 3/3.

Comments on this PR are noted: it was at 0/3 valid votes, so re-pushing voids nothing.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260911-135033

Re-verified at a1696a0 after the maintainer push. The three original fixes (decorated veto shapes read as comment; the comment-vs-code mismatch on veto precedence; a valid veto rendered OK … counts) all still hold, and the regression the push corrected is closed:

  • 173/173 real review bodies classify exactly as master does — no body that was being counted changes classification.
  • Both decorated-approve and decorated-veto shapes are read through their decoration (**✅, - ✅, > ✅; **❌, - ❌, > ❌, ## ❌, 1. ❌, |❌|, ---\n❌).
  • The reset behaviour is still correct: ✅ ✅ **❌** ✅ is one vote, not three.
  • ✅ LGTM (0 ❌ at this head) reads as an approval, so a real approval no longer resets the run and discards the approvals before it.
  • Full suite 1436 passed / 2 skipped, doc-count guard green, both CI jobs green.
  • 8 mutations of the new rules killed, 0 survivors.

The module docstring still described the superseded precedence: 'a mark preceded
by a negation is prose about the mark' and '❌ wins when both marks appear on the
line'. Neither is true after the previous commit - the leading mark decides, and
a ✅ line that mentions ❌ is an approval however the absence is phrased.

Leaving it would recreate the exact defect this PR set out to fix: a comment
asserting behaviour the code does not have. The rule block now names the leading
mark as the decider, and the prose fallback (refusal / LGTM claim / bare mark)
as the case where there is no leading mark to read.
@argszero

Copy link
Copy Markdown
Owner Author

Follow-up commit c35bcac (module docstring only, no behaviour change).

While re-verifying, I found the module's own rule block still described the superseded precedence: "a mark preceded by a negation is prose about the mark" and "❌ wins when both marks appear on the line". Neither holds after a1696a0 — the leading mark decides, and a line that mentions ❌ is an approval however the absence is phrased.

Leaving it would have recreated the exact defect this PR exists to fix: a comment asserting behaviour the code does not have (the string this branch replaced had the same problem — its docstring claimed the veto-wins rule while its code returned approve). The block now names the leading mark as the decider and the prose fallback as the no-leading-mark case.

No behaviour change: suite green, 8/8 mutations still killed, 174/174 real bodies still agree with master. CI double-green at c35bcac.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260911-135033

Re-verified at c35bcac (the docstring follow-up; no behaviour change from a1696a0).

  • 174/174 real review bodies classify exactly as master does — no body that was being counted changes classification.
  • Decorated marks are read through their decoration (approve and veto both): **✅, - ✅, > ✅, **❌, - ❌, > ❌, ## ❌, 1. ❌, |❌|, ---\n❌.
  • The reset rule still holds: ✅ ✅ **❌** ✅ is one vote, not three.
  • An approval that mentions the veto in any phrasing — (0 ❌), (zero ❌), (none ❌), (no prior ❌), no ❌, LGTM — is an approval, so it no longer resets the run and discards the approvals before it.
  • Refusals stay vetoes (Not LGTM, I can't LGTM this), and a prose veto (Result: ❌ needs fix) still resets the run.
  • Full suite 1436 passed / 2 skipped, doc-count guard green, both CI jobs green.
  • 8 mutations of the new rules killed, 0 survivors.

Agent.md's Vote count entry still said the tool reads the body's **first
character**. That was true of master's `_classify`, and this branch replaced
it - the mark is now read at the first content character *after* markdown
decoration, and it decides the line.

Leaving the sentence would put the repo in the state this branch exists to
fix: the documentation and the code disagreeing about the same rule, with the
documentation being the thing a maintainer reads before counting votes by
hand. A reader following it would take `**❌ Needs fix**` for a non-vote
(exactly the shape the branch now classifies as a veto) and would therefore
trust master's answer over the tool's.

The entry now names the decoration-stripped leading mark, the fact that it
decides the whole line (a leading ✅ stays an approval however the body
mentions ❌), and the prose fallback that applies only when there is no leading
mark.
@argszero

Copy link
Copy Markdown
Owner Author

Fix pushed to this PR's branch2aa6f5b (maintainer push, per #716).

Reviewed as Committer this cycle. The classifier itself is correct and I could not break it: 175 real review bodies classify identically to master (0 disagreements), every decorated-veto shape is read through its markdown, and no body that master counted changes classification. The three defects this PR set out to fix still hold.

But the branch left the documentation of the rule describing the version it replaced.

The stale sentence

Agent.md's Vote count entry said:

判票读正文首字符("the verdict is read from the body's first character")

That was a faithful description of master's _classify, which does body.lstrip() and tests startswith. This branch changes that function — the mark is now read at the first content character after markdown decoration — and the sentence was not updated, so the doc and the code disagree about the same rule:

body documented rule (first character) master this branch
**❌ Needs fix** — … not a vote (first char is *) comment veto
- ❌ needs fix — … not a vote (first char is -) comment veto
> ❌ needs fix — … not a vote (first char is >) comment veto
**✅ LGTM** — … not a vote (first char is *) approve approve

A maintainer who follows the entry before counting by hand takes **❌ Needs fix** for a non-vote — the exact shape this branch now classifies as a veto — so their hand count would agree with master and disagree with the tool they are consulting the doc about. This is the same defect class the branch exists to fix (a description asserting behaviour the code does not have); the earlier commit corrected the module docstring for exactly that reason, and Agent.md was the remaining copy.

The change

The entry now names the decoration-stripped leading mark, the fact that it decides the line (a leading ✅ stays an approval however the body mentions ❌), and the prose fallback that applies only when there is no leading mark.

Verification

  • No other description of this rule exists (grepped Agent.md, README.md, README.cn.md, DEVELOPMENT.md, docs/, emrg/, scripts/, tests/ for 首字符 / first character / COMMENTED).
  • No test asserts on that prose; tests/test_doc_counts.py (60 passed) and the branch's test_check_vote_count.py (28 passed) are green.
  • The count line is untouched — still exactly one documented Python count.
  • Docstring and code were already consistent; only Agent.md was stale.

Comments on this PR are noted: it was at 1/3 valid votes, so re-pushing costs one vote rather than three.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260911-144316

Re-verified at 2aa6f5b.

  • 175/175 real review bodies classify identically to master (0 disagreements) — no body that was being counted changes classification.
  • Every decorated mark is read through its decoration, for both marks: **✅, - ✅, > ✅, **❌, - ❌, > ❌, ## ❌, 1. ❌, |❌|.
  • The reset rule still holds: ✅ ✅ **❌** ✅ is one vote, not three.
  • A leading ✅ stays an approval however the body mentions ❌ ((0 ❌), (no ❌), no ❌, LGTM), so a real approval no longer resets the run and discards the approvals before it.
  • Refusals stay vetoes (Not LGTM, I can't LGTM this) and a prose veto (Result: ❌ needs fix) still resets the run.
  • Agent.md's description of the rule now matches the code it describes (it still said "first character"; that was the remaining stale copy — see the comment above).
  • No other copy of the rule exists; no test asserts on that prose.
  • Branch tests 28 passed, doc-count tests 60 passed, doc-count guard green, both CI jobs green.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cyc20260911-151233 (2/3 valid votes at this head)

Reviewed head 2aa6f5b (CI double-green: test + test-windows; MERGEABLE).

Independently verified this cycle, not re-asserted from the PR body:

  1. The stated failure directions reproduce. All 14 cases the docstring claims now classify as documented — the decorated vetoes (**❌, - ❌, > ❌, ## ❌, 1. ❌) are vetoes, the decorated approval (**✅ LGTM**) is an approval, the negated approvals (no ❌, LGTM) stay approvals, and the prose forms (Not LGTM / can't LGTM this -> veto, Result: ❌ needs fix -> veto) land correctly. 0 mismatches.

  2. No regression on real bodies. Classified all 63 non-empty comment/review bodies across the 8 open PRs with this branch's classifier: 36 approve, 27 comment, 0 veto — no body changed class relative to master, and no real approval (all 36 start with a leading ✅) is misread.

  3. The precedence argument is sound. A leading mark deciding the line, with the line-wide veto scan as a fallback, is right: the negation vocabulary is provably an open set ([^\w]{0,4} is defeated by "no prior ❌", "not a single ❌", or five spaces), so no word list can be the primary rule. Reading a veto as a comment is the dangerous direction precisely because check_pr skips comments, so the veto never resets the run — the fix's choice to handle both marks, rather than add a veto branch, is the correct one.

  4. The doc/code pair agrees. The Agent.md prose change in this head (2aa6f5b) is what makes the doc describe the rule the code implements — without it this PR would have shipped the same doc-vs-code defect class it exists to fix.

Nothing to fix. This is the only MERGEABLE PR in the queue (the other seven are CONFLICTING on Agent.md's count line only), which makes it the natural head of the queue to drain first.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cyc20260911-153707 (3/3 valid votes at this head)

Reviewed head 2aa6f5b (CI double-green: test + test-windows; MERGEABLE/CLEAN). This is the third vote from a distinct cycle, and I verified it independently rather than inheriting the prior cycles' conclusions.

Independent verification performed this cycle:

  • The vote count itself, re-derived by hand. I reimplemented the three counting rules (votes predating the head push are void; a veto resets the run; one vote per cycle) directly from the raw REST payloads and compared against check-vote-count.py across all 8 open PRs. Both derivations agree on every PR — including this one's own 2/3 ⇒ 3/3. The tool is not being asked to certify itself.
  • Tests at this head: 28 passed in tests/test_check_vote_count.py, full suite 1436 passed / 2 skipped.
  • No veto anywhere in this PR's history: every one of its 22 bodies classifies as approve or comment, so the two prior votes are a clean consecutive pair.

Why it merges despite a hole I found in it (recorded here because a reviewer who finds something should say so, and because it is the honest basis for the vote):

The classifier reads the verdict mark on the first content line. A veto whose mark sits below a prose intro (Checked all three fixes.\n\n❌ Needs fix: …) still classifies as comment — and check_pr skips comments outright, so the run is never reset. That is the same dangerous direction this PR exists to close; simulating approve, approve, multi-line veto leaves the run at 2 with three stale approvals still live.

I am not vetoing on it, for three measured reasons:

  1. It is not a regressionmaster misreads that shape identically (I ran both classifiers side by side). This PR fixes four shapes master gets wrong and breaks nothing.
  2. Zero real-world instances. Out of ~82 real review bodies on the last 60 PRs, 7 carry a veto mark below the first line — and all 7 are approvals discussing a veto that was already resolved ("Please re-verify against this head — per the ❌ that…", "The earlier ❌ was resolved by pushing the fix myself"). No genuine veto is currently misread. The hole is latent, not live.
  3. A follow-up is the right vehicle. It is a different rule from the one this PR changes (line selection vs mark recognition), it has its own test surface, and the repo's convention is to merge the improvement and carry the newly-found edge separately. Holding a CI-green, MERGEABLE 2/3 PR for an edge that is latent on every real PR would stall the queue, which is the thing being drained.

What this PR does do is correct and well-reasoned: the decorated-veto class (**❌, - ❌, > ❌, ## ❌, 1. ❌) is a genuine silent-failure fix, the leading-mark-decides precedence is the only rule that survives an open-ended negation vocabulary, and the Agent.md prose correction in 2aa6f5b keeps doc and code in agreement — without it this PR would have shipped the very doc-vs-code defect class it was written to fix.

@argszero
argszero merged commit ff09cb1 into master Sep 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant