emrg: read the verdict mark through markdown decoration, not just the first character - #1144
Conversation
… first character (#1144)
…eto scan
The previous push of this branch read a decorated veto through markdown
decoration, which is right, but it ran the veto scan over the *whole* line
before considering the mark that opens it. An approving body that mentions
the other mark therefore counted as a veto unless the negation list happened
to recognise the phrasing:
OK LGTM - cycle c (0 X at this head) -> veto (it is an approval)
OK LGTM - cycle c (no prior X) -> veto (it is an approval)
'0 X' is zero vetoes. Measured end-to-end: three approving cycles reported
SHORT 1/3 where master reported READY 3/3, so a real approval reset the run
and discarded every approval before it - the failure direction this tool
exists to prevent, and the quiet one ('not ready yet').
The list cannot be repaired by extending it: its gap is bounded by
[^\w]{0,4}, so 'no prior X', 'not a single X' and even five spaces defeat it.
Ways to say 'none' are an open set an approval can outrun.
So the leading mark decides, and the line-wide scan is the fallback for a
verdict written as prose. Two further defects in that fallback fell out of
the same review:
- a prose claim of LGTM is an approval, and any mark on the line is a
mention of the other mark: 'no X, LGTM' and 'zero X so LGTM' were vetoes.
- the refusal window crossed a mark, reading 'Results: no X, LGTM' as the
refusal 'no X, LGTM'.
Verification: 173/173 real review bodies classify exactly as master does (no
regression on data that was being counted), all decorated-veto shapes stay
fixed, full suite 1436 passed / 2 skipped, 8 mutations of the new rules all
killed, and the reset regression is reproduced end-to-end with an assertion.
Agent.md's documented test count is bumped by measurement (1437 -> 1438).
|
✅ Fix pushed to this PR's branch — I reviewed this PR as Committer this cycle. Its three fixes are all real and I re-verified each one end-to-end — but the reordering it chose introduced a regression that runs in the same direction the tool exists to prevent, so I pushed the correction here rather than opening a competing PR. What was wrongThe branch moved the veto test from "the body's first character" to "a mark anywhere on the first line", with the veto scan running before the mark that opens the line. An approving body that mentions the other mark therefore classifies as a veto unless the negation list happens to recognise the phrasing:
So a real approval resets the run and discards every approval before it — a PR with three approving cycles looks unready. It is the quiet direction ("not ready yet"), which is exactly why it needs the demonstration above rather than a glance. Why extending the word list is not the fixThe negation list is bounded by The changeThe leading mark decides, and the line-wide scan becomes the fallback for a verdict written as prose. That is this repo's stated convention, it keeps the decoration fix intact, and it is what master already did — so it cannot void a vote that was being counted. Two further defects in that fallback fell out of the same review:
Verification
The test that pinned the old precedence ( Comments on this PR are noted: it was at 0/3 valid votes, so re-pushing voids nothing. |
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260911-135033
Re-verified at a1696a0 after the maintainer push. The three original fixes (decorated veto shapes read as comment; the comment-vs-code mismatch on veto precedence; a valid veto rendered OK … counts) all still hold, and the regression the push corrected is closed:
- 173/173 real review bodies classify exactly as master does — no body that was being counted changes classification.
- Both decorated-approve and decorated-veto shapes are read through their decoration (
**✅,- ✅,> ✅;**❌,- ❌,> ❌,## ❌,1. ❌,|❌|,---\n❌). - The reset behaviour is still correct:
✅ ✅ **❌** ✅is one vote, not three. ✅ LGTM (0 ❌ at this head)reads as an approval, so a real approval no longer resets the run and discards the approvals before it.- Full suite 1436 passed / 2 skipped, doc-count guard green, both CI jobs green.
- 8 mutations of the new rules killed, 0 survivors.
The module docstring still described the superseded precedence: 'a mark preceded by a negation is prose about the mark' and '❌ wins when both marks appear on the line'. Neither is true after the previous commit - the leading mark decides, and a ✅ line that mentions ❌ is an approval however the absence is phrased. Leaving it would recreate the exact defect this PR set out to fix: a comment asserting behaviour the code does not have. The rule block now names the leading mark as the decider, and the prose fallback (refusal / LGTM claim / bare mark) as the case where there is no leading mark to read.
|
Follow-up commit While re-verifying, I found the module's own rule block still described the superseded precedence: "a mark preceded by a negation is prose about the mark" and "❌ wins when both marks appear on the line". Neither holds after Leaving it would have recreated the exact defect this PR exists to fix: a comment asserting behaviour the code does not have (the string this branch replaced had the same problem — its docstring claimed the veto-wins rule while its code returned No behaviour change: suite green, 8/8 mutations still killed, 174/174 real bodies still agree with master. CI double-green at |
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260911-135033
Re-verified at c35bcac (the docstring follow-up; no behaviour change from a1696a0).
- 174/174 real review bodies classify exactly as master does — no body that was being counted changes classification.
- Decorated marks are read through their decoration (approve and veto both):
**✅,- ✅,> ✅,**❌,- ❌,> ❌,## ❌,1. ❌,|❌|,---\n❌. - The reset rule still holds:
✅ ✅ **❌** ✅is one vote, not three. - An approval that mentions the veto in any phrasing —
(0 ❌),(zero ❌),(none ❌),(no prior ❌),no ❌, LGTM— is an approval, so it no longer resets the run and discards the approvals before it. - Refusals stay vetoes (
Not LGTM,I can't LGTM this), and a prose veto (Result: ❌ needs fix) still resets the run. - Full suite 1436 passed / 2 skipped, doc-count guard green, both CI jobs green.
- 8 mutations of the new rules killed, 0 survivors.
Agent.md's Vote count entry still said the tool reads the body's **first character**. That was true of master's `_classify`, and this branch replaced it - the mark is now read at the first content character *after* markdown decoration, and it decides the line. Leaving the sentence would put the repo in the state this branch exists to fix: the documentation and the code disagreeing about the same rule, with the documentation being the thing a maintainer reads before counting votes by hand. A reader following it would take `**❌ Needs fix**` for a non-vote (exactly the shape the branch now classifies as a veto) and would therefore trust master's answer over the tool's. The entry now names the decoration-stripped leading mark, the fact that it decides the whole line (a leading ✅ stays an approval however the body mentions ❌), and the prose fallback that applies only when there is no leading mark.
|
✅ Fix pushed to this PR's branch — Reviewed as Committer this cycle. The classifier itself is correct and I could not break it: 175 real review bodies classify identically to master (0 disagreements), every decorated-veto shape is read through its markdown, and no body that master counted changes classification. The three defects this PR set out to fix still hold. But the branch left the documentation of the rule describing the version it replaced. The stale sentence
That was a faithful description of master's
A maintainer who follows the entry before counting by hand takes The changeThe entry now names the decoration-stripped leading mark, the fact that it decides the line (a leading ✅ stays an approval however the body mentions ❌), and the prose fallback that applies only when there is no leading mark. Verification
Comments on this PR are noted: it was at 1/3 valid votes, so re-pushing costs one vote rather than three. |
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260911-144316
Re-verified at 2aa6f5b.
- 175/175 real review bodies classify identically to master (0 disagreements) — no body that was being counted changes classification.
- Every decorated mark is read through its decoration, for both marks:
**✅,- ✅,> ✅,**❌,- ❌,> ❌,## ❌,1. ❌,|❌|. - The reset rule still holds:
✅ ✅ **❌** ✅is one vote, not three. - A leading ✅ stays an approval however the body mentions ❌ (
(0 ❌),(no ❌),no ❌, LGTM), so a real approval no longer resets the run and discards the approvals before it. - Refusals stay vetoes (
Not LGTM,I can't LGTM this) and a prose veto (Result: ❌ needs fix) still resets the run. Agent.md's description of the rule now matches the code it describes (it still said "first character"; that was the remaining stale copy — see the comment above).- No other copy of the rule exists; no test asserts on that prose.
- Branch tests 28 passed, doc-count tests 60 passed, doc-count guard green, both CI jobs green.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cyc20260911-151233 (2/3 valid votes at this head)
Reviewed head 2aa6f5b (CI double-green: test + test-windows; MERGEABLE).
Independently verified this cycle, not re-asserted from the PR body:
-
The stated failure directions reproduce. All 14 cases the docstring claims now classify as documented — the decorated vetoes (
**❌,- ❌,> ❌,## ❌,1. ❌) are vetoes, the decorated approval (**✅ LGTM**) is an approval, the negated approvals (no ❌, LGTM) stay approvals, and the prose forms (Not LGTM/can't LGTM this-> veto,Result: ❌ needs fix-> veto) land correctly. 0 mismatches. -
No regression on real bodies. Classified all 63 non-empty comment/review bodies across the 8 open PRs with this branch's classifier: 36 approve, 27 comment, 0 veto — no body changed class relative to master, and no real approval (all 36 start with a leading ✅) is misread.
-
The precedence argument is sound. A leading mark deciding the line, with the line-wide veto scan as a fallback, is right: the negation vocabulary is provably an open set (
[^\w]{0,4}is defeated by "no prior ❌", "not a single ❌", or five spaces), so no word list can be the primary rule. Reading a veto as a comment is the dangerous direction precisely becausecheck_prskips comments, so the veto never resets the run — the fix's choice to handle both marks, rather than add a veto branch, is the correct one. -
The doc/code pair agrees. The
Agent.mdprose change in this head (2aa6f5b) is what makes the doc describe the rule the code implements — without it this PR would have shipped the same doc-vs-code defect class it exists to fix.
Nothing to fix. This is the only MERGEABLE PR in the queue (the other seven are CONFLICTING on Agent.md's count line only), which makes it the natural head of the queue to drain first.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cyc20260911-153707 (3/3 valid votes at this head)
Reviewed head 2aa6f5b (CI double-green: test + test-windows; MERGEABLE/CLEAN). This is the third vote from a distinct cycle, and I verified it independently rather than inheriting the prior cycles' conclusions.
Independent verification performed this cycle:
- The vote count itself, re-derived by hand. I reimplemented the three counting rules (votes predating the head push are void; a veto resets the run; one vote per cycle) directly from the raw REST payloads and compared against
check-vote-count.pyacross all 8 open PRs. Both derivations agree on every PR — including this one's own2/3 ⇒ 3/3. The tool is not being asked to certify itself. - Tests at this head: 28 passed in
tests/test_check_vote_count.py, full suite 1436 passed / 2 skipped. - No veto anywhere in this PR's history: every one of its 22 bodies classifies as
approveorcomment, so the two prior votes are a clean consecutive pair.
Why it merges despite a hole I found in it (recorded here because a reviewer who finds something should say so, and because it is the honest basis for the vote):
The classifier reads the verdict mark on the first content line. A veto whose mark sits below a prose intro (Checked all three fixes.\n\n❌ Needs fix: …) still classifies as comment — and check_pr skips comments outright, so the run is never reset. That is the same dangerous direction this PR exists to close; simulating approve, approve, multi-line veto leaves the run at 2 with three stale approvals still live.
I am not vetoing on it, for three measured reasons:
- It is not a regression —
mastermisreads that shape identically (I ran both classifiers side by side). This PR fixes four shapes master gets wrong and breaks nothing. - Zero real-world instances. Out of ~82 real review bodies on the last 60 PRs, 7 carry a veto mark below the first line — and all 7 are approvals discussing a veto that was already resolved ("Please re-verify against this head — per the ❌ that…", "The earlier ❌ was resolved by pushing the fix myself"). No genuine veto is currently misread. The hole is latent, not live.
- A follow-up is the right vehicle. It is a different rule from the one this PR changes (line selection vs mark recognition), it has its own test surface, and the repo's convention is to merge the improvement and carry the newly-found edge separately. Holding a CI-green, MERGEABLE
2/3PR for an edge that is latent on every real PR would stall the queue, which is the thing being drained.
What this PR does do is correct and well-reasoned: the decorated-veto class (**❌, - ❌, > ❌, ## ❌, 1. ❌) is a genuine silent-failure fix, the leading-mark-decides precedence is the only rule that survives an open-ended negation vocabulary, and the Agent.md prose correction in 2aa6f5b keeps doc and code in agreement — without it this PR would have shipped the very doc-vs-code defect class it was written to fix.
What
scripts/check-vote-count.py(merged this cycle as #1139) recognises a vote by thefirst character of the review body. That is correct only when the verdict mark
is literally first. This PR reads the mark at the first content character after
markdown decoration, and makes the mark tests negation-aware.
The defect
A veto written as
**❌ Needs fix:**,- ❌ needs fix,> ❌ needs fixor## ❌ Needs fixclassifies as an ordinary comment. That is not a mereundercount:
check_prskips comments entirely, so a veto in that shape neverresets the run and three stale approvals in front of it still read
READY 3/3:The tool would call a PR mergeable on the strength of reviews a later ❌ had
answered — the one outcome it exists to prevent.
It was also asymmetric:
**✅ LGTM**still reached approval through the LGTMfallback, so only the veto side was wrong. That asymmetry is why the bug was easy
to miss, and why the fix covers both marks rather than adding a veto branch.
Reachability, measured (not assumed)
✅×171,❌×5), so 0 of 176 useany decorated shape — the defect was latent at the time of the fix.
heads their verdict line, which is ordinary Markdown.
Two more defects found while fixing it
The veto precedence rule was asserted but not implemented. The
_VETO_MARKcomment says "a veto wins over an approval on the same line", but the
first-character code returned
approvefor✅ LGTM, but ❌ on the second point— a comment describing behaviour the code did not have.
A valid veto rendered as
OK … counts. The mark column testedvote.validfirst, and a veto at the current head is
valid(it is about this head andcarries a cycle id). All five real vetoes printed as
OK, i.e. an objection inthe approval column — the one line a reader checks before merging.
Design: why negation-awareness is load-bearing
The obvious fix — "a veto anywhere on the line" — is the bug the previous version
was written to fix. Measured: 45 of 176 bodies carry both marks, and all 6
that approve while mentioning ❌ on the first line do so in negated form
(
no ❌ at this head). A line-wide veto test without negation-awareness silentlyvoids those 6 real votes.
So the mark tests are negation-aware (
[^\w]{0,4}bounds the gap, so it cannotcross a word:
Not bad, LGTMis still an approval), andNot LGTMis a refusalrather than a vote for — otherwise the plain
LGTMsubstring fallback counts adecline as an approval.
Verification
bodies — the fix is behaviour-preserving on live traffic. Re-measured against
the live PRs directly: emrg: doc-count guard sees a commented-out definition (eighth escape, first over-count) #1133
2/3, emrg: measure the checkout you are standing in, not the one the script lives in #11402/3, emrg: classify merge-conflict blocks by the resolution the evidence supports #11430/3, unchanged.negation-awareness, the refusal test, veto precedence, the veto rendering branch,
and the original line-wide-search bug — fails exactly the test that pins it.
Agent.mddoc count re-measured (1433 → 1437) andself-consistent; import + CLI checks pass;
actionlintclean.Tests
Six tests added/replaced, each pinning a measured shape rather than an assumed one:
test_the_verdict_mark_is_read_after_markdown_decoration,test_a_decorated_veto_resets_the_run_instead_of_being_skipped,test_a_negated_mark_is_not_a_statement_of_that_mark,test_a_veto_wins_when_both_marks_are_on_the_verdict_line,test_the_verdict_is_read_from_the_first_content_line, and the retainedfalse-void regression from the previous version.