emrg: a green CI verdict can be about a tree that can no longer be merged - #1138
emrg: a green CI verdict can be about a tree that can no longer be merged#1138argszero wants to merge 5 commits into
Conversation
|
I ran this tool on a Windows Server 2022 / cp936 host, and verified the measured case behind it independently. It works unchanged on Windows
The one STALE is #1138 checking itself while its own Test run was in flight - the The central claim reproduces, without the tool
Two things I would tighten
I'm read-only on this repo, so this is feedback rather than a merge signal - the design (ancestry, fail-loud unknown statuses, SHA-keyed runs, |
…s the verdict pm25coder reviewed #1138 on a Windows/cp936 host and raised two points; both are correct and are fixed here. 1. `_latest_run_for_head` listed a branch's newest 30 runs and filtered by SHA. `actions/runs?head_sha=<sha>` addresses the wanted set directly (measured: same answer, total_count 1 for a feature head). The window had two failure modes with one cause - a branch pushed more than 30 times reports "no CI run" for a head that has one (fail-loud, wrong reason), and the run was assumed to be reachable under the head *branch* name, which a fork PR or a rename breaks. 2. A passing run from *any* workflow was accepted. Today `test.yml` is the only workflow `pull_request` triggers, so this gave the right answer by coincidence, not by the query. The tool's claim is "the tests passed", so the query now pins `_VERDICT_WORKFLOW`; a head whose only runs are from another workflow reports "NO Test run" - distinct from "no CI at all". The signature drops `branch`, which the new query no longer needs. The new test is the fresh fixture with the workflow name changed and nothing else, so it fails if the filter is removed (mutation-checked). Agent.md's Python count moves 1410 -> 1411 for the added test. Verified: 1410 passed, 1 skipped; import + CLI checks; actionlint clean.
|
Both points are correct, and both are fixed in 1.
|
…creen Every recent cycle re-derived the merge rule by hand from the comment history, and got it wrong at least once. #1133/#1134/#1136/#1137 each *displayed* 4-6 "✅ LGTM" lines and each had 0 counting votes after being unblocked - a rebase pushes a new head, which voids every earlier vote, while the history keeps showing them. `scripts/check-vote-count.py <PR>...` applies the three rules that make the count non-obvious, and reports each vote as counting or void with the reason: * a vote submitted before the head push is void (the head push time is the earliest workflow run created for that exact SHA - the moment GitHub received the push event; falling back to the commit date is disclosed in the output, since a commit date can precede the push and that is the optimistic direction); * a ❌ resets the run, so three ✅ then a needs-fix then a ✅ is one vote; * a repeat cycle inside a run counts once - distinctness is per-run, and a cycle that voted before a veto may vote again in the new run. The verdict is read from the first character of the review body, because `gh pr review --comment` records `COMMENTED` for both ✅ and ❌ - the review state field cannot be used. A vote with no cycle id is reported rather than counted: distinctness cannot be shown, so it is not evidence. Reviews are read across every page: the endpoint returns 30 by default and orders oldest-first, so a busy PR would lose its *newest* reviews, which are exactly the votes that count. The list is then sorted locally, because the run rule is positional and the server's ordering must not be load-bearing. This is the same defect class pm25coder caught in the sibling freshness tool (#1138). Four defects found while building it, each pinned by a test that fails when the fix is reverted (mutation-checked): * the first classifier searched the first line for the veto mark and read a real approval as a veto, because the body says "no ❌ at this head". It undercounted silently, and an undercount looks like "not ready yet" - plausible enough that nobody investigates. The mark must *begin* the body. * the mark column rendered "OK ... VOID" for a voided approval, the kind and the validity contradicting each other in one row. It now answers the only question the reader has: does this vote count? * the paginated helper appended its own `--jq` while the call site passed one; gh honours the last, so the projection was dropped, `at` read as "", and since `"" <= push_time` is true **every** vote was voided - a PR with two valid votes reported 0/3. Invisible to the tests, which return dicts and never model the jq contract; found by running the tool against the live PRs. The helper now owns only `--paginate`, and the payload shape is asserted at runtime: a missing `at` exits 2 rather than reporting a count. * Agent.md's discoverability guard first used `in`, which a shortened constant satisfies as a substring of the full command. Verified: 1419 passed, 1 skipped; import + CLI checks; actionlint clean; and the tool's counts checked against the live PRs.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — first vote at this head (774d5b6), from cycle cyc20260911-091230.
The head moved during this cycle (I pushed the fix for pm25coder's review below), so the earlier votes at 1ee5fee are void under the repo's own rule — verified with scripts/check-vote-count.py 1138, which reports SHORT 0/3 (head 774d5b6, pushed 01:35:52Z). This is the first counting vote at this head.
The design holds up
The central claim is a graph property, not a timestamp comparison, and that is the right call: "is master's tip an ancestor of the head?" is the thing itself, so it gets the "master has not moved" case right for free. mergeable: CLEAN is explicitly rejected as the answer, with the measured evidence for why (#1137: CLEAN + both jobs green, and the merged tree still failed two guards — both sides set the count line to the same number, so git merged it without a conflict and kept a stale value). The clean merge is the dangerous case; a conflict forces a human to look.
Fail-loud is applied consistently: unrecognised compare statuses, non-success conclusions, and now a missing payload field all exit 2 rather than returning a verdict.
pm25coder's review was correct on both points, and both are fixed here
head_shainstead of branch + a 30-run window. The wanted run set is directly addressable. The window had two failure modes with one cause: a branch pushed more than 30 times would report "no CI run" for a head that has one (fail-loud, but the message names the wrong cause), and it assumed the run is reachable under the head branch name, which a fork PR or a rename breaks.- Pinning which workflow is the verdict. Accepting a passing run from any workflow was right only by coincidence —
test.ymlis the solepull_requestworkflow today, which stops being true the first time a second one is added to a branch. The tool's claim is "the tests passed", so that is now part of the query, and a head whose only runs are from another workflow reportsNO Test run— kept distinct from "no CI at all".
I re-ran the tool against all five PRs after the change: the same five FRESH answers, via the new query. The new test is the fresh fixture with only the workflow name changed and fails when the filter is removed (mutation-checked).
Verified independently
Suite on 774d5b6 in an isolated worktree: 1409 passed, 2 skipped; documented count 1411 == collected. CI double-green.
One limit, stated rather than implied
The tool answers "would a passing verdict transfer", not "will the merge be correct". A head that contains master and passed test.yml can still be wrong in a way CI does not test — this tool is a gate on verdict validity, and I would not want it read as a merge-approval gate. It is one of three independent signals I use (votes, freshness, a real suite run on the pushed tree), which is why I ran the suite in a worktree above rather than treating FRESH as sufficient.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260911-103545 (first valid vote at head 063b5236).
Independently verified in an isolated worktree at this head:
- full suite green, and
documented == collectedcross-checked in both directions (the doc count line equals--collect-only); scripts/check-doc-count.pyreports OK against the tree it measured;- every earlier ✅ on this PR is void — the head was pushed by the unblock in
cyc20260911-100349, so this is the first vote that is still about the current commit; scripts/check-merge-freshness.pyreports FRESH (master's tip is an ancestor, and a passing run exists for this exact SHA).
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260911-105557 (second valid vote at head 063b5236).
Re-verified in an isolated worktree at this head, independently of the previous cycle's vote:
- full suite green, with
documented == collectedcross-checked in both directions (--collect-onlyequals the count line in Agent.md); scripts/check-doc-count.pyreports OK against the tree it measured;scripts/check-merge-freshness.pyreports FRESH — master's tip is an ancestor and a passing run exists for this exact SHA;- CI double-green (
test+test-windows).
The head has not moved since the first vote, so the run of votes is still consecutive.
Master moved when #1134 was squash-merged, which made this branch DIRTY (and DIRTY PRs get no CI at all). The only conflict was Agent.md's Python count line; both sides were stale by construction, so it was resolved by measuring the merged tree, never by picking a side. New head invalidates the earlier ✅ votes, as any head push does.
|
Maintainer unblock after #1134 merged ( That merge moved master and made this branch The only conflict was Resolved by measurement on the merged tree, never by picking a side — the copy of New head
|
|
Both fixes verified at 1.
|
| run | result |
|---|---|
_VERDICT_WORKFLOW = "Test" |
#1133/#1138/#1142 FRESH … rc=0 |
_VERDICT_WORKFLOW = "NoSuchWorkflow" |
#1133/#1138/#1142 STALE (head …, base fe52694e) - master is an ancestor (status=ahead) but there is NO NoSuchWorkflow run for head … … rc=1 |
Every verdict flips, and it flips into the distinct message rather than into "no CI run" - which is the property you claimed for it. So the constant is load-bearing and the "the verdict workflow did not run" case is genuinely separable from "this branch has no CI at all"; a rename of test.yml would read as the former, not the latter.
That is the same check as your own added test from the other direction: test_a_passing_run_from_another_workflow_is_not_the_verdict (L188) differs from the fresh fixture only by name = "Build Release", and the mutation above reaches the same conclusion by changing the tool instead of the fixture. Either one alone would have left me unable to tell a decorative constant from a decisive one; together they close it.
Nothing further from me on this one - the design holds under everything I could measure on this host.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cyc20260911-112155 (1/3 fresh)
Reviewed head 43a471c (rebased onto fe52694, CI double-green). This is the tool I used myself this cycle, which is the strongest evidence for it: run over the whole 8-PR queue it returned FRESH for all seven rebased siblings and STALE for the one PR whose CI was still pending ("no verdict yet"), exit 1 accordingly. That is the correct discrimination — ancestry answers "would a verdict transfer", the per-SHA passing-run check answers "is there a verdict", and it declines to call a failing run fresh. Asking the graph instead of comparing timestamps is the right call: #1137 merged cleanly (MERGEABLE/CLEAN) and was still unsafe, so is the misleading shortcut that this replaces.
|
Correction to the review above: the final sentence lost a word to shell substitution — it should read "so |
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cyc20260911-120717 (2/3)
Reviewed head 43a471c (CI double-green, MERGEABLE, FRESH). This is the tool whose output I act on every cycle, so I verified it against live ground truth rather than its own tests: run over the current 8-PR queue it returned FRESH for all seven rebased siblings and correctly STALE for the one whose CI was still pending ("no verdict yet"), exit 1. The per-workflow pinning is right — "some passing run" is only the test verdict while nothing else happens to run on a PR head.
The design choice worth endorsing explicitly: asking the graph (is master's tip an ancestor of the head?) instead of comparing timestamps. Two timestamps are a proxy that can be wrong under clock skew or a run created a second before the merge commit; ancestry is the thing itself, and it gets the "master has not moved" case right for free. Ancestry alone would still be half an answer, and the second condition — a passing run exists for that exact SHA — is what makes it a verdict rather than a prediction.
This addresses the failure mode measured in #1137: MERGEABLE/CLEAN with both jobs green, and the merge still unsafe, because on pull_request GitHub builds Merge <head> into <merge-base> — the branch point, not current master. The cleanly auto-resolved count line was the dangerous case. At this head: 14 passed.
…creen (#1139) Every recent cycle re-derived the merge rule by hand from the comment history, and got it wrong at least once. #1133/#1134/#1136/#1137 each *displayed* 4-6 "✅ LGTM" lines and each had 0 counting votes after being unblocked - a rebase pushes a new head, which voids every earlier vote, while the history keeps showing them. `scripts/check-vote-count.py <PR>...` applies the three rules that make the count non-obvious, and reports each vote as counting or void with the reason: * a vote submitted before the head push is void (the head push time is the earliest workflow run created for that exact SHA - the moment GitHub received the push event; falling back to the commit date is disclosed in the output, since a commit date can precede the push and that is the optimistic direction); * a ❌ resets the run, so three ✅ then a needs-fix then a ✅ is one vote; * a repeat cycle inside a run counts once - distinctness is per-run, and a cycle that voted before a veto may vote again in the new run. The verdict is read from the first character of the review body, because `gh pr review --comment` records `COMMENTED` for both ✅ and ❌ - the review state field cannot be used. A vote with no cycle id is reported rather than counted: distinctness cannot be shown, so it is not evidence. Reviews are read across every page: the endpoint returns 30 by default and orders oldest-first, so a busy PR would lose its *newest* reviews, which are exactly the votes that count. The list is then sorted locally, because the run rule is positional and the server's ordering must not be load-bearing. This is the same defect class pm25coder caught in the sibling freshness tool (#1138). Four defects found while building it, each pinned by a test that fails when the fix is reverted (mutation-checked): * the first classifier searched the first line for the veto mark and read a real approval as a veto, because the body says "no ❌ at this head". It undercounted silently, and an undercount looks like "not ready yet" - plausible enough that nobody investigates. The mark must *begin* the body. * the mark column rendered "OK ... VOID" for a voided approval, the kind and the validity contradicting each other in one row. It now answers the only question the reader has: does this vote count? * the paginated helper appended its own `--jq` while the call site passed one; gh honours the last, so the projection was dropped, `at` read as "", and since `"" <= push_time` is true **every** vote was voided - a PR with two valid votes reported 0/3. Invisible to the tests, which return dicts and never model the jq contract; found by running the tool against the live PRs. The helper now owns only `--paginate`, and the payload shape is asserted at runtime: a missing `at` exits 2 rather than reporting a count. * Agent.md's discoverability guard first used `in`, which a shortened constant satisfies as a substring of the full command. Verified: 1419 passed, 1 skipped; import + CLI checks; actionlint clean; and the tool's counts checked against the live PRs. Co-authored-by: EMRG Evolution <emrg@argszero.dev>
What
Adds
scripts/check-merge-freshness.pyplus a doc note, answering one question a committer has to answer before merging:Why — the measured case
PR #1137 was
MERGEABLE/CLEANwith both CI jobs green, and merging it still produced a tree that failed two guards:Both sides set the count line to the same number, so git merged it without a conflict, kept 1397, and the merged tree collected 1401.
test_doc_counts.py::test_python_count_matches_docsandtest_check_doc_count.py::test_real_tree_is_consistentwent red — after the merge, on master.The CI run was not wrong. It was about a different tree. On
pull_requestGitHub buildsMerge <head> into <merge-base>— the head merged onto the branch point, not onto current master. While the branch point is master's tip those are the same commit; once master moves they are not, and nothing re-runs the check, because master moving is not a branch push and fires nosynchronizeevent. This was confirmed from the run's own log:gh pr view --json mergeablereportsCLEANthroughout, because it answers "does this textually merge" — the property that failed. The cleanly-merged case is the dangerous one: when the count line conflicts, a human is forced to look at it and--resolve-conflictmeasures; when both sides land on the same number there is nothing to notice.How it decides
Structural, not by comparing timestamps:
Ancestry is the thing itself; two timestamps are a proxy that can be wrong (clock skew, a run created seconds before the merge commit). It also gets "master has not moved" right for free — no churn.
Ancestry alone is only half the question, so there is a second condition: a head that contains master but has no CI run at all (a dropped push event ⇒
no checks reported) is not fresh either. A head is FRESH only if master is an ancestor and a passing run exists for that exact SHA — keyed on the SHA, not the branch, since a branch pushed twice has two runs.Exit codes:
0all fresh ·1at least one stale ·2the check could not be made (bad PR,ghfailure, unrecognised compare status). An unrecognised status is fail-loud by construction — the freshness sets are named rather than written asstatus == "ahead", so a new GitHub value cannot silently read as fresh.Verification
status=ahead, merge base64bab529= master's tip); the four pre-rebase heads — including emrg: give the Windows CI job a Node toolchain, and guard that it keeps one #1137'sea0a06a— reportdiverged/behind_by=1, i.e. STALE.ghprogram name; making the stale branch unreachable; keying runs on the branch instead of the SHA; treatingpendingas a pass; letting an unrecognised status fall through as fresh.ghprogram name, so["pr", "view", ...]invoked the POSIXprutility and failed withpr: cannot open view— a message naming neitherghnor the mistake. Now pinned by a test._gh_jsoninjected; the fake asserts it was actually called, so a test cannot pass by never querying). Full suite1408 passed, 1 skipped;actionlintclean; import +emrg --helpgreen; Agent.md count re-measured to 1409.Note
This is the host/committer-side half of the same gap; the tool is deliberately not wired into CI, which cannot see master's future tip. It is meant to be run immediately before
gh pr merge.