Skip to content

test(ci): gate PRs on changed-line coverage (#1418) - #1447

Merged
thymikee merged 3 commits into
mainfrom
devin/1785235385-changed-line-coverage-gate
Jul 28, 2026
Merged

test(ci): gate PRs on changed-line coverage (#1418)#1447
thymikee merged 3 commits into
mainfrom
devin/1785235385-changed-line-coverage-gate

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a changed-line coverage gate to the Coverage CI job (umbrella #1412 Track B). The gate joins the lcov report pnpm test:coverage already wrote with git diff --unified=0 <base>...HEAD and scores coverage of the contributor's own added lines only, failing when it drops below a single-source threshold. Closes #1418.

New scripts/coverage-changed/ (mirrors the scripts/check-affected/ split — pure model.ts + I/O run.ts + adjacent tests; parsing is hand-rolled to keep the repo dependency-free, matching check-affected):

  • Gate: changed-line coverage < CHANGED_LINE_COVERAGE_THRESHOLD (70, declared once in model.ts) fails the job. pnpm check:coverage-changed reuses coverage/lcov.info; it never runs coverage a second time.
  • Points at the contributor's lines: failure output lists each offending file with its uncovered changed line numbers (added lines whose DA: hits are 0).
  • Deterministic vs renames/deletes: only new-side + lines count. +++ /dev/null (delete) and pure renames contribute nothing; a rename+edit's added lines are attributed to the destination path.
  • Exclusions match vitest, without a second copy of the globs. vitest runs coverage with all on, so every includable src/**/*.ts file appears in lcov. The coverable universe is therefore read from lcov itself; an includable source file absent from lcov was dropped by an exclude glob. Test files are excluded too.
  • Escape hatch: the coverage-waiver PR label sets AGENT_DEVICE_COVERAGE_WAIVER, which skips the failure but still prints every number to the job summary.
  • Non-gating rows for the quality-delta comment: changed-branch coverage (from BRDA) and the count of changed executable lines excluded from coverage (excluded paths + v8/c8/istanbul ignore directives), so exclusions cannot silently absorb new logic.

Kept deliberately small: one markdown renderer feeds both stdout and GITHUB_STEP_SUMMARY, and the CLI surface is just --base (waiver comes from the env var CI already sets).

CI wiring in the Coverage job:

- name: Test changed-line coverage gate
  run: pnpm check:coverage-changed:test          # fast node --test, before the 30-min coverage run
- name: Run coverage
  run: pnpm test:coverage                          # unchanged — produces coverage/lcov.info
- name: Enforce changed-line coverage gate
  if: always() && github.event_name == 'pull_request'
  env:
    AGENT_DEVICE_COVERAGE_WAIVER: ${{ contains(github.event.pull_request.labels.*.name, 'coverage-waiver') }}
  run: pnpm check:coverage-changed --base "${{ github.event.pull_request.base.sha }}"

Core join (see computeChangedCoverage):

for each changed file with added lines that is includable src (non-test):
  cov = lcov[path]
  if !cov            -> excluded-by-path: tally its code-like added lines (non-gating)
  else               -> total += added ∩ DA;  covered += those with hits>0
                        uncovered = added ∩ DA(hits==0)
                        + branches on added lines (non-gating)
                        + added lines dropped by an ignore directive (non-gating tally)
pct = covered/total  (null when total==0 → docs-only PR is a trivial pass)
passed = waived || pct==null || pct >= 70

Testing

  • pnpm check:coverage-changed:test (17 cases: lcov/diff parsing incl. rename & delete, threshold pass/fail, waiver + job summary, docs-only n/a, excluded-path + ignore tallies, real-git end-to-end in run.test.ts).
  • pnpm format:check, pnpm lint, pnpm typecheck, pnpm check:fallow green on the diff.

Link to Devin session: https://app.devin.ai/sessions/c5a4c346eb64422ba381d1d638932f7a
Requested by: @thymikee

thymikee and others added 2 commits July 28, 2026 10:43
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@thymikee thymikee self-assigned this Jul 28, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-28 12:25 UTC

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.87 MB 1.87 MB 0 B
JS gzip 598.7 kB 598.7 kB 0 B
npm tarball 714.4 kB 714.4 kB +32 B
npm unpacked 2.50 MB 2.50 MB +250 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 25.8 ms 26.4 ms +0.6 ms
CLI --help 55.4 ms 56.0 ms +0.6 ms

Top changed chunks: no changes in the largest emitted chunks.

@thymikee

Copy link
Copy Markdown
Member

Readiness review: clean. The gate uses the immutable PR base SHA, correctly joins added executable lines to lcov DA records, handles deletes and rename destinations, reports exact uncovered lines plus non-gating branch/exclusion metrics, and keeps waiver runs visible. Workflow execution and all CI checks are green. Residual risk is the known #1429 gate-of-gates gap: this PR does not itself prove the workflow step can never be removed.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 28, 2026
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@thymikee
thymikee merged commit 61f8696 into main Jul 28, 2026
27 checks passed
@thymikee
thymikee deleted the devin/1785235385-changed-line-coverage-gate branch July 28, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: changed-line coverage gate on PRs

1 participant