Skip to content

feat(benchmark): add in-process review executor behind --in-process (A6) - #483

Merged
rianjs merged 1 commit into
mainfrom
refactor/453-benchmark-executor
Jul 10, 2026
Merged

feat(benchmark): add in-process review executor behind --in-process (A6)#483
rianjs merged 1 commit into
mainfrom
refactor/453-benchmark-executor

Conversation

@rianjs

@rianjs rianjs commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Implements architecture proposal A6: in-process review execution for cr benchmark run, flag-gated. Closes #453 (the dedup half shipped in #470).

What

  • New ReviewExecutor seam with two implementations: subprocessExecutor (the existing path, moved verbatim — still the default, always used with --cr-bin) and inProcessExecutor (app.Open + pipeline.DryRun, typed results serialized straight to review.json, no --json stdout reparse).
  • --in-process flag on cr benchmark run; combining it with --cr-bin is a usage error. The default is not flipped — that waits on a parity run (follow-up issue).
  • The dry-run mapping moved mechanically from reviewcmd to shared view.NewReviewDryRun (line-by-line verified move; only package prefixes and two unexported constructor names changed). Both cr review and the executor reuse it.

Judge-flagged traps, addressed

  • app.OpenRequest embeds per-case PRRef/MaxAgents/MaxConcurrency: a fresh runtime is opened per candidate×case with cleanup deferred per cell; a counting-fake test pins this.
  • Panics (including cleanup panics) are recovered per cell and recorded as child_process_error — one bad cell cannot abort a multi-hour LLM-billed suite.
  • Failure-taxonomy parity: usage→2/usage_error, auth-config→3/auth_config_error, upstream→5/upstream_error, generic→1/child_exit_nonzero, cancellation/panic→−1/child_process_error. missing/invalid_review_json remain subprocess-only (in-process results are typed).
  • Manifests record cr_bin: "in-process" — an explicit marker rather than a misleading executable path.

Verification

Zero behavior change without the flag (subprocess coverage retained, incl. the stream/exit-code capture test, renamed to the executor seam). Full unsandboxed go test ./... green; repo-wide golangci-lint clean; architecture harness green.

@rianjs
rianjs merged commit c7652c2 into main Jul 10, 2026
10 checks passed
@rianjs
rianjs deleted the refactor/453-benchmark-executor branch July 10, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor Phase 6: benchmark dedup + in-process review executor

1 participant