feat(benchmark): add in-process review executor behind --in-process (A6) - #483
Merged
Conversation
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements architecture proposal A6: in-process review execution for
cr benchmark run, flag-gated. Closes #453 (the dedup half shipped in #470).What
ReviewExecutorseam with two implementations:subprocessExecutor(the existing path, moved verbatim — still the default, always used with--cr-bin) andinProcessExecutor(app.Open+pipeline.DryRun, typed results serialized straight toreview.json, no--jsonstdout reparse).--in-processflag oncr benchmark run; combining it with--cr-binis a usage error. The default is not flipped — that waits on a parity run (follow-up issue).reviewcmdto sharedview.NewReviewDryRun(line-by-line verified move; only package prefixes and two unexported constructor names changed). Bothcr reviewand the executor reuse it.Judge-flagged traps, addressed
app.OpenRequestembeds per-casePRRef/MaxAgents/MaxConcurrency: a fresh runtime is opened per candidate×case with cleanup deferred per cell; a counting-fake test pins this.child_process_error— one bad cell cannot abort a multi-hour LLM-billed suite.usage_error, auth-config→3/auth_config_error, upstream→5/upstream_error, generic→1/child_exit_nonzero, cancellation/panic→−1/child_process_error.missing/invalid_review_jsonremain subprocess-only (in-process results are typed).cr_bin: "in-process"— an explicit marker rather than a misleading executable path.Verification
Zero behavior change without the flag (subprocess coverage retained, incl. the stream/exit-code capture test, renamed to the executor seam). Full unsandboxed
go test ./...green; repo-widegolangci-lintclean; architecture harness green.