fix: framework detection in /validate and /benchmark commands - #33
Merged
Conversation
gh pr diff --name-only fails silently with shallow clones.
Use /pulls/{pr}/files API endpoint instead.
MDA2AV
added a commit
that referenced
this pull request
Jul 24, 2026
Closes #702. Each profile's 100-point reference was computed over the rows left after filtering, so filtering changed the scores themselves. Because every profile's leader moves by a different factor when a subset is selected, the sum could reorder — the reported case, reproduced exactly: main this branch unfiltered rage 265 (#33) roda 233 (#36) rage 265 (#33) roda 233 (#36) filter ruby rage 678 (#3) roda 775 (#1) rage 265 (#1) roda 233 (#2) Rage outscores Roda overall but placed below it as soon as the board was filtered to Ruby, with both scores inflated ~2.9x. The predicate that decided which rows to show was also deciding what to normalize against. Those are two different questions, so they are now two predicates: outOfLeague is this entry ranked somewhere else entirely? Types are separate leagues — engine entries are scored on their own subset of profiles, so an engine result must never set the reference for a framework entry. hidden is it merely filtered out of view? The search box and the tuned toggle are display filters and now have no effect on any score. Normalization uses only the first, so the numbers are a property of the framework rather than of the current view. This covers the per-profile maxima, the memory-efficiency maxima, and the json-comp bytes-per-response reference, which had the same problem. Per the discussion on the issue, the old behaviour is still worth having — comparing a subset against itself is a real use case — so it becomes an explicit "Rescale to selection" toggle next to the memory one, off by default, carried in the URL as rescale=1. With it on, the filtered view reproduces the previous numbers exactly (678 / 775 above). The header blurb states which rule is active, so a screenshot is never ambiguous. Verified in headless Chrome against the real data, on this branch and on main: unfiltered ordering is unchanged, filtering to Ruby now leaves both scores untouched, and toggling rescale on reproduces main's filtered output value for value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the 'No framework detected' issue when using
/benchmarkor/validateon PRs.Problem:
gh pr diff --name-onlyfails silently on the self-hosted runner, returning empty output.Fix: Use the GitHub API (
/pulls/{pr}/files) to get the list of changed files instead. This is more reliable and doesn't depend on git clone depth.