Let /benchmark compare against another framework

.github/PULL_REQUEST_TEMPLATE.md

@@ -8,11 +8,21 @@
 
 | Command | Description |
 |---------|-------------|
-| `/benchmark -f <framework>` | Run all benchmark tests |
-| `/benchmark -f <framework> -t <test>` | Run a specific test |
-| `/benchmark -f <framework> --save` | Run and save results (updates leaderboard on merge) |
+| `/benchmark -f <framework>` | Run every test the framework subscribes to |
+| `/benchmark -f <framework> -t <test>` | Run one test only |
+| `/benchmark -f <framework> --save` | Run and save results (updates the leaderboard on merge) |
+| `/benchmark -f <framework> -t <test> --save` | Run one test and save results |
+| `/benchmark -f <framework> --compare <other>` | Measure the deltas against another framework instead of this one |
 
-Always specify `-f <framework>`. Results are automatically compared against the current leaderboard.
+Always specify `-f <framework>`; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory.
+
+**What the deltas are measured against.** By default, this framework's own results published on `main` - answering *"did this change help?"*. When you are tuning a variant or a successor entry, `--compare` re-bases them on another entry instead:
+
+```
+/benchmark -f genhttp-11 --compare genhttp
+```
+
+The reply states which baseline it used, and profiles the other framework does not run show `n/a` rather than a delta.
 
 ---
 

.github/workflows/benchmark-pr.yml

@@ -17,6 +17,10 @@ on:
         description: 'Save results (true/false)'
         required: false
         default: ''
+      compare:
+        description: 'Compare against another framework instead of this one (e.g. genhttp)'
+        required: false
+        default: ''
 
 permissions:
   contents: write
@@ -108,14 +112,20 @@ jobs:
 
       - name: Compare with main
         id: compare
+        env:
+          FRAMEWORK: ${{ inputs.framework }}
+          PROFILE: ${{ inputs.profile }}
+          COMPARE: ${{ inputs.compare }}
         run: |
           # Preserve benchmark results before overwriting site/data for comparison
           cp -r site/data /tmp/bench_site_data 2>/dev/null || true
           # Use main's site/data for comparison
           if [ -d /tmp/main_site_data ]; then
             cp -r /tmp/main_site_data/* site/data/ 2>/dev/null || true
           fi
-          comparison=$(./scripts/compare.sh "${{ inputs.framework }}" "${{ inputs.profile }}" 2>/dev/null || echo "")
+          compare_args=()
+          [ -n "$COMPARE" ] && compare_args=(--compare "$COMPARE")
+          comparison=$(./scripts/compare.sh "$FRAMEWORK" "$PROFILE" "${compare_args[@]}" 2>/dev/null || echo "")
           echo "$comparison" > /tmp/bench_comparison.md
           # Restore benchmark site/data (not original PR data)
           if [ -d /tmp/bench_site_data ]; then
@@ -232,7 +242,9 @@ jobs:
           ' "$log" "$fw" /tmp/bench_comparison.md
 
           body="## Benchmark Results"$'\n\n'
-          body+="**Framework:** \`${fw}\` | **Test:** \`${profile_label}\`"$'\n\n'
+          body+="**Framework:** \`${fw}\` | **Test:** \`${profile_label}\`"
+          [ -n "${{ inputs.compare }}" ] && body+=" | **Compared to:** \`${{ inputs.compare }}\`"
+          body+=$'\n\n'
           body+=$(cat /tmp/bench_body.txt 2>/dev/null || echo "No results captured")
           body+=$'\n\n'
 

.github/workflows/pr-commands.yml

@@ -64,18 +64,33 @@ jobs:
             EXPLICIT_TEST=$(echo "$COMMENT_BODY" | grep -oP '/benchmark\s+\K[^-]\S*' || echo "")
           fi
 
+          # Parse --compare <framework>: baseline the deltas on another entry's
+          # published results instead of this framework's own (#741).
+          COMPARE_FW=$(echo "$COMMENT_BODY" | grep -oP '/benchmark\s+.*--compare[= ]\K\S+' || echo "")
+
           # Parse --save flag
           SAVE_FLAG=""
           if echo "$COMMENT_BODY" | grep -q '\-\-save'; then
             SAVE_FLAG="true"
           fi
 
           FRAMEWORK="${EXPLICIT_FW:-$AUTO_FRAMEWORK}"
+
+          # These values come from a PR comment, which anyone can write, and are
+          # interpolated into later steps. Restrict them to the characters a
+          # framework directory or profile name can actually contain, so a
+          # crafted comment cannot smuggle shell syntax through
+          # (e.g. `--compare a";touch${IFS}/tmp/x;#`). Anything else becomes empty.
+          safe() { printf '%s' "$1" | grep -oP '^[A-Za-z0-9._-]{1,64}$' || true; }
+          FRAMEWORK=$(safe "$FRAMEWORK")
+          EXPLICIT_TEST=$(safe "$EXPLICIT_TEST")
+          COMPARE_FW=$(safe "$COMPARE_FW")
           echo "framework=$FRAMEWORK" >> "$GITHUB_OUTPUT"
           echo "profile=$EXPLICIT_TEST" >> "$GITHUB_OUTPUT"
           echo "save=$SAVE_FLAG" >> "$GITHUB_OUTPUT"
+          echo "compare=$COMPARE_FW" >> "$GITHUB_OUTPUT"
           echo "Detected framework: $FRAMEWORK (auto: $AUTO_FRAMEWORK, explicit: $EXPLICIT_FW)"
-          echo "Profile: $EXPLICIT_TEST, Save: $SAVE_FLAG"
+          echo "Profile: $EXPLICIT_TEST, Save: $SAVE_FLAG, Compare: ${COMPARE_FW:-<own published results>}"
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           COMMENT_BODY: ${{ github.event.comment.body }}
@@ -114,14 +129,22 @@ jobs:
 
       - name: Run benchmark
         if: steps.parse.outputs.framework != ''
-        run: |
-          gh workflow run benchmark-pr.yml \
-            -f pr=${{ steps.parse.outputs.pr }} \
-            -f framework=${{ steps.parse.outputs.framework }} \
-            -f profile="${{ steps.parse.outputs.profile }}" \
-            -f save="${{ steps.parse.outputs.save }}"
+        # Values reach the shell as environment variables rather than being
+        # pasted into the script text, so nothing in them can be parsed as code.
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PR_NUMBER: ${{ steps.parse.outputs.pr }}
+          FRAMEWORK: ${{ steps.parse.outputs.framework }}
+          PROFILE: ${{ steps.parse.outputs.profile }}
+          SAVE: ${{ steps.parse.outputs.save }}
+          COMPARE: ${{ steps.parse.outputs.compare }}
+        run: |
+          gh workflow run benchmark-pr.yml \
+            -f pr="$PR_NUMBER" \
+            -f framework="$FRAMEWORK" \
+            -f profile="$PROFILE" \
+            -f save="$SAVE" \
+            -f compare="$COMPARE"
 
       - name: No framework detected
         if: steps.parse.outputs.framework == ''

README.md

@@ -18,12 +18,21 @@ HTTP framework benchmark platform.
 
 | Command | Description |
 |---------|-------------|
-| `/benchmark -f <framework>` | Run all benchmark tests |
-| `/benchmark -f <framework> -t <test>` | Run a specific test |
-| `/benchmark -f <framework> --save` | Run and save results (updates leaderboard on merge) |
-| `/benchmark -f <framework> -t <test> --save` | Run specific test and save results |
+| `/benchmark -f <framework>` | Run every test the framework subscribes to |
+| `/benchmark -f <framework> -t <test>` | Run one test only |
+| `/benchmark -f <framework> --save` | Run and save results (updates the leaderboard on merge) |
+| `/benchmark -f <framework> -t <test> --save` | Run one test and save results |
+| `/benchmark -f <framework> --compare <other>` | Measure the deltas against another framework instead of this one |
 
-Always specify `-f <framework>`. Results are automatically compared against the current leaderboard.
+Always specify `-f <framework>`; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory.
+
+**What the deltas are measured against.** By default, this framework's own results published on `main` - answering *"did this change help?"*. When you are tuning a variant or a successor entry, `--compare` re-bases them on another entry instead:
+
+```
+/benchmark -f genhttp-11 --compare genhttp
+```
+
+The reply states which baseline it used, and profiles the other framework does not run show `n/a` rather than a delta.
 
 ---
 

scripts/compare.sh

@@ -1,17 +1,38 @@
 #!/usr/bin/env bash
 # Compare benchmark results for a framework against published data on main.
-# Usage: ./scripts/compare.sh <framework> [profile]
+#
+# Usage: ./scripts/compare.sh <framework> [profile] [--compare <framework>]
+#
+# By default the baseline is the same framework's own published results, which
+# answers "did my change help?". --compare swaps in another framework's
+# published results instead, which answers "how does this tuned build stack up
+# against the entry it derives from?" — e.g.
+#
+#     ./scripts/compare.sh genhttp-11 --compare genhttp
+#
 # Output: Markdown table with deltas (suitable for PR comments)
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ROOT_DIR="$SCRIPT_DIR/.."
 
-FRAMEWORK="${1:-}"
-PROFILE_FILTER="${2:-}"
+FRAMEWORK=""
+PROFILE_FILTER=""
+COMPARE_FRAMEWORK=""
+POSITIONAL=()
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --compare)   COMPARE_FRAMEWORK="${2:-}"; shift ;;
+        --compare=*) COMPARE_FRAMEWORK="${1#*=}" ;;
+        *)           POSITIONAL+=("$1") ;;
+    esac
+    shift
+done
+FRAMEWORK="${POSITIONAL[0]:-}"
+PROFILE_FILTER="${POSITIONAL[1]:-}"
 
 if [ -z "$FRAMEWORK" ]; then
-    echo "Usage: $0 <framework> [profile]" >&2
+    echo "Usage: $0 <framework> [profile] [--compare <framework>]" >&2
     exit 1
 fi
 
@@ -23,6 +44,20 @@ if [ -f "$META_FILE" ]; then
     [ -n "$dn" ] && DISPLAY_NAME="$dn"
 fi
 
+# The baseline is looked up in site/data by display_name, so resolve the
+# comparison framework's the same way. A directory that doesn't exist is a
+# typo worth reporting rather than silently comparing against nothing.
+COMPARE_DISPLAY=""
+if [ -n "$COMPARE_FRAMEWORK" ]; then
+    COMPARE_META="$ROOT_DIR/frameworks/$COMPARE_FRAMEWORK/meta.json"
+    if [ ! -f "$COMPARE_META" ]; then
+        echo "No such framework to compare against: \`$COMPARE_FRAMEWORK\` (frameworks/$COMPARE_FRAMEWORK/meta.json not found)"
+        exit 0
+    fi
+    COMPARE_DISPLAY=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1])).get('display_name',sys.argv[2]))" "$COMPARE_META" "$COMPARE_FRAMEWORK" 2>/dev/null)
+    [ -n "$COMPARE_DISPLAY" ] || COMPARE_DISPLAY="$COMPARE_FRAMEWORK"
+fi
+
 # Find new results (just benchmarked)
 RESULTS_DIR="$ROOT_DIR/results"
 SITE_DATA="$ROOT_DIR/site/data"
@@ -36,6 +71,12 @@ display_name = sys.argv[2]
 results_dir = sys.argv[3]
 site_data = sys.argv[4]
 profile_filter = sys.argv[5] if len(sys.argv) > 5 else ''
+compare_display = sys.argv[6] if len(sys.argv) > 6 else ''
+
+# Baseline: this framework's own published results by default, another
+# framework's when --compare was given.
+baseline_name = compare_display or display_name
+cross = bool(compare_display)
 
 # Find new results
 new_results = {}
@@ -75,7 +116,7 @@ for key, new_data in new_results.items():
             with open(site_file) as f:
                 entries = json.load(f)
             for entry in entries:
-                if entry.get('framework') == display_name:
+                if entry.get('framework') == baseline_name:
                     old_results[key] = entry
                     break
         except:
@@ -98,7 +139,7 @@ def fmt_rps(n):
 
 def delta_pct(new_val, old_val):
     if old_val is None or old_val == 0 or new_val is None:
-        return 'NEW'
+        return 'n/a' if cross else 'NEW'
     pct = ((new_val - old_val) / old_val) * 100
     if abs(pct) < 0.1:
         return '~0%'
@@ -151,6 +192,14 @@ for key in sorted(new_results.keys()):
 # Output markdown
 has_comparison = bool(old_results)
 
+if cross:
+    if has_comparison:
+        print(f'Deltas are against **{baseline_name}** published on \`main\`, not against '
+              f'\`{display_name}\`\'s own results.')
+    else:
+        print(f'**{baseline_name}** has no published results for these profiles - nothing to compare against.')
+    print()
+
 for profile, conns_list in profiles.items():
     print(f'### {profile}')
     print()
@@ -189,4 +238,4 @@ for profile, conns_list in profiles.items():
         print(row)
 
     print()
-" "$FRAMEWORK" "$DISPLAY_NAME" "$RESULTS_DIR" "$SITE_DATA" "$PROFILE_FILTER"
+" "$FRAMEWORK" "$DISPLAY_NAME" "$RESULTS_DIR" "$SITE_DATA" "$PROFILE_FILTER" "$COMPARE_DISPLAY"

site/content/docs/running-locally/scripts/compare.md

@@ -6,7 +6,7 @@ weight: 4
 Compare a framework's benchmark results against the published leaderboard data on the main branch. Outputs a Markdown table with deltas, suitable for PR comments.
 
 ```bash
-./scripts/compare.sh <framework> [profile]
+./scripts/compare.sh <framework> [profile] [--compare <framework>]
 ```
 
 ## Options
@@ -15,6 +15,7 @@ Compare a framework's benchmark results against the published leaderboard data o
 |-----------|-------------|
 | `<framework>` | Name of the framework to compare |
 | `[profile]` | Optional - compare only this test profile |
+| `--compare <framework>` | Optional - use another framework's published results as the baseline instead of this framework's own |
 
 ## What it does
 
@@ -23,6 +24,22 @@ Compare a framework's benchmark results against the published leaderboard data o
 3. Matches the framework by its `display_name` from `meta.json`
 4. Outputs a Markdown table for each profile with columns per connection count
 
+## Comparing against a different framework
+
+By default the baseline is the framework's own published results, which answers *"did my change help?"*. When you are developing a tuned or successor entry, the more useful question is often *"how does this compare to the entry it derives from?"* - `--compare` swaps the baseline:
+
+```bash
+./scripts/compare.sh genhttp-11 --compare genhttp
+```
+
+Every delta is then measured against `genhttp`'s published results, and the output says so explicitly so the numbers cannot be mistaken for a regression against the framework's own history. Profiles the comparison framework does not run show `n/a` rather than `NEW`.
+
+The same flag is available from a pull request:
+
+```
+/benchmark -f genhttp-11 --compare genhttp
+```
+
 ## Metrics compared
 
 For each profile and connection count, the table shows:
@@ -51,4 +68,4 @@ Each value includes a percentage delta against the main branch. New frameworks w
 
 ## CI usage
 
-The `benchmark-pr.yml` workflow calls this script automatically after benchmarking a PR branch, and posts the comparison table as a PR comment.
+The `benchmark-pr.yml` workflow calls this script automatically after benchmarking a PR branch, and posts the comparison table as a PR comment. Adding `--compare <framework>` to the `/benchmark` command forwards it to this script, and the comment header records which framework the deltas are against.