Make the benchmarks able to resolve real effects - #15

Merged

brunoborges merged 19 commits into

mainfrom

improve-benchmark-statistics

Aug 5, 2026

Commits on Aug 5, 2026

Make the focused benchmark able to resolve real effects

brunoborges
and
Copilot
committed
f14f9aa
Keep the wrapper cache key stable between seeding and measuring

brunoborges
and
Copilot
committed
1a624da
Restore one shared cache entry in both arms

brunoborges
and
Copilot
committed
7761291
Stop the matrix from measuring its own cache contention

brunoborges
and
Copilot
committed
01f3f0d
Apply the paired methodology to the version sweep and JDK cache

brunoborges
and
Copilot
committed
ebd51cf
Keep PetClinic's build from linting the benchmark scripts

brunoborges
and
Copilot
committed
319c61b
Stop one stalled runner from holding a benchmark run open

brunoborges
and
Copilot
committed
a4fc67a
Discard the first slot of every measurement job

brunoborges
and
Copilot
committed
8cd13cf
Stop ranking the uncached versions against main

brunoborges
and
Copilot
committed
efd11fd
Stop a stalled slot from being reported as an effect

brunoborges
and
Copilot
committed
77334eb
Bound each measured setup so a stalled restore costs one runner

brunoborges
and
Copilot
committed
c017f73
Bound how long a failed cache download can stall a slot

brunoborges
and
Copilot
committed
5f3c985
Discard stalled runners in the version sweep too

brunoborges
and
Copilot
committed
de4e834
Give the configuration matrix a verdict it can support

brunoborges
and
Copilot
committed
e5f870c
Fail on missing report inputs instead of publishing undefined

brunoborges
and
Copilot
committed
8a3c51c
Compare version benchmarks within compatible cohorts

brunoborges
and
Copilot
committed
ac70095
Rank only the versions that do the same work

brunoborges
and
Copilot
committed
2b3afa4
Report the sweep as main against each version, not the reverse

brunoborges
and
Copilot
committed
a967acc
Ask each benchmark a question it can answer

brunoborges
and
Copilot
committed
e1cb348

Navigation Menu

Uh oh!

Make the benchmarks able to resolve real effects - #15

Make the benchmarks able to resolve real effects#15
brunoborges merged 19 commits into
mainfrom
improve-benchmark-statistics

Commits on Aug 5, 2026

Make the focused benchmark able to resolve real effects

Keep the wrapper cache key stable between seeding and measuring

Restore one shared cache entry in both arms

Stop the matrix from measuring its own cache contention

Apply the paired methodology to the version sweep and JDK cache

Keep PetClinic's build from linting the benchmark scripts

Stop one stalled runner from holding a benchmark run open

Discard the first slot of every measurement job

Stop ranking the uncached versions against main

Stop a stalled slot from being reported as an effect

Bound each measured setup so a stalled restore costs one runner

Bound how long a failed cache download can stall a slot

Discard stalled runners in the version sweep too

Give the configuration matrix a verdict it can support

Fail on missing report inputs instead of publishing undefined

Compare version benchmarks within compatible cohorts

Rank only the versions that do the same work

Report the sweep as main against each version, not the reverse

Ask each benchmark a question it can answer