The worked proof that CodeWeave finds energy/latency hotspots in real software and ships only the changes it can prove. This is one complete 8-phase run against PyTorch built from source (CPU), running distilgpt2 text generation in a fixed-time hot loop.
Every figure below is drawn from committed artifacts under integration-test/reports/ — primarily baseline.json, family-correction.json, the per-cycle ab-comparison-opt*.json, and the synthesized phase8-report.md. Per-iteration metrics are the verdict inputs throughout; wall-clock time is pinned by the fixed-time loop and never used.
| Variable | Value | Meaning |
|---|---|---|
| Target | PyTorch, built from source, CPU-only | A/B compares two builds of the same source tree |
GENAI_MODEL |
distilgpt2 |
Model the benchmark exercises |
GENAI_MAX_SECONDS |
30 |
Fixed-time budget per measurement run |
| Energy tracking | CodeCarbon EmissionsTracker |
Records joules and CO₂ per run |
Source: integration-test/reports/baseline.json (5 runs).
| Metric | Value |
|---|---|
| Median iteration latency (mean) | 2773.65 ms (std 22.47 ms) |
Coefficient of variation (cv_iter) |
0.81% |
Noise floor — Minimum Detectable Effect (mde_pct) |
1.62% |
| Iterations per run | 11.0 |
| Energy per iteration | 52.43 J |
| Mean CO₂ per run | 0.0429 (CodeCarbon units) |
| Stable / energy valid | true / true |
The 1.62% MDE is the smallest end-to-end change trustworthy at this run count. Ops whose end-to-end effect falls below it by construction are judged on a per-op microbenchmark instead.
Source: family-correction.json + per-cycle ab-comparison-opt*.json. Verdicts are the family-corrected decision. Ranked KEEP → INVESTIGATE → REVERT.
| Opt | Target op | File changed | Verdict | Signal | Headline Δ (p) | Δ energy/iter | Fuzz |
|---|---|---|---|---|---|---|---|
| 4 | aten::topk |
TopKImpl.h |
KEEP | microbench | +41.18% op-level (p≈0) | +0.63% | 48 cases, 0 ULP |
| 2 | aten::exponential_ |
DistributionKernels.cpp |
KEEP | microbench | +2.75% op-level (p=6.96e⁻¹⁸) | +0.32% | 64 cases, 0 ULP |
| 1 | aten::addmm |
LinearAlgebra.cpp |
KEEP | e2e | +2.18% end-to-end (p=0.003731) | +2.06% | 24 cases, 0 ULP |
| 5 | aten::multinomial |
MultinomialKernel.cpp |
INVESTIGATE | trend | +0.47% op-level (p=0.0033, below 2.0% floor) | +1.76% | 64 cases, 0 ULP |
| 3 | aten::cat |
TensorShape.cpp |
REVERT | micro-regression | −8.62% op-level (p=1.65e⁻¹¹⁰) | +0.28% | — |
Opt 1 details (ab-comparison-opt1.md): baseline median 2836.16 ms → variant 2774.38 ms; Welch t=4.25, Cohen's d=2.69; e2e MDE 1.90%. The op-level microbench delta (−0.23%, p=0.796) is not significant — consistent with the change cutting dispatch overhead rather than the inner GEMM.
Source: family-correction.json (Holm-Bonferroni, FWER = 0.05, family size = 5).
- Demoted: 0 of 5. Every per-cycle verdict survived the correction.
- PR-worthy (KEEP): 3 — opt 4, opt 2, opt 1.
- INVESTIGATE: 1 — opt 5 (positive trend, below the practical floor; recommend a higher-power re-measurement).
- REVERT: 1 — opt 3 (confirmed regression; not submitted).
The paired A/B design (fresh base re-measured back-to-back with each variant) protects each verdict from cross-cycle machine drift. Drift is still reported against the Phase 6 anchor:
| Opt | A-side median | Drift vs baseline | Effect on verdict |
|---|---|---|---|
| 1 | 2836.16 ms | +2.25% ⚠ | Margin (+2.18%) ≈ drift — confirming re-run recommended before PR |
| 2 | 2884.35 ms | +3.99% ⚠ | None — signal is op-level microbench, not e2e |
| 3 | 2917.45 ms | +5.18% ⚠ | None — regression (−8.62%) far exceeds drift |
| 4 | 2744.94 ms | −1.03% | None |
| 5 | 2753.97 ms | −0.71% | None |
- CodeWeave rejected a change that looked like an optimization (opt 3,
aten::cat) because measurement said it was an 8.6% regression — exactly the failure mode an unverified AI patch would ship. - It shipped only proven wins, each bit-exact under differential fuzz before any timing run, each surviving a family-wise correction.
- It flagged its own uncertainty honestly: opt 1's improvement margin is close to the measured environment drift, so the report recommends a confirming re-run rather than presenting it as settled — and opt 5 is held at INVESTIGATE rather than shipped.
Recommended PR submission order (from phase8-report.md): opt 4 → opt 2 → opt 1 (opt 1 after a confirming re-run). Full per-optimization PR drafts are in integration-test/reports/phase8-report.md.