From f5cd34191a434b4cc41ea97ff5dc734d9e597e60 Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 11:26:38 +0200 Subject: [PATCH 01/10] docs: rework README around the product story Reframe the README from workflow reference documentation into a conversion-focused product story: - Lead with a one-sentence value proposition and three concrete, evidence-backed proof points (correctness gate, statistical verdict, auditable proof/ trail) instead of an implementation walkthrough. - Add an "in action" first screen: minimal config + single dispatch, and the five terminal verdict states. - Add vision, jobs-to-be-done use cases, a shortest-path quickstart with an explicit "aha" moment, outcome-oriented capabilities, a simplified how-it-works with Mermaid diagrams, and a fair comparison table. - Separate working / experimental / planned functionality and add a maturity warning. - Mark unverified facts with explicit placeholders (benchmarks, demo, portability, community files) rather than inventing them. Preserves the prior reference material in docs/ and executive-summary.md. Co-Authored-By: Claude Opus 4.8 --- README.md | 422 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 237 insertions(+), 185 deletions(-) diff --git a/README.md b/README.md index 5d62c1d..77478f1 100644 --- a/README.md +++ b/README.md @@ -1,205 +1,257 @@ # CodeWeave -CodeWeave is an automation system that drives the GitHub Copilot CLI across an **8-phase pipeline** to analyse an external codebase and then improve it under measurement. It clones a target repository and runs Copilot through two stages: +**An autonomous code-optimization pipeline that ships only the changes it can prove faster** — every candidate clears a correctness gate and wins a statistical A/B verdict before it becomes a PR. No change lands on an agent's unverified claim. -- **Documentation (Phases 1–4)** — generate an architecture book (+ PDF), Architecture Decision Records (+ PDF), a performance-measurement harness specification, and the runnable integration-test harness. Each phase is a generate→validate loop with file-based early exit: only the validator writes the completion marker, and the generator never self-certifies. -- **Measurement & optimization (Phases 5–8)** — build the target from source (ccache-backed), establish a statistical performance baseline, run autonomous optimization cycles each gated by a correctness suite (import/smoke/unit/op-suite/differential-fuzz) and an A/B verdict, then synthesise the results into a ranked report with per-optimization PR drafts. +[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](LICENSE) +![Pipeline: GitHub Actions](https://img.shields.io/badge/pipeline-GitHub%20Actions-2088FF?logo=githubactions&logoColor=white) +![Status: Experimental](https://img.shields.io/badge/status-experimental-orange) +![Target: PyTorch (CPU)](https://img.shields.io/badge/reference%20target-PyTorch%20CPU-EE4C2C) -Every iteration's changes are committed and all run artifacts are saved to `proof/`. The pipeline runs as a set of **GitHub Actions workflows** (below). **Full per-phase documentation lives in [`docs/`](docs/index.md)** (the phase table below links into it). +> _CodeWeave drives the GitHub Copilot CLI across an 8-phase pipeline: it clones a target repo, documents it from the ground up, builds a measurement harness, establishes a statistical baseline, then runs autonomous optimization cycles — each accepted or rejected by an A/B experiment, not by the agent that wrote it._ -## Workflow: CodeWeave + -**Orchestrator:** `.github/workflows/codeweave.yml` — a thin dispatcher that wires the per-phase **reusable workflows** (`.github/workflows/phase-1-book.yml` … `phase-5-6-build-baseline.yml`) via `needs`/`if` (resume + skip logic), with `finalize` inline. Shared bootstrap (Node + Copilot CLI + `codeweave.config` + git identity) lives in the `.github/actions/codeweave-setup` composite action. Phases 7–8 are the separate auto-chaining workflows `phase-7-optimize.yml` / `phase-8-report.yml`; once Phase 6 produces a baseline, `codeweave.yml`'s `trigger-phase-7` job dispatches the first optimization cycle, so a single dispatch runs Phases 1–8 end-to-end. +**Three things that make the verdicts trustworthy:** -### Overview +- **Correctness before performance.** Every change passes a six-stage gate — import + op check, integration smoke, a targeted unit test, OpInfo, an output diff, and a *blocking differential fuzz* against a golden captured on the base build — **before a single timing run is spent on it.** +- **A real speedup, not noise.** A change is only KEEP when Welch's t-test rejects (`p < 0.05`) **and** the effect clears a measured noise floor (Minimum Detectable Effect). Phase 8 then applies a Holm–Bonferroni correction across every cycle and demotes any KEEP that doesn't survive the campaign-wide test. +- **Auditable end to end.** Every generate/validate log, session transcript, gate diagnostic, and measurement record is written to `proof/`, with git history as the diff trail. Nothing is self-certified. -Phases 1–6 run inside `codeweave.yml`; the Phase 7 optimization cycles (`phase-7-optimize.yml`) and the Phase 8 report (`phase-8-report.yml`) run as two dedicated, auto-chaining workflows. After Phase 6 produces a baseline, `codeweave.yml` automatically dispatches the first Phase 7 cycle, which self-chains through the remaining cycles and into Phase 8 — so one dispatch of `codeweave.yml` carries the run through all eight phases. The eight phases are summarised below — each row links to its full `docs/` page (gate, inputs, process, outputs). +### In action -| # | Phase | What it does | -|---|-------|--------------| -| 1 | [Book generation](docs/phase-1-book-generation.md) | Iterative generate→validate passes produce an architecture book + PDF. | -| 2 | [ADR generation](docs/phase-2-adr-generation.md) | Produce Architecture Decision Records (pushed to the work branch) + PDF. | -| 3 | [Harness design](docs/phase-3-harness-design.md) | Design the three-document performance-measurement harness specification. | -| 4 | [Integration test generation](docs/phase-4-test-generation.md) | Generate the runnable harness (tests, `setup.sh`, `run.sh`, `_tools/`) behind a smoke-test gate. | -| 5 | [Source build](docs/phase-5-source-build.md) | Copilot authors `build-source.sh`; the pipeline builds the target from source (ccache-backed). | -| 6 | [Baseline execution](docs/phase-6-baseline-execution.md) | Deterministic baseline measurement (per-iteration latency + energy) → `baseline.json` + flamegraph. | -| 7 | [Optimization cycles](docs/phase-7-optimization-cycles.md) | Per-optimization generate→build→correctness-gate→A/B verdict; auto-chains one optimization per dispatch. | -| 8 | [Aggregate report](docs/phase-8-aggregate-report.md) | Synthesise all verdicts into a ranked report + per-optimization PR drafts. | +Point CodeWeave at a repository and dispatch one workflow: -> The optimization-cycle and Phase 8 workflows assume a self-hosted runner with the `src/` checkout, built `integration-test/.venv`, and warm ccache persisted from Phase 5/6 (`clean: false` checkout). Dispatch with `dry_run=true` first to smoke-test structure and the auto-chain. +```bash +# .github/codeweave.config — the whole run is configured here, nothing is hardcoded +EXTERNAL_REPO_NAME=pytorch +EXTERNAL_REPO_URL=https://github.com/your-org/pytorch +EXTERNAL_REPO_BRANCH=main +PHASE7_MAX_OPTIMIZATIONS=5 # how many hotspots to attempt -### Configuration +# one dispatch carries the run through all eight phases (Phase 7/8 auto-chain) +gh workflow run codeweave.yml -f dry_run=true # smoke-test the structure first +gh workflow run codeweave.yml # then the real run +``` -Settings are stored in `.github/codeweave.config` and loaded at runtime: +Each optimization cycle ends in one of five recorded terminal states — only the first is PR-worthy: -```env -# External repository to process -EXTERNAL_REPO_NAME=example -EXTERNAL_REPO_URL=https://github.com/example/repo.git -EXTERNAL_REPO_BRANCH=main -EXTERNAL_REPO_WORK_BRANCH=copilot-work - -# Iteration settings -PHASE1_MAX_ITERATIONS=10 -PHASE2_MAX_ITERATIONS=5 -PHASE3_MAX_ITERATIONS=5 -PHASE4_MAX_ITERATIONS=3 -PHASE6_BASELINE_RUNS=5 - -# Model schedule per phase: model:iterations,model:iterations,...,model_for_remaining -# The last entry without a count covers all remaining iterations. -PHASE1_MODEL_SCHEDULE=claude-sonnet-4.6:3,claude-haiku-4.5 -PHASE2_MODEL_SCHEDULE=claude-haiku-4.5 -PHASE3_MODEL_SCHEDULE=claude-opus-4.6:1,claude-sonnet-4.6 -PHASE4_MODEL_SCHEDULE=claude-opus-4.6:1,claude-sonnet-4.6 -PHASE6_MODEL=claude-sonnet-4.6 - -# Git commit author identity -GIT_USER_NAME=github-actions[bot] -GIT_USER_EMAIL=41898282+github-actions[bot]@users.noreply.github.com -``` +| State | Measured? | Meaning | +|-------|-----------|---------| +| **KEEP** | yes | Significant improvement on the point's primary signal → Phase 8 drafts a PR | +| **INVESTIGATE** | yes | Measured but ambiguous → report recommends manual re-measurement, not a PR | +| **REVERT** | yes | A regression, or no detectable effect → not submitted (the conservative default) | +| **FAILED** | no | The change was incorrect (failed the gate) → branch never pushed, never counted as a regression | +| **INCOMPLETE** | no | Built and gated, but the stats couldn't be trusted → excluded, flagged for re-run | + +**→ Start here:** [Quickstart](#quickstart) · [Full per-phase docs](docs/index.md) · [Executive summary](executive-summary.md) · [How it works](#how-it-works) + +--- + +## Why CodeWeave exists + +LLM coding agents are good at *proposing* performance changes and bad at *proving* them. Ask one to speed up a hot path and you get a confident diff and a confident claim — "~10% faster." Verifying that claim is the actual work: is the change even correct on the edge cases? Is the speedup real, or is it thermal drift and cache warmth? Would it survive being measured a second time? Multiply that by dozens of candidates and the verification cost swamps the generation cost. + +So agent-proposed optimizations mostly don't ship. The bottleneck was never generating ideas — it was **trusting them.** + +CodeWeave is built on one conviction: **the agent that writes a change must never be the thing that certifies it.** Generators propose; a deterministic, statistically rigorous pipeline disposes. The agent edits source and authors specs; the pipeline builds, fuzzes, measures, and rules. `ab_compare.py` is the single source of truth for every number. "Done" is always a file written by an independent check, never a claim. + +The long-term goal is **greener software**. Energy and carbon are measured directly (via CodeCarbon) and treated as the ultimate objective; per-iteration latency is the lever we can resolve precisely enough to act on. The result is an autonomous loop that can walk into an unfamiliar codebase, understand it, and improve it under measurement — leaving behind an audit trail a human reviewer can actually check. + +**Design philosophy, in five lines:** + +- Generators propose; independent validators dispose. +- The pipeline owns truth and every commit. +- Correctness before performance — always. +- Measure the right thing, and only trust what you can resolve. +- Energy is the goal; latency is the lever. + +--- + +## Who it's for and what you'd use it for + +CodeWeave targets people responsible for large, performance-sensitive systems where "make it faster" is a real, recurring job: + +- **"Find and prove wins in a hot library."** Run the full pipeline against a compute-heavy codebase (the reference target is PyTorch on CPU) and get back a ranked set of PR-ready branches, each with a statistical verdict — plus honest REVERT/INVESTIGATE records for the ideas that didn't pan out. +- **"Understand a codebase I inherited."** Phases 1–4 alone produce an architecture book (+ PDF), per-area Architecture Decision Records committed alongside the code, and a runnable measurement harness — grounded in the actual source, not generic assumptions. +- **"Vet an agent's optimization before I trust it."** The correctness gate + drift-controlled A/B measurement is the review you'd otherwise do by hand for every candidate, run automatically and recorded. +- **"Stand up repeatable performance measurement."** Phases 3–6 give you a fixed-time hot-loop harness, a statistical baseline with a computed noise floor, and a hotspot profile — reusable infrastructure independent of the optimization stage. -| Setting | Description | -|---|---| -| `EXTERNAL_REPO_NAME` | Name of the external repository (used for organizing output) | -| `EXTERNAL_REPO_URL` | HTTPS URL of the external Git repository | -| `EXTERNAL_REPO_BRANCH` | Branch to clone from the external repository | -| `EXTERNAL_REPO_WORK_BRANCH` | Branch name to create and work on (keeps original branch clean) | -| `PHASE1_MAX_ITERATIONS` | Maximum number of Phase 1 (book generation) iterations (1-99) | -| `PHASE2_MAX_ITERATIONS` | Maximum number of Phase 2 (ADR generation) iterations (1-99) | -| `PHASE3_MAX_ITERATIONS` | Maximum number of Phase 3 (performance measurement) iterations (1-99) | -| `PHASE4_MAX_ITERATIONS` | Maximum number of Phase 4 (integration test code generation) iterations (1-99) | -| `PHASE6_BASELINE_RUNS` | Number of measurement runs to collect in Phase 6 (default 5) | -| `PHASE1_MODEL_SCHEDULE` | Model schedule for Phase 1. Format: `model:count,...,model_for_remaining` (e.g. `claude-sonnet-4.6:3,claude-haiku-4.5`) | -| `PHASE2_MODEL_SCHEDULE` | Model schedule for Phase 2. A single model name is valid (e.g. `claude-haiku-4.5`) | -| `PHASE3_MODEL_SCHEDULE` | Model schedule for Phase 3. Format same as above (e.g. `claude-opus-4.6:1,claude-sonnet-4.6`) | -| `PHASE4_MODEL_SCHEDULE` | Model schedule for Phase 4. Format same as above (e.g. `claude-opus-4.6:1,claude-sonnet-4.6`) | -| `PHASE6_MODEL` | Model for Phase 6's single execution pass (no schedule needed, e.g. `claude-sonnet-4.6`) | -| `PHASE5_MODEL` / `PHASE5_MAX_ITERATIONS` | Phase 5 source-build model and repair-iteration cap (default 3) | -| `PHASE7_MAX_OPTIMIZATIONS` / `PHASE7_MAX_ITERATIONS` | Number of optimization points (default 5) and per-optimization repair-loop cap (default 3) | -| `PHASE7_MODEL_SCHEDULE` / `PHASE7_SELECTION_MODEL` | Model schedule for the optimization passes; model for the hotspot-selection pass | -| `PHASE7_FUZZ_REQUIRED` | 7f differential fuzz gate, blocking by default (`1`); set `0` only for an op that cannot be fuzzed | -| `PHASE7_BASELINE_RUNS` / `PHASE7_MICROBENCH_MIN_SECONDS` | A/B measurement runs per side (default 5) and microbenchmark min run-time per side (default 10) | -| `PHASE8_MODEL` | Model for the Phase 8 aggregate-report pass | -| `GIT_USER_NAME` | Git commit author name | -| `GIT_USER_EMAIL` | Git commit author email | - -### Trigger - -Manually triggered via `workflow_dispatch`. Optional inputs: `start_from_phase` (choices `1`–`6`) resumes from a specific phase (defaults to `1`; later phases still check that prerequisite artifacts exist); `dry_run` (boolean) skips Copilot invocations and overrides prerequisite checks to smoke-test workflow structure. - -### Permissions - -- `contents: write` — required to commit and push changes back to the branch. +--- + +## Quickstart + +CodeWeave runs as **GitHub Actions workflows**, not a local CLI. Phases 5–8 build and measure a native target, so they require a persistent machine. + +### Prerequisites + +- A **self-hosted GitHub Actions runner** (`[self-hosted, Linux, X64]`) that persists the built `src/` tree, the editable `integration-test/.venv`, and a warm ccache between phases. `[VERIFY SUPPORTED PLATFORMS — reference target is Linux/x64 + PyTorch CPU]` +- **GitHub Copilot CLI** access (installed automatically by the workflow via `npm i -g @github/copilot`). +- Two fine-grained PATs stored as repository secrets: + - `COPILOT_TOKEN` — authenticates the Copilot CLI. Needs **Copilot user requests: Read** (account permission; no repo permissions). + - `PUSH_TOKEN` — **Contents: Read and write** on the *target* repo only (Phase 2 pushes ADRs; Phase 7 pushes optimization branches). +- Toolchain for the target build. For the default PyTorch/CPU target that means **Python 3.12** specifically (see [`constraints/project.md`](constraints/project.md) for why — newer interpreters lack prebuilt wheels for the harness stack). ### Steps -1. **Checkout** — Checks out this repository with full history and credential persistence. -2. **Setup Node.js 22** — Required to install the Copilot CLI. -3. **Install Copilot CLI** — Installs `@github/copilot` globally via npm. -4. **Load configuration** — Sources `.github/codeweave.config`, extracts the first repo's configuration, and exports settings as environment variables. -5. **Configure git identity** — Sets the git commit author from `codeweave.config` variables. -6. **Clone external repository** — Clones the specified branch from the external repo (shallow, single branch) into `src`, then creates and checks out the work branch. The `src/` directory is excluded from git tracking via `.git/info/exclude`. -7. **Run the CodeWeave pipeline (Phases 1–6)** — The `codeweave.yml` orchestrator calls a per-phase reusable workflow for each phase: Phases 1–4 are generate+validate loops with file-based early exit, Phase 5 builds the target from source, and Phase 6 collects the deterministic baseline. Each phase's gate, inputs, and outputs are documented in [`docs/`](docs/index.md) (see the phase table above). It finally writes `proof/final-status.md` summarising all phases. -8. **Push** — Pushes all outer-repo commits back to the triggering branch. - -> **Phases 7–8 are separate workflows, dispatched automatically.** When Phase 6 produces a baseline, `codeweave.yml`'s `trigger-phase-7` job dispatches the first optimization cycle (Phase 7, `phase-7-optimize.yml`); each cycle self-chains to the next, and the last chains into the aggregate report (Phase 8, `phase-8-report.yml`) — one optimization per dispatch. No manual step is needed; to start them by hand instead, run `gh workflow run phase-7-optimize.yml -f optimization_index=1`. See the Overview above for what each does. - -### Proof Artifacts - -Each iteration produces output files organized in `proof/`: - -| File | Contents | -|---|---| -| `1-book-generation-N.md` | Generate pass output log for Phase 1 iteration N | -| `1-book-generation-session-N.md` | Generate pass session transcript for Phase 1 iteration N | -| `1-book-validation-N.md` | Validate pass output log for Phase 1 iteration N | -| `1-book-validation-session-N.md` | Validate pass session transcript for Phase 1 iteration N | -| `1-book-validation-report-N.md` | Copy of `book/BOOK-VALIDATION.md` after Phase 1 iteration N | -| `2-adrs-generation-N.md` | Generate pass output log for Phase 2 iteration N | -| `2-adrs-generation-session-N.md` | Generate pass session transcript for Phase 2 iteration N | -| `2-adrs-validation-N.md` | Validate pass output log for Phase 2 iteration N | -| `2-adrs-validation-session-N.md` | Validate pass session transcript for Phase 2 iteration N | -| `2-adrs-validation-report-N.md` | Copy of `src/ADR-VALIDATION.md` after Phase 2 iteration N | -| `3-harness-generation-N.md` | Generate pass output log for Phase 3 iteration N | -| `3-harness-generation-session-N.md` | Generate pass session transcript for Phase 3 iteration N | -| `3-harness-validation-N.md` | Validate pass output log for Phase 3 iteration N | -| `3-harness-validation-session-N.md` | Validate pass session transcript for Phase 3 iteration N | -| `3-harness-validation-report-N.md` | Copy of `integration-test/HARNESS-VALIDATION.md` after Phase 3 iteration N | -| `4-tests-generation-N.md` | Generate pass output log for Phase 4 iteration N | -| `4-tests-generation-session-N.md` | Generate pass session transcript for Phase 4 iteration N | -| `4-tests-validation-N.md` | Validate pass output log for Phase 4 iteration N | -| `4-tests-validation-session-N.md` | Validate pass session transcript for Phase 4 iteration N | -| `4-tests-validation-report-N.md` | Copy of `integration-test/TESTS-VALIDATION.md` after Phase 4 iteration N | -| `5-build-source-N.md` / `5-build-source-session-N.md` | Phase 5 build authoring pass log / transcript (attempt N) | -| `5-build-output-N.log` | Phase 5 build execution output (attempt N) | -| `6-baseline-run-N.log` | Phase 6 baseline measurement run N output | -| `6-trace.log` | Phase 6 tracing-pass output | -| `6-repair-N.md` / `6-repair-session-N.md` | Phase 6 repair pass N log / transcript (if any) | -| `final-status.md` | Summary of all phases: PDF generation, ADR coverage, strategy status, test code status, and baseline execution status | - -Git commit history provides the natural diff trail between iterations. - -### Required Secrets - -| Secret | Purpose | -|---|---| -| `COPILOT_TOKEN` | Fine-grained PAT used to authenticate the Copilot CLI (`GH_TOKEN` in the step environment). Needs the **Copilot user requests: Read** user permission (Account permissions — no repository permissions required). | -| `PUSH_TOKEN` | Fine-grained PAT used to push to the target repository — Phase 2 pushes ADR changes to the work branch, and Phase 7 pushes each gate-passing optimization branch. Needs the **Contents: Read and write** repository permission on the target repository only. | - -## Work Definition and Constraints - -The workflow improves code based on tasks and constraints you define: - -- **`work/1-generate-book.md`** — Phase 1 generation prompt. Copilot writes/expands chapters each iteration without self-certifying completion. -- **`work/1-validate-book.md`** — Phase 1 validation prompt. Run after each generation pass; enforces all quality gates and exclusively writes `book/manuscript-complete.md` on PASS. -- **`work/2-generate-adrs.md`** — Phase 2 generation prompt. Directs Copilot to generate per-folder `ADR.md` files in `./src` using the book markdown sources as reference. -- **`work/2-validate-adrs.md`** — Phase 2 validation prompt. Run after each ADR generation pass; validates coverage against the scope map and exclusively writes `src/adrs-complete.md` on PASS. -- **`work/3-generate-harness.md`** — Phase 3 generation prompt. Directs Copilot to produce a three-tier performance measurement document set: `integration-test/AGENTS.md` (permanent harness operating rules), `integration-test/SOURCE-UNDER-INVESTIGATION.md` (target-specific profile derived from the book, ADRs, and constraints), and `integration-test/WORK.md` (execution agent checklist). -- **`work/3-validate-harness.md`** — Phase 3 validation prompt. Run after each harness design pass; validates all three harness documents and exclusively writes `integration-test/harness-complete.md` on PASS. -- **`work/4-generate-tests.md`** — Phase 4 generation prompt. Directs Copilot to read `integration-test/AGENTS.md`, `SOURCE-UNDER-INVESTIGATION.md`, and `WORK.md` for context, check `./src/` for existing build documentation, and generate all test files, `setup.sh`, `run.sh`, and `_tools/` helpers. Instructs the generator to derive all install commands and environment variables from `SOURCE-UNDER-INVESTIGATION.md` rather than from general knowledge. -- **`work/4-validate-tests.md`** — Phase 4 validation prompt. Run after each code generation pass; validates all generated files against `SOURCE-UNDER-INVESTIGATION.md`, reads `integration-test/smoke-test-report.md` as part of Check 1, and exclusively writes `integration-test/tests-complete.md` on PASS. -- **`work/6-repair-tests.md`** — Phase 6 repair agent prompt. Invoked only when a probe run of `run.sh` fails; fixes the specific runtime error in `tests/` or `_tools/`. Baseline collection itself (build via `setup.sh`, `PHASE6_BASELINE_RUNS` runs of `run.sh`, `_tools/ab_compare.py --mode baseline`) is deterministic pipeline logic, not a prompt. -- **`work/5-build-source.md`** — Phase 5 prompt. Copilot *authors* `integration-test/build-source.sh`; the pipeline *executes* it to build the target from source (the agent does not run the build itself). -- **`work/7-select-hotspots.md`** — Phase 7 hotspot-selection prompt (index 1 only). Reads `profiler-summary.md` + the ADR index and writes `integration-test/optimization-plan.md` (ranked target ops + per-point measurement path). -- **`work/7-generate-optimization.md`** — Phase 7 optimization prompt. Implements one source change in `./src` and authors the gate-7f differential-fuzz spec; `work/fuzz-examples/opt{1,2}_fuzz.py` are worked templates it references. -- **`work/8-aggregate-report.md`** — Phase 8 prompt. Synthesises every Phase 7 verdict into a ranked report + per-optimization PR drafts — an authoring task: it does not build, run, or measure (the verdicts are already computed by `ab_compare.py`). -- **`constraints/project.md`** — Repository-specific constraints and requirements that must be respected (e.g., framework versions, architecture decisions, tech stack limitations). The default includes a sample constraint; **when forking, replace it with your actual constraints**. -- **`constraints/harness.md`** — *(optional)* Target execution constraints for Phase 3: hardware requirements, scope limitations, time budgets, and isolation rules. Phase 3 incorporates this content into `integration-test/SOURCE-UNDER-INVESTIGATION.md`. -- **`constraints/harness-context.md`** — *(optional)* Domain context for Phase 3 scenario and observability design. Provides target-specific knowledge about what to instrument, what representative scenarios look like, and known performance-sensitive paths. - - -## Key Files to Know - -| File | Purpose | -|---|---| -| `.github/workflows/codeweave.yml` | Orchestrator (glue): dispatch + `needs`/`if` calling the per-phase reusable workflows; `finalize` inline | -| `.github/workflows/phase-*.yml` | Per-phase reusable workflows (1–6) + `phase-7-optimize.yml` / `phase-8-report.yml` | -| `.github/actions/codeweave-setup/action.yml` | Composite action: shared bootstrap (Node + Copilot CLI + `codeweave.config` + git identity) | -| `.github/codeweave.config` | Runtime configuration (external repo, branch, iterations, git identity) | -| `.github/scripts/generate-indexes.js` | Generates `book/BOOK-INDEX.md` (Phase 1) and `src/ADR-INDEX.md` (Phase 2) | -| `work/1-generate-book.md` | Phase 1 generation prompt (no self-certification) | -| `work/1-validate-book.md` | Phase 1 validation prompt (exclusively owns `manuscript-complete.md`) | -| `work/2-generate-adrs.md` | Phase 2 generation prompt (no self-certification) | -| `work/2-validate-adrs.md` | Phase 2 validation prompt (exclusively owns `adrs-complete.md`) | -| `work/3-generate-harness.md` | Phase 3 generation prompt (no self-certification) | -| `work/3-validate-harness.md` | Phase 3 validation prompt (exclusively owns `harness-complete.md`) | -| `work/4-generate-tests.md` | Phase 4 generation prompt (no self-certification) | -| `work/4-validate-tests.md` | Phase 4 validation prompt (exclusively owns `tests-complete.md`) | -| `work/6-repair-tests.md` | Phase 6 repair agent prompt — fixes runtime errors in `tests/`/`_tools/` (baseline collection is deterministic pipeline logic) | -| `work/5-build-source.md` | Phase 5 prompt — authors `build-source.sh` (pipeline executes it) | -| `work/7-select-hotspots.md` / `work/7-generate-optimization.md` | Phase 7 hotspot selection (→ `optimization-plan.md`) and per-optimization implementation + 7f fuzz spec | -| `work/8-aggregate-report.md` | Phase 8 prompt — ranked aggregate report + PR drafts | -| `integration-test/harness-manifest.json` | Machine-readable toolchain manifest (venv layout, smoke-check commands, profiler enable-env, hotspot-report path, incremental-build recipe, and op-suite/import-op gate commands) the deterministic pipeline reads instead of hardcoding Python/pytest/venv. Emitted by Phase 4 from `SOURCE-UNDER-INVESTIGATION.md §08`; a target whose toolchain matches the built-in defaults emits a no-op manifest. | -| `integration-test/_tools/ab_compare.py` | A/B + baseline statistics (v2.2: Welch's t-test + MDE floor, directional, measurement-path-aware verdict; `--mode family` Holm-Bonferroni) | -| `integration-test/_tools/op_microbench.py` / `diff_fuzz.py` | Per-op microbenchmark (framework-native benchmark timer) and the 7f differential-fuzz gate driver | -| `constraints/project.md` | Repository-specific constraints and requirements (customize for your fork) | -| `constraints/harness.md` | *(optional)* Target execution constraints (hardware, scope, time budgets) consumed by Phase 3 | -| `constraints/harness-context.md` | *(optional)* Domain context for Phase 3 scenario and observability design | -| `proof/` | Output artifacts directory (auto-created): run logs, session transcripts, final status | -| `.git/info/exclude` | Excludes `src/` from git tracking (auto-configured by workflow) | +1. **Fork this repository.** +2. **Configure the run** in [`.github/codeweave.config`](.github/codeweave.config) — target repo URL/branch, per-phase iteration caps, and per-phase model schedules. Nothing is hardcoded in the workflow logic. +3. **Replace the constraints.** [`constraints/project.md`](constraints/project.md) ships with a sample constraint — swap in your target's real constraints (toolchain versions, build env vars, scope limits). `constraints/harness.md` and `constraints/harness-context.md` are optional target-specific inputs for Phase 3. +4. **Add the two secrets** (`COPILOT_TOKEN`, `PUSH_TOKEN`). +5. **Smoke-test the structure first:** + ```bash + gh workflow run codeweave.yml -f dry_run=true + ``` + `dry_run` skips Copilot invocations and prerequisite checks, so you can confirm the eight-phase wiring and the Phase 7 auto-chain before spending model time. +6. **Run it for real:** + ```bash + gh workflow run codeweave.yml + ``` + +### The "aha" moment + +Watch a single dispatch cascade: Phases 1–6 document and baseline the target, then `codeweave.yml` automatically dispatches Phase 7, which optimizes one hotspot, gates it, measures a paired A/B, records a verdict — and dispatches the *next* cycle itself. The last cycle chains into Phase 8, which writes a ranked report and drafts a PR per PR-worthy win. You dispatched once; the pipeline ran an entire measurement campaign and handed you reviewable branches. + +Resume from any phase with `-f start_from_phase=N` (1–6). Full per-phase gates, inputs, and outputs: **[`docs/index.md`](docs/index.md)**. + +--- + +## Core capabilities + +### Understand an unfamiliar codebase (Phases 1–4) + +**What it enables:** an architecture book (+ PDF via Pandoc → Typst), per-area ADRs committed into the target's source tree, and a runnable measurement harness — all grounded in the real source. + +**Why it matters:** every later measurement decision traces back to a documented claim, so the harness reflects the actual system instead of the model's priors. + +**How it stays honest:** each phase is a **generate → validate loop**. A generate pass edits artifacts; a *separate* validate pass checks them against an explicit checklist and is the **only** thing allowed to write the phase's completion marker. The validator writes a *Required Actions* list, the next generator burns that list down first, and durable state files record the high-water mark — so the manuscript expands and deepens across iterations instead of churning. + +**Limitation:** requires enough model budget for multiple passes per phase; iteration caps are set per phase in `codeweave.config`. + +### Measure with a trustworthy substrate (Phases 5–6) + +**What it enables:** the target built *from source* (ccache-backed), plus a statistical baseline (`baseline.json`) and a hotspot profile. + +**Why it matters:** A/B comparisons compare two builds of the *same* source tree, and the baseline computes a **noise floor** — a coefficient of variation and a Minimum Detectable Effect — that every later verdict is held to. + +**Key design decision:** the workload is a **fixed-time hot loop**, so wall-clock time carries no signal (a faster build just completes more iterations). **Every verdict uses per-iteration metrics** (`median_iter_ms`, iterations, joules/iter) — never wall clock. + +### Optimize under a correctness-first gate (Phase 7) + +**What it enables:** one optimization per dispatch — generate the change, incrementally rebuild (with rebuild verification so a zero-compile edit can't slip through), run the **7a–7f correctness gate**, and only *then* measure a paired A/B. + +**Why it matters:** the change must be provably *correct* — including a blocking differential fuzz against a base-build golden — before any timing run is spent. The A-side base is re-measured **every cycle**, back-to-back with the variant, to cancel machine drift over a long run. + +**Two measurement paths:** ops whose end-to-end effect falls below the noise floor by construction are judged on a per-op microbenchmark (`torch.utils.benchmark`); others on end-to-end latency. The verdict logic is path-aware and directional (regressions tested first). + +**Limitation:** energy is measured and reported but **does not gate** — CodeCarbon's resolution is too coarse to judge a single optimization, so a latency win that regresses energy still KEEPs and is merely flagged. + +### Report with campaign-wide rigor (Phase 8) + +**What it enables:** a ranked report plus a drafted PR per PR-worthy optimization. + +**Why it matters:** running many cycles at `α = 0.05` inflates the odds of at least one false KEEP, so Phase 8 applies a **Holm–Bonferroni correction** across all cycles and demotes KEEPs that don't survive. + +**Limitation:** Phase 8 *drafts* PRs; opening them is a deliberate human step. `INVESTIGATE` is a classification, not an action — the system surfaces ambiguous candidates and the reason (`decision_signal`), but never re-measures or investigates on its own. + +--- + +## How it works + +One `workflow_dispatch` runs eight gated phases. Phases 1–6 run inside `codeweave.yml`; Phases 7 and 8 are dedicated **auto-chaining** workflows — Phase 7 handles one optimization per dispatch and triggers the next, and the last chains into Phase 8. + +```mermaid +flowchart TD + start(["workflow_dispatch"]) --> clone["Clone target into src/ · create work branch"] + clone --> P1 + subgraph gen["Phases 1-4 · generate → validate loops"] + P1["1 · Book"] --> P2["2 · ADRs"] --> P3["3 · Harness design"] --> P4["4 · Integration tests"] + end + P4 --> P5["5 · Build from source (ccache)"] + subgraph meas["Phases 5-8 · measure & optimize"] + P5 --> P6["6 · Statistical baseline → baseline.json"] + P6 --> P7["7 · Optimize one point
gate 7a–7f → paired A/B → verdict"] + P7 --> more{"more points?"} + more -- "yes · auto-chain N+1" --> P7 + more -- no --> P8["8 · Aggregate report + PR drafts"] + end + P8 --> done(["ranked report + PR-worthy branches"]) +``` + +The generate → validate loop is the structural backbone of Phases 1–4: + +```mermaid +flowchart TD + enter(["enter phase"]) --> del["Delete completion marker"] + del --> genp["Generate pass · Copilot edits artifacts"] + genp --> smoke{"Phase 4 only:
smoke test?"} + smoke -- fail --> report["Write findings · skip validator"] --> iter + smoke -- "pass / N/A" --> val["Validate pass · writes marker on PASS"] + val --> marker{"marker present?"} + marker -- yes --> exitok(["early exit"]) + marker -- no --> iter{"iterations left?"} + iter -- yes --> del + iter -- no --> stopmax(["stop — max iterations"]) +``` + +**Operating rules that make it dependable:** + +- **The pipeline owns every commit.** Copilot runs non-interactively (`--no-ask-user`) and is denied git in every phase except the Phase 7 optimization agent (which works on its own branch and reviews its diff). +- **Markers are deleted before each generate pass**, so a stale "done" file can never short-circuit the next cycle. +- **Toolchain lives in a manifest, not the code.** Phase 4 emits `integration-test/harness-manifest.json` (venv layout, smoke checks, profiler enable-env, hotspot-report path, gate commands) so the deterministic pipeline reads the target's toolchain instead of hardcoding Python/pytest/venv — the seam intended to carry the engine beyond the PyTorch reference target. + +Full detail — every gate, input, and output per phase — is in **[`docs/`](docs/index.md)** and the **[executive summary](executive-summary.md)**. + +--- + +## How it's different + +CodeWeave isn't a coding assistant and isn't a benchmark runner — it's the pipeline between them that makes an agent's performance claims trustworthy. + +| | Ask an agent directly | Hand-roll a benchmark + review | **CodeWeave** | +|---|---|---|---| +| Correctness check before measuring | You do it, per change | You do it | Automated 6-stage gate incl. differential fuzz | +| Speedup vs. noise | Agent's word | Manual stats, if any | Welch's t-test **and** measured MDE floor | +| Machine drift over a long run | Ignored | Manual re-runs | Contemporaneous A/B, base re-measured each cycle | +| False positives across many changes | Unaddressed | Rarely corrected | Holm–Bonferroni across the campaign | +| Output | A diff + a claim | A number you produced | Ranked, gated PR branches + full `proof/` trail | +| Self-certification | The agent says "done" | — | Only an independent validator writes "done" | + +This is a positioning of *discipline*, not a knock on coding agents — CodeWeave uses one (GitHub Copilot CLI) as its generator. The difference is what happens to a change after it's written. + +--- + +## Project status + +CodeWeave is **experimental**. It is a working, end-to-end pipeline that has been built and iterated against a real target (PyTorch, CPU), but it has not been hardened for arbitrary repositories or published with reproducible headline results. + +- **Working today:** the full eight-phase run against the reference target — generate→validate documentation loops, source build, statistical baseline, correctness-gated optimization cycles with paired A/B verdicts, and the aggregate report with PR drafts. +- **Experimental:** portability beyond the PyTorch/CPU reference target. The manifest seam is designed for it, but other toolchains are unverified. `[VERIFY portability on a second target]` +- **Planned / manual by design:** opening PRs (Phase 8 drafts them); acting on `INVESTIGATE` results (surfaced, never auto-investigated). +- **Known limitations:** + - Requires a **self-hosted, persistent runner** — Phases 6–8 reuse the build and venv in place; there is no ephemeral-runner path. + - Requires GitHub Copilot CLI access and two fine-grained PATs. + - Energy/carbon is **measured and reported but not gating** (resolution too coarse per change). + - Model and CI time cost scales with iteration caps and optimization count. + - No published benchmark results yet. `[ADD BENCHMARK — headline results from a real run]` + +> **Maturity warning:** treat CodeWeave as a research-grade automation harness. Review every drafted PR and read the `proof/` trail before shipping anything it produces. + +--- + +## Documentation, community & trust + +- **Documentation:** per-phase reference in [`docs/index.md`](docs/index.md); the design rationale in [`executive-summary.md`](executive-summary.md). +- **Examples:** prompt and constraint files under [`work/`](work) and [`constraints/`](constraints); a sample run's artifacts appear in `proof/` (auto-created). `[ADD LINK TO A PUBLISHED EXAMPLE RUN]` +- **Roadmap:** `[ADD ROADMAP LINK]` — near-term focus is verifying portability to a second target via the manifest seam. +- **Contributing:** `[ADD CONTRIBUTING.md]` — issues and PRs welcome; please include the relevant `proof/` artifacts when reporting pipeline behavior. +- **Support:** `[ADD SUPPORT CHANNEL — GitHub Issues / Discussions]` +- **Security:** the pipeline handles two PATs and pushes branches to a target repo; scope tokens minimally as described in [Prerequisites](#quickstart). `[ADD SECURITY.md / disclosure policy]` ## License -Copyright (C) 2026 Hightech ICT B.V. -This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE file for details. +Copyright © 2026 Hightech ICT B.V. + +Licensed under the **GNU General Public License v3.0 or later**. See [`LICENSE`](LICENSE). From e9b4cc38b0a564dc4d52a9532988e0cd96251e7d Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 11:34:17 +0200 Subject: [PATCH 02/10] docs: add CONTRIBUTING, SECURITY, and ROADMAP Add the community and trust files the README references, grounded in the actual pipeline rather than boilerplate: - CONTRIBUTING.md: how to exercise a workflows-only project (dry-run first), what a good proof/-backed report contains, and the core invariant contributors must preserve (generators propose, the independent pipeline disposes; no self-certification). - SECURITY.md: private vulnerability reporting, plus concrete credential handling for COPILOT_TOKEN / PUSH_TOKEN and the self-hosted runner that holds them. - ROADMAP.md: now/next/later framed around publishing a real run, portability via the harness-manifest seam, and making energy a gating signal. Placeholders remain only for facts that require a human decision (security contact, response SLA, code of conduct). Co-Authored-By: Claude Opus 4.8 --- CONTRIBUTING.md | 71 +++++++++++++++++++++++++++++++++++++++++++++++++ ROADMAP.md | 53 ++++++++++++++++++++++++++++++++++++ SECURITY.md | 70 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 194 insertions(+) create mode 100644 CONTRIBUTING.md create mode 100644 ROADMAP.md create mode 100644 SECURITY.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..91cffa7 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,71 @@ +# Contributing to CodeWeave + +Thanks for your interest in CodeWeave. It is an experimental, research-grade +automation pipeline, so contributions that make it more **portable**, more +**auditable**, or more **statistically rigorous** are especially welcome. + +Please read this guide before opening an issue or a pull request. + +## Ways to contribute + +- **Report pipeline behavior** — a phase that stalls, a gate that misfires, a + verdict that looks wrong. Include the relevant `proof/` artifacts (see below). +- **Improve portability** — CodeWeave was built against a PyTorch/CPU reference + target. Making it run against a second target (via the + `integration-test/harness-manifest.json` seam) is the highest-value area. +- **Sharpen the measurement science** — the verdict logic lives in + `integration-test/_tools/ab_compare.py`. Changes here must come with a written + rationale and, ideally, a worked example. +- **Docs** — clarifications to `docs/`, the `README.md`, or `executive-summary.md`. + +## Before you start + +- CodeWeave runs as **GitHub Actions workflows on a self-hosted runner**, not as a + local CLI. There is no `npm install && npm test` loop; changes are exercised by + dispatching the workflows. See the [Quickstart](README.md#quickstart). +- Smoke-test structural changes with a dry run before a real run: + ```bash + gh workflow run codeweave.yml -f dry_run=true + ``` + `dry_run` skips Copilot invocations and prerequisite checks, so you can validate + workflow wiring cheaply. + +## Reporting an issue + +A good report includes: + +1. **Which phase** (1–8) and the workflow that ran it. +2. **What you expected vs. what happened.** +3. **The relevant `proof/` artifacts** — generate/validate logs, session + transcripts, gate diagnostics, and measurement records are all written there. + These are the primary evidence for any pipeline behavior; attach or paste the + relevant files rather than describing them. +4. **Your configuration** — the relevant parts of `.github/codeweave.config` + (redact anything sensitive) and your `constraints/` files. + +Do **not** include secrets (`COPILOT_TOKEN`, `PUSH_TOKEN`) or their values. + +## Pull requests + +- **Branch naming:** use a descriptive prefix, e.g. `docs/…`, `pipeline/…`, + `config/…`, `work/…`, matching the existing history. +- **Commit messages:** follow the repository style — a `scope: imperative summary` + subject line, then a body explaining *why*. Keep one logical change per commit. +- **Scope:** keep PRs focused. A change to prompt files (`work/`) is separate from + a change to workflow logic (`.github/workflows/`). +- **The core invariant:** CodeWeave's whole premise is that *generators propose and + an independent, deterministic pipeline disposes*. Changes must preserve that + separation — an agent must never be able to self-certify a phase as complete, and + the pipeline must remain the single owner of commits and verdicts. +- **Measurement changes** to `ab_compare.py` or the gate logic should explain how + they affect the trustworthiness of a verdict (significance, noise floor, drift + control, family-wise correction). + +## Code of conduct + +Be respectful and constructive. `[ADD CODE_OF_CONDUCT.md IF ADOPTING ONE]` + +## License + +By contributing, you agree that your contributions are licensed under the project's +**GNU General Public License v3.0 or later** (see [`LICENSE`](LICENSE)). diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..1698d9f --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,53 @@ +# CodeWeave Roadmap + +CodeWeave is **experimental**. It runs end to end against a PyTorch/CPU reference +target, but it is not yet hardened for arbitrary repositories and has no published +headline results. This roadmap describes the direction, not a dated commitment; +priorities and ordering may change. + +> Have an opinion on ordering, or want to pick something up? Open an issue or see +> [`CONTRIBUTING.md`](CONTRIBUTING.md). + +## Now — proving the core + +The immediate focus is turning "it works against one target" into "it demonstrably +works, and you can see the evidence." + +- **Publish a real run.** Link an example `proof/` trail and a Phase 8 report from a + full eight-phase run, including at least one `KEEP` verdict with its statistics. + _(Fills the `[ADD BENCHMARK]` / `[ADD LINK TO A PUBLISHED EXAMPLE RUN]` gaps in the + README.)_ +- **A demo asset.** A terminal recording or annotated report showing a cycle go + correctness-gate → paired A/B → verdict. + +## Next — portability beyond the reference target + +CodeWeave was built against PyTorch on CPU. The seam for going further already +exists — `integration-test/harness-manifest.json` lets the deterministic pipeline +read a target's toolchain instead of assuming Python/pytest/venv. + +- **Verify a second target end to end** through the manifest seam, and document what + a new target requires. _(Fills the `[VERIFY portability on a second target]` gap.)_ +- **Reduce runner assumptions.** Explore paths that don't require a single persistent + self-hosted runner to carry state across Phases 5–8. + +## Later — sharper science and less manual glue + +- **Act on `INVESTIGATE`.** Today it is a classification, not an action; the pipeline + surfaces ambiguous candidates and a `decision_signal` but never re-measures. A + guarded auto-re-measurement path is a candidate. +- **Optional PR opening.** Phase 8 currently *drafts* PRs; opening them is a + deliberate human step. An opt-in flag to open the drafted PRs directly. +- **Energy as more than a report.** Energy/carbon is measured but does not gate, + because CodeCarbon's per-change resolution is too coarse. Investigate measurement + approaches precise enough to make energy a first-class gating signal, in line with + the project's "energy is the goal" philosophy. +- **Broader statistical options.** Additional verdict models and configurable + multiple-comparison corrections beyond Holm–Bonferroni. + +## Non-goals (for now) + +- Becoming a general-purpose local coding assistant — CodeWeave is a measurement and + verification pipeline that *uses* a coding agent, not a replacement for one. +- Auto-merging changes. A human reviews every drafted PR and the `proof/` trail + before anything ships. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..02cfe12 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,70 @@ +# Security Policy + +CodeWeave is an automation pipeline that authenticates as a GitHub Copilot user, +holds two fine-grained Personal Access Tokens, and pushes branches to a target +repository. Its security posture matters. This document explains how to report a +vulnerability and how the pipeline handles sensitive credentials. + +## Reporting a vulnerability + +**Please do not open a public issue for security vulnerabilities.** + +Preferred channel: use GitHub's **private vulnerability reporting** on this +repository (the *Security* tab → *Report a vulnerability*). This keeps the report +confidential until a fix is available. + +Alternatively, email `[ADD SECURITY CONTACT EMAIL]`. + +When reporting, please include: + +- A description of the issue and its impact. +- Steps to reproduce, or a proof of concept. +- The affected component (a workflow, a `work/` prompt, `ab_compare.py`, etc.). +- Any suggested remediation. + +**Please redact secrets** from anything you attach — never include the values of +`COPILOT_TOKEN`, `PUSH_TOKEN`, or any other credential. + +### What to expect + +- **Acknowledgement:** `[CONFIRM RESPONSE-TIME SLA]` (suggested: within 5 business days). +- We will confirm the issue, assess severity, and keep you updated on remediation. +- Please give us a reasonable window to release a fix before public disclosure. + +## Supported versions + +CodeWeave is **experimental** and pre-release. Security fixes are applied to the +default branch (`main`) only. `[CONFIRM SUPPORTED-VERSION POLICY IF RELEASES BEGIN]` + +## Credential handling (how the pipeline treats secrets) + +CodeWeave requires two fine-grained PATs, both stored as GitHub Actions repository +secrets and scoped as narrowly as possible: + +| Secret | Scope it needs | Used by | +|--------|----------------|---------| +| `COPILOT_TOKEN` | **Copilot user requests: Read** (account permission only — no repository permissions) | Every Copilot phase, as `GH_TOKEN` | +| `PUSH_TOKEN` | **Contents: Read and write** on the *target* repository only | Phase 2 (push ADRs) and Phase 7 (push optimization branches) | + +Guidance: + +- **Scope tokens minimally.** Grant only the permissions listed above. `PUSH_TOKEN` + should be limited to the single target repository, not an org-wide token. +- **Rotate tokens** on the cadence your organization requires, and immediately if a + runner or log is suspected of exposure. +- **Self-hosted runner trust.** Phases 5–8 run on a persistent self-hosted runner + that retains the built `src/` tree, the `.venv`, and a warm ccache between phases. + Treat that runner as a sensitive host: restrict who can dispatch workflows and who + can access the runner, since a compromised runner has access to both PATs at + runtime. +- **`proof/` artifacts** contain logs and session transcripts. Review them before + publishing; do not attach them to public issues without checking for anything + sensitive. + +## Scope + +This policy covers the CodeWeave pipeline in this repository (workflows, composite +action, prompts, and tooling under `integration-test/_tools/`). Vulnerabilities in +the *target* repositories CodeWeave analyzes, or in third-party dependencies (the +GitHub Copilot CLI, PyTorch, CodeCarbon, etc.), should be reported to their +respective maintainers. From 0336afaacfc625d270c0d25c16d002460a54d3a0 Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 11:34:17 +0200 Subject: [PATCH 03/10] docs: wire README community links to the new files Replace the [ADD ...] placeholders in the README community section with links to CONTRIBUTING.md, SECURITY.md, ROADMAP.md, and the issues tab. Co-Authored-By: Claude Opus 4.8 --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 77478f1..c5981b3 100644 --- a/README.md +++ b/README.md @@ -245,10 +245,10 @@ CodeWeave is **experimental**. It is a working, end-to-end pipeline that has bee - **Documentation:** per-phase reference in [`docs/index.md`](docs/index.md); the design rationale in [`executive-summary.md`](executive-summary.md). - **Examples:** prompt and constraint files under [`work/`](work) and [`constraints/`](constraints); a sample run's artifacts appear in `proof/` (auto-created). `[ADD LINK TO A PUBLISHED EXAMPLE RUN]` -- **Roadmap:** `[ADD ROADMAP LINK]` — near-term focus is verifying portability to a second target via the manifest seam. -- **Contributing:** `[ADD CONTRIBUTING.md]` — issues and PRs welcome; please include the relevant `proof/` artifacts when reporting pipeline behavior. -- **Support:** `[ADD SUPPORT CHANNEL — GitHub Issues / Discussions]` -- **Security:** the pipeline handles two PATs and pushes branches to a target repo; scope tokens minimally as described in [Prerequisites](#quickstart). `[ADD SECURITY.md / disclosure policy]` +- **Roadmap:** [`ROADMAP.md`](ROADMAP.md) — near-term focus is publishing a real run and verifying portability to a second target via the manifest seam. +- **Contributing:** [`CONTRIBUTING.md`](CONTRIBUTING.md) — issues and PRs welcome; please include the relevant `proof/` artifacts when reporting pipeline behavior. +- **Support:** open a [GitHub issue](../../issues). `[ADD DISCUSSIONS LINK IF ENABLED]` +- **Security:** the pipeline handles two PATs and pushes branches to a target repo; scope tokens minimally as described in [Prerequisites](#quickstart). Report vulnerabilities via [`SECURITY.md`](SECURITY.md). ## License From 32484df10210e7ab3377b4f7d2974cbb1d3a9503 Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 11:43:30 +0200 Subject: [PATCH 04/10] fix: make deterministic tooling file I/O encoding-safe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ab_compare.py writes markdown containing non-ASCII (the Delta and CO2 symbols) via open(path, 'w') with no explicit encoding. That works on the Linux runner (UTF-8 locale) but crashes under any default codec that isn't UTF-8 (e.g. cp1252 on Windows) with UnicodeEncodeError — a latent portability defect in the code that produces every verdict. Surfaced by the new verdict-engine test suite. Make all file I/O in ab_compare.py, op_microbench.py, and diff_fuzz.py explicitly encoding='utf-8' for reads and writes. Co-Authored-By: Claude Opus 4.8 --- integration-test/_tools/ab_compare.py | 22 +++++++++++----------- integration-test/_tools/diff_fuzz.py | 2 +- integration-test/_tools/op_microbench.py | 2 +- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/integration-test/_tools/ab_compare.py b/integration-test/_tools/ab_compare.py index 2a28832..863b28d 100644 --- a/integration-test/_tools/ab_compare.py +++ b/integration-test/_tools/ab_compare.py @@ -76,7 +76,7 @@ def load_v2_records(path: Path, label: str): wall_clock_ms. Exits with a clear error if any record is stale-format: mixing schemas silently is exactly the contamination v2 exists to prevent. """ - with open(path, "r") as f: + with open(path, "r", encoding="utf-8") as f: records = json.load(f) if not records: @@ -145,10 +145,10 @@ def baseline_mode(runs_path: Path, output_dir: Path): } output_dir.mkdir(parents=True, exist_ok=True) - with open(output_dir / "baseline.json", "w") as f: + with open(output_dir / "baseline.json", "w", encoding="utf-8") as f: json.dump(baseline_result, f, indent=2) - with open(output_dir / "baseline-summary.md", "w") as f: + with open(output_dir / "baseline-summary.md", "w", encoding="utf-8") as f: f.write("# Baseline Statistics (v2 — per-iteration metrics)\n\n") f.write("| Metric | Value |\n") f.write("|--------|-------|\n") @@ -163,7 +163,7 @@ def baseline_mode(runs_path: Path, output_dir: Path): f.write(f"| Wall clock, mean (ms) — informational | {wall_stats['mean']:.0f} |\n") f.write(f"| Mean CO₂ (g) | {mean_co2:.6f} |\n") - with open(output_dir / "baseline-complete.md", "w") as f: + with open(output_dir / "baseline-complete.md", "w", encoding="utf-8") as f: f.write("# Baseline Complete\n\n") f.write(f"Baseline measurement completed with {iter_stats['n']} runs " f"(v2 per-iteration metrics).\n\n") @@ -186,7 +186,7 @@ def baseline_mode(runs_path: Path, output_dir: Path): def load_micro(path: Path): """Load an op_microbench.py output file: median_ns + raw_times_ns.""" - with open(path, "r") as f: + with open(path, "r", encoding="utf-8") as f: data = json.load(f) if "median_ns" not in data: print(f"Error: {path} is not a valid microbench file (no median_ns)", @@ -312,7 +312,7 @@ def drift_check(phase6_baseline_path: Path, current_a_iter_stats): Drift does not change the verdict (the paired design protects it) but is flagged for the Phase 8 report. """ - with open(phase6_baseline_path, "r") as f: + with open(phase6_baseline_path, "r", encoding="utf-8") as f: ref = json.load(f) if "median_iter_ms_mean" not in ref or "cv_iter" not in ref: @@ -434,10 +434,10 @@ def stat_block(iter_s, iterations_s, jpi_s, wall_s, energy_s): } output_prefix.parent.mkdir(parents=True, exist_ok=True) - with open(output_prefix.with_suffix(".json"), "w") as f: + with open(output_prefix.with_suffix(".json"), "w", encoding="utf-8") as f: json.dump(comparison_result, f, indent=2) - with open(output_prefix.with_suffix(".md"), "w") as f: + with open(output_prefix.with_suffix(".md"), "w", encoding="utf-8") as f: f.write("# A/B Comparison Results (v2)\n\n") f.write(f"## Decision: **{decision}** (signal: `{decision_signal}`)\n\n") f.write(f"Measurement path: `{measurement_path}`\n\n") @@ -533,7 +533,7 @@ def family_mode(reports_dir: Path, output_dir: Path, alpha=0.05): rows = [] for fp in sorted(reports_dir.glob("ab-comparison-opt*.json")): try: - with open(fp) as fh: + with open(fp, encoding="utf-8") as fh: d = json.load(fh) except (OSError, json.JSONDecodeError): continue @@ -573,10 +573,10 @@ def family_mode(reports_dir: Path, output_dir: Path, alpha=0.05): "family_size": m, "n_demoted": len(demoted), "cycles": rows} output_dir.mkdir(parents=True, exist_ok=True) - with open(output_dir / "family-correction.json", "w") as f: + with open(output_dir / "family-correction.json", "w", encoding="utf-8") as f: json.dump(result, f, indent=2) - with open(output_dir / "family-correction.md", "w") as f: + with open(output_dir / "family-correction.md", "w", encoding="utf-8") as f: f.write("# Family-Wise Correction (Holm-Bonferroni, FWER = " f"{alpha})\n\n") f.write(f"Family of {m} measured comparison(s); each cycle's primary-path " diff --git a/integration-test/_tools/diff_fuzz.py b/integration-test/_tools/diff_fuzz.py index ff663ad..33c76f8 100644 --- a/integration-test/_tools/diff_fuzz.py +++ b/integration-test/_tools/diff_fuzz.py @@ -358,7 +358,7 @@ def compare(mod, golden_path: Path, report_path: Path | None, spec_hash: str): } if report_path: report_path.parent.mkdir(parents=True, exist_ok=True) - with open(report_path, "w") as f: + with open(report_path, "w", encoding="utf-8") as f: json.dump(report, f, indent=2) n_fail = sum(1 for r in results if not r["passed"]) diff --git a/integration-test/_tools/op_microbench.py b/integration-test/_tools/op_microbench.py index 9d526f0..5827bf0 100644 --- a/integration-test/_tools/op_microbench.py +++ b/integration-test/_tools/op_microbench.py @@ -111,7 +111,7 @@ def main(): } args.output.parent.mkdir(parents=True, exist_ok=True) - with open(args.output, "w") as f: + with open(args.output, "w", encoding="utf-8") as f: json.dump(result, f, indent=2) print(f"Microbench {spec['op']}: median={measurement.median * 1e6:.1f}us " From b81a7b1cc0cd85aa95e0b06bebae179d873bc7bf Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 11:43:47 +0200 Subject: [PATCH 05/10] test: add a verdict-engine test suite (31 tests) ab_compare.py is the single source of truth for every KEEP / INVESTIGATE / REVERT verdict, and had no tests. Add tests/tools/test_ab_compare.py covering: - compute_stats (empty, single, known-value, zero-mean guard) - the directional decision table for both measurement paths, incl. the noise-floor gate (significant-but-sub-MDE must not KEEP), micro corroboration, regression priority, and the microbench 'no primary signal -> REVERT' rule - Holm-Bonferroni step-down (known thresholds, monotone failure, input-order preservation) - micro_comparison significance (raw-sample t-test, unknown-without-samples, sub-floor delta) - baseline / compare / family CLI modes end to end on synthetic records, incl. stale-format rejection and a KEEP demoted by the family-wise correction Add pytest.ini (scopes CI to repo tooling, not the generated target harness) and requirements-dev.txt (pinned pytest/numpy/scipy). All 31 pass locally on Python 3.12. Co-Authored-By: Claude Opus 4.8 --- pytest.ini | 7 + requirements-dev.txt | 7 + tests/tools/test_ab_compare.py | 336 +++++++++++++++++++++++++++++++++ 3 files changed, 350 insertions(+) create mode 100644 pytest.ini create mode 100644 requirements-dev.txt create mode 100644 tests/tools/test_ab_compare.py diff --git a/pytest.ini b/pytest.ini new file mode 100644 index 0000000..1b592f6 --- /dev/null +++ b/pytest.ini @@ -0,0 +1,7 @@ +[pytest] +# Repo-tooling tests only (the deterministic verdict/measurement engine). +# The target harness under integration-test/tests/ is generated per-target and +# runs inside a pipeline job against the built target, not in repo CI. +testpaths = tests +python_files = test_*.py +addopts = -q diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..32ea15c --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,7 @@ +# Development / CI dependencies for the CodeWeave repo tooling tests. +# These cover only the deterministic pipeline tools under integration-test/_tools/ +# (the statistics/verdict engine), NOT the target harness dependencies, which are +# installed per-target from SOURCE-UNDER-INVESTIGATION.md during a pipeline run. +pytest>=8,<10 +numpy>=1.26 +scipy>=1.11 diff --git a/tests/tools/test_ab_compare.py b/tests/tools/test_ab_compare.py new file mode 100644 index 0000000..1f5e9ac --- /dev/null +++ b/tests/tools/test_ab_compare.py @@ -0,0 +1,336 @@ +"""Unit + integration tests for the verdict engine (integration-test/_tools/ab_compare.py). + +`ab_compare.py` is the single source of truth for every KEEP / INVESTIGATE / REVERT +verdict CodeWeave produces. These tests lock in the behaviour that matters for +correctness: + +- the directional decision table (`decide`) — including the noise-floor gate that + stops a statistically-significant-but-trivial change from becoming a KEEP, +- the Holm-Bonferroni family-wise correction (`holm_bonferroni`), +- the microbench significance rule (`micro_comparison`), +- basic descriptive stats (`compute_stats`), +- and the three CLI modes end to end (baseline / compare / family) against + synthetic run records. + +The module is a CLI script; we load it by path so the tests run regardless of cwd. +""" + +import importlib.util +import json +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] +MODULE_PATH = REPO_ROOT / "integration-test" / "_tools" / "ab_compare.py" + + +def _load_module(): + spec = importlib.util.spec_from_file_location("ab_compare", MODULE_PATH) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +ab = _load_module() + + +# --------------------------------------------------------------------------- # +# compute_stats +# --------------------------------------------------------------------------- # + +def test_compute_stats_empty(): + assert ab.compute_stats([]) == {"mean": 0, "std": 0, "cv": 0, "n": 0} + + +def test_compute_stats_single_value_has_no_spread(): + s = ab.compute_stats([42.0]) + assert s["n"] == 1 + assert s["mean"] == 42.0 + assert s["std"] == 0 + assert s["cv"] == 0 + + +def test_compute_stats_known_values(): + # Sample std of [2,4,4,4,5,5,7,9] is 2.13809 (n-1 denominator), mean 5. + s = ab.compute_stats([2, 4, 4, 4, 5, 5, 7, 9]) + assert s["n"] == 8 + assert s["mean"] == pytest.approx(5.0) + assert s["std"] == pytest.approx(2.138090, abs=1e-5) + assert s["cv"] == pytest.approx(2.138090 / 5.0, abs=1e-5) + + +def test_compute_stats_zero_mean_cv_is_zero_not_error(): + s = ab.compute_stats([-1, 0, 1]) + assert s["mean"] == 0 + assert s["cv"] == 0 # guarded division, must not raise + + +# --------------------------------------------------------------------------- # +# decide — the directional decision table +# --------------------------------------------------------------------------- # + +def _micro(delta, significant): + return {"delta_micro_pct": delta, "micro_significant": significant} + + +class TestDecideEndToEnd: + PATH = "end-to-end" + + def test_significant_faster_is_keep(self): + d, sig = ab.decide(5.0, 0.001, None, self.PATH, mde_pct=2.0) + assert (d, sig) == ("KEEP", "e2e") + + def test_significant_slower_is_revert_regression(self): + d, sig = ab.decide(-5.0, 0.001, None, self.PATH, mde_pct=2.0) + assert (d, sig) == ("REVERT", "e2e-regression") + + def test_significant_but_below_noise_floor_is_only_a_trend(self): + # p rejects, but |Δ| (1%) < MDE (2%): the floor gate must block the KEEP. + d, sig = ab.decide(1.0, 0.001, None, self.PATH, mde_pct=2.0) + assert (d, sig) == ("INVESTIGATE", "trend") + + def test_not_significant_positive_is_trend(self): + d, sig = ab.decide(5.0, 0.20, None, self.PATH, mde_pct=2.0) + assert (d, sig) == ("INVESTIGATE", "trend") + + def test_flat_no_signal_is_revert_no_effect(self): + d, sig = ab.decide(0.0, 0.9, None, self.PATH, mde_pct=2.0) + assert (d, sig) == ("REVERT", "no-effect") + + def test_micro_corroboration_can_keep_when_e2e_flat(self): + # e2e not significant, but a significant positive microbench → KEEP. + d, sig = ab.decide(0.5, 0.4, _micro(8.0, True), self.PATH, mde_pct=2.0) + assert (d, sig) == ("KEEP", "microbench") + + def test_micro_regression_takes_priority_over_path(self): + d, sig = ab.decide(0.0, 0.9, _micro(-9.0, True), self.PATH, mde_pct=2.0) + assert (d, sig) == ("REVERT", "micro-regression") + + +class TestDecideMicrobenchPath: + PATH = "microbench" + + def test_absent_microbench_reverts_not_falls_back_to_e2e(self): + # e2e looks like a big significant win, but on the microbench path a + # missing primary signal must REVERT rather than act on the sub-floor e2e. + d, sig = ab.decide(9.0, 0.001, None, self.PATH, mde_pct=2.0) + assert (d, sig) == ("REVERT", "no-microbench") + + def test_unknown_significance_reverts(self): + d, sig = ab.decide(9.0, 0.001, _micro(9.0, None), self.PATH, mde_pct=2.0) + assert (d, sig) == ("REVERT", "no-microbench") + + def test_significant_micro_win_is_keep(self): + d, sig = ab.decide(0.0, 0.9, _micro(6.0, True), self.PATH, mde_pct=2.0) + assert (d, sig) == ("KEEP", "microbench") + + def test_e2e_win_unsupported_by_micro_is_investigate(self): + # e2e significant + positive, micro present but not significant → the e2e + # "win" is implausible for a sub-floor op, so INVESTIGATE (drift suspect). + d, sig = ab.decide(9.0, 0.001, _micro(0.3, False), self.PATH, mde_pct=2.0) + assert (d, sig) == ("INVESTIGATE", "e2e-unsupported-by-micro") + + def test_micro_regression_reverts(self): + d, sig = ab.decide(0.0, 0.9, _micro(-6.0, True), self.PATH, mde_pct=2.0) + assert (d, sig) == ("REVERT", "micro-regression") + + +# --------------------------------------------------------------------------- # +# holm_bonferroni +# --------------------------------------------------------------------------- # + +def test_holm_step_down_known_example(): + # m=3, alpha=0.05. Sorted thresholds: 0.05/3, 0.05/2, 0.05/1. + pvals = [0.001, 0.04, 0.03] + survives, thresholds = ab.holm_bonferroni(pvals, 0.05) + assert survives == [True, False, False] + assert thresholds[0] == pytest.approx(0.05 / 3) + assert thresholds[2] == pytest.approx(0.05 / 2) + assert thresholds[1] == pytest.approx(0.05 / 1) + + +def test_holm_all_survive_when_all_tiny(): + survives, _ = ab.holm_bonferroni([0.001, 0.002, 0.003], 0.05) + assert survives == [True, True, True] + + +def test_holm_is_monotone_once_one_fails_all_larger_fail(): + # 0.02 passes 0.05/3=0.0167? No. So nothing survives despite 0.001 being tiny? + # 0.001 is rank 1 (thr 0.0167) -> survives. 0.02 rank2 (thr 0.025) -> 0.02<=0.025 survives. + # 0.049 rank3 (thr 0.05) -> survives. Use a clearer failing case: + survives, _ = ab.holm_bonferroni([0.001, 0.03, 0.9], 0.05) + # rank1 0.001<=0.0167 T; rank2 0.03<=0.025 F -> stop; rank3 F + assert survives == [True, False, False] + + +def test_holm_preserves_input_order(): + # Smallest p is last in input; survival must map back to index 2. + survives, _ = ab.holm_bonferroni([0.9, 0.9, 0.0001], 0.05) + assert survives == [False, False, True] + + +# --------------------------------------------------------------------------- # +# micro_comparison +# --------------------------------------------------------------------------- # + +def test_micro_comparison_invalid_baseline_median(): + out = ab.micro_comparison({"median_ns": 0}, {"median_ns": 100}) + assert out["micro_significant"] is False + assert "invalid" in out["micro_note"] + + +def test_micro_comparison_significant_speedup_with_raw_samples(): + # Base ~1000ns, variant ~900ns, tight samples → significant + clears 2% floor. + base = {"median_ns": 1000.0, "raw_times_ns": [1000, 1001, 999, 1000, 1002, 998]} + var = {"median_ns": 900.0, "raw_times_ns": [900, 901, 899, 900, 902, 898]} + out = ab.micro_comparison(base, var) + assert out["delta_micro_pct"] == pytest.approx(10.0) + assert out["micro_significant"] is True + assert out["micro_p_value"] < 0.05 + + +def test_micro_comparison_without_raw_samples_is_unknown_not_true(): + base = {"median_ns": 1000.0} + var = {"median_ns": 800.0} + out = ab.micro_comparison(base, var) + # A bare median delta must NOT read as significant; significance is UNKNOWN. + assert out["micro_significant"] is None + assert out["micro_p_value"] is None + + +def test_micro_comparison_tiny_delta_not_significant_even_if_p_small(): + # Very tight, but only a 0.5% median delta — below the 2% practical floor. + # (Small real spread so the t-test is well-defined, not degenerate.) + base = {"median_ns": 1000.0, "raw_times_ns": [999, 1000, 1001, 1000, 999, 1001, 1000, 1000]} + var = {"median_ns": 995.0, "raw_times_ns": [994, 995, 996, 995, 994, 996, 995, 995]} + out = ab.micro_comparison(base, var) + assert abs(out["delta_micro_pct"]) < ab.MICRO_PRACTICAL_FLOOR_PCT + assert out["micro_significant"] is False + + +# --------------------------------------------------------------------------- # +# CLI modes end to end (synthetic records) +# --------------------------------------------------------------------------- # + +def _run_record(run_id, median_iter_ms, iterations, wall_ms=30000, energy=100.0): + return { + "run_id": run_id, + "median_iter_ms": median_iter_ms, + "iterations": iterations, + "wall_clock_ms": wall_ms, + "energy_joules": energy, + "co2_grams": 0.01, + } + + +def _write_json(path, obj): + path.write_text(json.dumps(obj), encoding="utf-8") + + +def test_baseline_mode_writes_expected_stats(tmp_path): + runs = [_run_record(i, 10.0 + (i % 2) * 0.1, 3000) for i in range(5)] + runs_path = tmp_path / "run-records.json" + _write_json(runs_path, runs) + out_dir = tmp_path / "reports" + + result = ab.baseline_mode(runs_path, out_dir) + + assert result["schema"] == "v2" + assert result["n"] == 5 + # mde_pct is exactly MDE_FACTOR * cv * 100 by construction. + assert result["mde_pct"] == pytest.approx(ab.MDE_FACTOR * result["cv_iter"] * 100) + assert result["energy_valid"] is True + assert (out_dir / "baseline.json").exists() + assert (out_dir / "baseline-summary.md").exists() + assert (out_dir / "baseline-complete.md").exists() + + +def test_baseline_mode_flags_zero_energy_as_invalid(tmp_path): + runs = [_run_record(i, 10.0, 3000, energy=0.0) for i in range(5)] + runs_path = tmp_path / "run-records.json" + _write_json(runs_path, runs) + result = ab.baseline_mode(runs_path, tmp_path / "reports") + assert result["energy_valid"] is False + + +def test_load_v2_records_rejects_stale_format(tmp_path): + stale = [{"run_id": 1, "wall_clock_ms": 30000}] # no iterations / median_iter_ms + p = tmp_path / "stale.json" + _write_json(p, stale) + with pytest.raises(SystemExit): + ab.load_v2_records(p, "stale") + + +def test_compare_mode_end_to_end_keep(tmp_path): + # Variant clearly faster per iteration (9 vs 10 ms), tight variance → KEEP. + base = [_run_record(i, 10.0 + (i % 2) * 0.05, 3000) for i in range(5)] + var = [_run_record(i, 9.0 + (i % 2) * 0.05, 3333) for i in range(5)] + base_p = tmp_path / "base.json" + var_p = tmp_path / "var.json" + _write_json(base_p, base) + _write_json(var_p, var) + prefix = tmp_path / "ab-comparison-opt1" + + result = ab.compare_mode(base_p, var_p, prefix, measurement_path="end-to-end") + + assert result["decision"] == "KEEP" + assert result["decision_signal"] == "e2e" + assert result["delta_e2e_pct"] > 0 + assert (tmp_path / "ab-comparison-opt1.json").exists() + assert (tmp_path / "ab-comparison-opt1.md").exists() + + +def test_compare_mode_requires_two_runs_per_group(tmp_path): + base = [_run_record(0, 10.0, 3000)] + var = [_run_record(0, 9.0, 3333)] + base_p, var_p = tmp_path / "b.json", tmp_path / "v.json" + _write_json(base_p, base) + _write_json(var_p, var) + with pytest.raises(SystemExit): + ab.compare_mode(base_p, var_p, tmp_path / "out", measurement_path="end-to-end") + + +def test_family_mode_demotes_non_surviving_keep(tmp_path): + reports = tmp_path / "reports" + reports.mkdir() + # Family of 3 KEEPs at alpha=0.05. Holm step-down thresholds (ascending): + # 0.05/3=0.0167, 0.05/2=0.025, 0.05/1=0.05. + # opt1 p=0.001 -> rank1, 0.001<=0.0167 survives (stays KEEP) + # opt2 p=0.03 -> rank2, 0.03 >0.025 fails (demoted) + # opt3 p=0.04 -> rank3, step-down stops after opt2 fails (demoted) + _write_json(reports / "ab-comparison-opt1.json", { + "measurement_path": "end-to-end", "decision": "KEEP", + "decision_signal": "e2e", "p_value_e2e": 0.001, + }) + _write_json(reports / "ab-comparison-opt2.json", { + "measurement_path": "end-to-end", "decision": "KEEP", + "decision_signal": "e2e", "p_value_e2e": 0.03, + }) + _write_json(reports / "ab-comparison-opt3.json", { + "measurement_path": "end-to-end", "decision": "KEEP", + "decision_signal": "e2e", "p_value_e2e": 0.04, + }) + + result = ab.family_mode(reports, reports, alpha=0.05) + + assert result["family_size"] == 3 + by_opt = {r["opt"]: r for r in result["cycles"]} + assert by_opt["1"]["corrected_decision"] == "KEEP" + assert by_opt["2"]["corrected_decision"] == "INVESTIGATE" + assert by_opt["2"]["corrected_signal"] == "keep-not-family-significant" + assert by_opt["3"]["corrected_decision"] == "INVESTIGATE" + assert result["n_demoted"] == 2 + assert (reports / "family-correction.json").exists() + assert (reports / "family-correction.md").exists() + + +def test_family_mode_never_promotes_a_revert(tmp_path): + reports = tmp_path / "reports" + reports.mkdir() + _write_json(reports / "ab-comparison-opt1.json", { + "measurement_path": "end-to-end", "decision": "REVERT", + "decision_signal": "no-effect", "p_value_e2e": 0.0001, + }) + result = ab.family_mode(reports, reports, alpha=0.05) + assert result["cycles"][0]["corrected_decision"] == "REVERT" From dcbde626709af38ea4ac9e46796b4dd378706b22 Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 11:43:47 +0200 Subject: [PATCH 06/10] ci: add repo self-validation workflow and .gitattributes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add .github/workflows/ci.yml (hosted-runner CI, separate from the CodeWeave pipeline and touching no secrets/target/Copilot): runs the verdict-engine tests on every push and PR, and lints the pipeline's own workflow YAML (actionlint) and shell scripts (shellcheck). Add .gitattributes to LF-normalize repo text — the pipeline runs on Linux and its shell scripts must stay LF — which also silences the CRLF churn on Windows checkouts. Co-Authored-By: Claude Opus 4.8 --- .gitattributes | 17 +++++++++++++++ .github/workflows/ci.yml | 46 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) create mode 100644 .gitattributes create mode 100644 .github/workflows/ci.yml diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..b6b626f --- /dev/null +++ b/.gitattributes @@ -0,0 +1,17 @@ +# Normalize line endings. The pipeline runs on Linux; shell scripts MUST stay LF +# or they break on the runner. Keep repo text LF-normalized regardless of the OS +# a contributor commits from (silences the CRLF warnings on Windows checkouts). +* text=auto eol=lf + +# Explicitly LF for executable/interpreted files that are line-ending sensitive. +*.sh text eol=lf +*.py text eol=lf +*.js text eol=lf +*.yml text eol=lf +*.yaml text eol=lf +*.md text eol=lf + +# Treat these as binary (no EOL munging, no diffs). +*.pdf binary +*.png binary +*.jpg binary diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..0357605 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,46 @@ +name: CI + +# Repo self-validation, distinct from the CodeWeave pipeline (codeweave.yml). +# Runs on hosted runners and does NOT touch the target repo, secrets, or Copilot. +# It exercises the deterministic tooling that decides every verdict, and lints the +# pipeline's own workflow YAML and shell scripts. + +on: + push: + branches: [main] + pull_request: + workflow_dispatch: + +permissions: + contents: read + +jobs: + tools-tests: + name: Verdict-engine tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install dev dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements-dev.txt + - name: Run tooling tests + run: python -m pytest tests -q + + lint-workflows: + name: Lint workflows & shell + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: actionlint (GitHub Actions workflow linter) + uses: raven-actions/actionlint@v2 + with: + fail-on-error: true + - name: shellcheck (shell script linter) + uses: ludeeus/action-shellcheck@master + with: + scandir: "." + severity: warning From c7b233d1a5fca64caf3137dc746bd7a1fecdfd30 Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 11:43:47 +0200 Subject: [PATCH 07/10] docs: note verdict-engine test coverage in README status MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a truthful 'Tested' line to the project status: the verdict engine is now covered by a unit + integration suite run in CI. Does not change the overall experimental status — the remaining blockers (a published run, second-target portability) are empirical. Co-Authored-By: Claude Opus 4.8 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c5981b3..70c6e44 100644 --- a/README.md +++ b/README.md @@ -228,6 +228,7 @@ This is a positioning of *discipline*, not a knock on coding agents — CodeWeav CodeWeave is **experimental**. It is a working, end-to-end pipeline that has been built and iterated against a real target (PyTorch, CPU), but it has not been hardened for arbitrary repositories or published with reproducible headline results. - **Working today:** the full eight-phase run against the reference target — generate→validate documentation loops, source build, statistical baseline, correctness-gated optimization cycles with paired A/B verdicts, and the aggregate report with PR drafts. +- **Tested:** the verdict engine (`integration-test/_tools/ab_compare.py`) — the code that decides every KEEP / INVESTIGATE / REVERT — is covered by a unit + integration suite (decision table, noise-floor gate, Holm–Bonferroni correction, and all three CLI modes). CI ([`.github/workflows/ci.yml`](.github/workflows/ci.yml)) runs it on every push and PR and lints the pipeline's workflow YAML and shell scripts. - **Experimental:** portability beyond the PyTorch/CPU reference target. The manifest seam is designed for it, but other toolchains are unverified. `[VERIFY portability on a second target]` - **Planned / manual by design:** opening PRs (Phase 8 drafts them); acting on `INVESTIGATE` results (surfaced, never auto-investigated). - **Known limitations:** From cf20b1f68c471af6030f3843d5c908bd7c422728 Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 12:06:39 +0200 Subject: [PATCH 08/10] ci: make the lint gate pass on real defects, not style noise The first CI run flagged 23 shellcheck findings in the existing pipeline workflows: 21 info/style advisories + 2 warning-level unused variables. - Remove the two genuinely dead variables (real fixes, no behavior change): BOOK_INDEX_BUILT in phase-2-adr.yml (its sibling ADR_INDEX_BUILT is the one echoed to GITHUB_OUTPUT) and the shadowed VENV_PY assignment in the Phase 5 block of phase-5-6-build-baseline.yml (the used copy is re-declared in the Phase 6 block). - Gate shellcheck at severity>=warning (SHELLCHECK_OPTS) so info/style hints on the already-in-production pipeline shell don't block, while real defects still fail CI. Rewriting the 21 advisories belongs in a dedicated shell-hardening pass, not a docs PR. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/ci.yml | 12 ++++++++++-- .github/workflows/phase-2-adr.yml | 2 -- .github/workflows/phase-5-6-build-baseline.yml | 1 - 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0357605..696340b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,13 +33,21 @@ jobs: lint-workflows: name: Lint workflows & shell runs-on: ubuntu-latest + # The pipeline's workflow shell is authored (and partly Copilot-generated) and + # already runs in production. We gate on genuine defects (shellcheck severity + # >= warning: unused vars, quoting bugs that change behavior, etc.), not on + # info/style advisories (SC2086 word-splitting hints, SC2129 redirect grouping, + # ...), which would be high-churn/low-value to rewrite blindly here. Raise the + # bar in a dedicated shell-hardening pass, not a docs PR. + env: + SHELLCHECK_OPTS: --severity=warning steps: - uses: actions/checkout@v4 - - name: actionlint (GitHub Actions workflow linter) + - name: actionlint (GitHub Actions workflow linter, incl. embedded shellcheck) uses: raven-actions/actionlint@v2 with: fail-on-error: true - - name: shellcheck (shell script linter) + - name: shellcheck (standalone .sh scripts) uses: ludeeus/action-shellcheck@master with: scandir: "." diff --git a/.github/workflows/phase-2-adr.yml b/.github/workflows/phase-2-adr.yml index 4d5c73f..2292764 100644 --- a/.github/workflows/phase-2-adr.yml +++ b/.github/workflows/phase-2-adr.yml @@ -249,7 +249,6 @@ jobs: - name: Generate navigation indexes id: build-indexes run: | - BOOK_INDEX_BUILT=0 ADR_INDEX_BUILT=0 node .github/scripts/generate-indexes.js --generate-adr-index 2>&1 | tee "proof/adr-index.md" || true @@ -263,7 +262,6 @@ jobs: git add book/BOOK-INDEX.md if ! git diff --cached --quiet; then git commit -m "Phase 2: update BOOK-INDEX.md" - BOOK_INDEX_BUILT=1 fi fi diff --git a/.github/workflows/phase-5-6-build-baseline.yml b/.github/workflows/phase-5-6-build-baseline.yml index 4511998..65e4da8 100644 --- a/.github/workflows/phase-5-6-build-baseline.yml +++ b/.github/workflows/phase-5-6-build-baseline.yml @@ -145,7 +145,6 @@ jobs: # Copilot AUTHORS integration-test/build-source.sh (read/write/edit only — # no shell needed); the workflow EXECUTES it directly so build output # streams into the job log and no tool permissions can block the build. - VENV_PY="$PWD/integration-test/${HARNESS_VENV_PYTHON:-.venv/bin/python}" BUILD_OK=false for ATTEMPT in $(seq 1 "${PHASE5_MAX_ITERATIONS}"); do if [ -f integration-test/build-error.md ]; then MODE=repair; else MODE=author; fi From e2d49fa63e17444def9dc628a3d02179a5de3741 Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 12:16:21 +0200 Subject: [PATCH 09/10] docs: center the energy mission, generalize to any codebase, embed the design pillars Reframe the documentation around what CodeWeave is actually for: finding energy and CO2 hotspots and shipping proven-real efficiency fixes. - Lead with the energy/carbon objective; keep the honest note that energy is the target and per-iteration latency is the resolvable lever. - Present CodeWeave as language- and stack-agnostic (toolchain read from a manifest); remove target-specific framing and examples from the docs. The config defaults and constraints still describe a concrete target and are left to a separate change. - Add a 'What CodeWeave takes seriously' section making three deliberate design pillars explicit: statistical proof embedded in the acceptance path, the quality of the context built before any change, and integration tests reverse-engineered from the real system. - Convert the docs to flowing prose and remove em-dashes throughout. Co-Authored-By: Claude Opus 4.8 --- CONTRIBUTING.md | 101 ++++++++++--------- README.md | 260 +++++++++++++++++++++++------------------------- ROADMAP.md | 100 +++++++++---------- SECURITY.md | 10 +- 4 files changed, 232 insertions(+), 239 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 91cffa7..9025e07 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,65 +1,72 @@ # Contributing to CodeWeave Thanks for your interest in CodeWeave. It is an experimental, research-grade -automation pipeline, so contributions that make it more **portable**, more -**auditable**, or more **statistically rigorous** are especially welcome. - -Please read this guide before opening an issue or a pull request. +automation pipeline, so the contributions that help most are the ones that make it +more portable, more auditable, or more statistically rigorous. Please read this +guide before opening an issue or a pull request. ## Ways to contribute -- **Report pipeline behavior** — a phase that stalls, a gate that misfires, a - verdict that looks wrong. Include the relevant `proof/` artifacts (see below). -- **Improve portability** — CodeWeave was built against a PyTorch/CPU reference - target. Making it run against a second target (via the - `integration-test/harness-manifest.json` seam) is the highest-value area. -- **Sharpen the measurement science** — the verdict logic lives in - `integration-test/_tools/ab_compare.py`. Changes here must come with a written - rationale and, ideally, a worked example. -- **Docs** — clarifications to `docs/`, the `README.md`, or `executive-summary.md`. +The most valuable areas are portability, measurement science, and reporting real +behavior. On portability, CodeWeave is designed to run against any codebase by +reading the target's toolchain from a manifest, and driving it end to end against a +new kind of target (through the `integration-test/harness-manifest.json` seam) is +the highest-leverage work available. On measurement science, the verdict logic +lives in `integration-test/_tools/ab_compare.py`, and changes there should arrive +with a written rationale and, ideally, a worked example. Reporting pipeline +behavior is also genuinely useful: a phase that stalls, a gate that misfires, or a +verdict that looks wrong all make good issues, as do clarifications to `docs/`, the +`README.md`, or the executive summary. ## Before you start -- CodeWeave runs as **GitHub Actions workflows on a self-hosted runner**, not as a - local CLI. There is no `npm install && npm test` loop; changes are exercised by - dispatching the workflows. See the [Quickstart](README.md#quickstart). -- Smoke-test structural changes with a dry run before a real run: - ```bash - gh workflow run codeweave.yml -f dry_run=true - ``` - `dry_run` skips Copilot invocations and prerequisite checks, so you can validate - workflow wiring cheaply. +CodeWeave runs as GitHub Actions workflows on a self-hosted runner rather than as a +local command-line tool. There is no local install-and-test loop; changes are +exercised by dispatching the workflows, as described in the +[Quickstart](README.md#quickstart). Smoke-test structural changes cheaply with a +dry run before a real run, because a dry run skips the Copilot invocations and the +prerequisite checks: -## Reporting an issue +```bash +gh workflow run codeweave.yml -f dry_run=true +``` -A good report includes: +The deterministic tooling under `integration-test/_tools/` is covered by tests you +can run locally. Install the development dependencies and run the suite: -1. **Which phase** (1–8) and the workflow that ran it. -2. **What you expected vs. what happened.** -3. **The relevant `proof/` artifacts** — generate/validate logs, session - transcripts, gate diagnostics, and measurement records are all written there. - These are the primary evidence for any pipeline behavior; attach or paste the - relevant files rather than describing them. -4. **Your configuration** — the relevant parts of `.github/codeweave.config` - (redact anything sensitive) and your `constraints/` files. +```bash +pip install -r requirements-dev.txt +python -m pytest +``` + +## Reporting an issue -Do **not** include secrets (`COPILOT_TOKEN`, `PUSH_TOKEN`) or their values. +A good report names the phase involved and the workflow that ran it, states what +you expected against what actually happened, and includes the relevant `proof/` +artifacts. Those artifacts, the generate and validate logs, the session +transcripts, the gate diagnostics, and the measurement records, are the primary +evidence for any pipeline behavior, so please attach or paste the relevant files +rather than describing them from memory. Include the relevant parts of your +`.github/codeweave.config` and your `constraints/` files as well, with anything +sensitive redacted. Never include the values of the `COPILOT_TOKEN` or `PUSH_TOKEN` +secrets. ## Pull requests -- **Branch naming:** use a descriptive prefix, e.g. `docs/…`, `pipeline/…`, - `config/…`, `work/…`, matching the existing history. -- **Commit messages:** follow the repository style — a `scope: imperative summary` - subject line, then a body explaining *why*. Keep one logical change per commit. -- **Scope:** keep PRs focused. A change to prompt files (`work/`) is separate from - a change to workflow logic (`.github/workflows/`). -- **The core invariant:** CodeWeave's whole premise is that *generators propose and - an independent, deterministic pipeline disposes*. Changes must preserve that - separation — an agent must never be able to self-certify a phase as complete, and - the pipeline must remain the single owner of commits and verdicts. -- **Measurement changes** to `ab_compare.py` or the gate logic should explain how - they affect the trustworthiness of a verdict (significance, noise floor, drift - control, family-wise correction). +Use a descriptive branch prefix that matches the existing history, such as `docs/`, +`pipeline/`, `config/`, or `work/`. Follow the repository's commit style: a subject +line of the form `scope: imperative summary`, then a body that explains why. Keep +one logical change per commit, and keep pull requests focused, so that a change to +prompt files under `work/` stays separate from a change to workflow logic under +`.github/workflows/`. + +One invariant matters above the rest. CodeWeave's whole premise is that generators +propose and an independent, deterministic pipeline disposes, so any change must +preserve that separation. An agent must never be able to certify its own phase as +complete, and the pipeline must remain the single owner of commits and verdicts. +Changes to the verdict logic in `ab_compare.py` or to the correctness gate should +explain how they affect the trustworthiness of a verdict, in terms of significance, +the noise floor, drift control, or the family-wise correction. ## Code of conduct @@ -68,4 +75,4 @@ Be respectful and constructive. `[ADD CODE_OF_CONDUCT.md IF ADOPTING ONE]` ## License By contributing, you agree that your contributions are licensed under the project's -**GNU General Public License v3.0 or later** (see [`LICENSE`](LICENSE)). +GNU General Public License v3.0 or later. See [`LICENSE`](LICENSE). diff --git a/README.md b/README.md index 70c6e44..884c64c 100644 --- a/README.md +++ b/README.md @@ -1,258 +1,246 @@ # CodeWeave -**An autonomous code-optimization pipeline that ships only the changes it can prove faster** — every candidate clears a correctness gate and wins a statistical A/B verdict before it becomes a PR. No change lands on an agent's unverified claim. +**An autonomous pipeline that hunts energy and CO₂ hotspots in a codebase and ships the efficiency fixes it can prove real.** It works against a codebase of any language or stack, reading the target's toolchain from a manifest rather than assuming one. Every candidate change is measured for energy and carbon, cleared by a correctness gate, and accepted only when a statistical test says the improvement is genuine. The result is greener code that you do not have to take an agent's word for. [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](LICENSE) ![Pipeline: GitHub Actions](https://img.shields.io/badge/pipeline-GitHub%20Actions-2088FF?logo=githubactions&logoColor=white) ![Status: Experimental](https://img.shields.io/badge/status-experimental-orange) -![Target: PyTorch (CPU)](https://img.shields.io/badge/reference%20target-PyTorch%20CPU-EE4C2C) +![Objective: energy + CO₂](https://img.shields.io/badge/objective-energy%20%2B%20CO%E2%82%82-2ea44f) -> _CodeWeave drives the GitHub Copilot CLI across an 8-phase pipeline: it clones a target repo, documents it from the ground up, builds a measurement harness, establishes a statistical baseline, then runs autonomous optimization cycles — each accepted or rejected by an A/B experiment, not by the agent that wrote it._ +> _CodeWeave exists to make software cost less energy. It drives the GitHub Copilot CLI across an eight-phase pipeline: it clones a target repository, documents it from the ground up, builds an energy- and latency-aware measurement harness, profiles the target for the paths that burn the most, and then runs autonomous optimization cycles. It measures energy and carbon (via CodeCarbon) on every change, and it accepts or rejects each change by experiment rather than by the judgement of the agent that wrote it._ + Recommended hero asset: a terminal recording, or the Phase 8 report table, showing one + optimization cycle running from correctness gate to paired A/B measurement to a verdict, + with the per-iteration energy and carbon figures alongside. + [ADD DEMO GIF] [ADD LINK TO A REAL proof/ RUN] --> -**Three things that make the verdicts trustworthy:** +### What it does -- **Correctness before performance.** Every change passes a six-stage gate — import + op check, integration smoke, a targeted unit test, OpInfo, an output diff, and a *blocking differential fuzz* against a golden captured on the base build — **before a single timing run is spent on it.** -- **A real speedup, not noise.** A change is only KEEP when Welch's t-test rejects (`p < 0.05`) **and** the effect clears a measured noise floor (Minimum Detectable Effect). Phase 8 then applies a Holm–Bonferroni correction across every cycle and demotes any KEEP that doesn't survive the campaign-wide test. -- **Auditable end to end.** Every generate/validate log, session transcript, gate diagnostic, and measurement record is written to `proof/`, with git history as the diff trail. Nothing is self-certified. +CodeWeave begins by finding where the energy actually goes. It profiles the target to rank the hotspots that dominate its energy and carbon cost, and it measures energy and carbon on every candidate change through CodeCarbon, normalized per iteration so the figure reflects the code rather than the wall clock. Energy and CO₂ are the objective that the entire pipeline is pointed at. + +It then fixes those hotspots without breaking the program. Each proposed change must pass a six-stage correctness gate, covering a build and import check, an integration smoke test, a targeted unit test, a run of the target's own broad test suite, an output diff, and a blocking differential fuzz against a golden output captured on the base build. No measurement run is spent on a change until it has been shown to be correct. + +Finally, it ships only the improvements it can prove. A change is accepted when the improvement is statistically real, which requires Welch's t-test to reject the null hypothesis at `p < 0.05` and the effect size to clear a measured noise floor. A Holm-Bonferroni correction is then applied across the whole campaign so that running many experiments does not manufacture a false winner. Every log, session transcript, gate diagnostic, and measurement record is written to `proof/`, so nothing is self-certified and every decision can be audited after the fact. + +> **The honest mechanism.** Energy and carbon are the goal, and they are measured and reported on every change. CodeCarbon's resolution, however, is too coarse to arbitrate a single optimization on its own. The accept-or-reject verdict therefore runs on per-iteration latency, which is the tightly resolvable proxy for the same hot path, with energy and carbon reported next to each verdict. Put simply, energy is the target and latency is the lever precise enough to act on. ### In action -Point CodeWeave at a repository and dispatch one workflow: +You point CodeWeave at a repository and dispatch a single workflow: ```bash -# .github/codeweave.config — the whole run is configured here, nothing is hardcoded -EXTERNAL_REPO_NAME=pytorch -EXTERNAL_REPO_URL=https://github.com/your-org/pytorch +# .github/codeweave.config: the whole run is configured here; nothing is hardcoded +EXTERNAL_REPO_NAME=your-project +EXTERNAL_REPO_URL=https://github.com/your-org/your-repo EXTERNAL_REPO_BRANCH=main PHASE7_MAX_OPTIMIZATIONS=5 # how many hotspots to attempt -# one dispatch carries the run through all eight phases (Phase 7/8 auto-chain) +# one dispatch carries the run through all eight phases (Phase 7 and 8 auto-chain) gh workflow run codeweave.yml -f dry_run=true # smoke-test the structure first gh workflow run codeweave.yml # then the real run ``` -Each optimization cycle ends in one of five recorded terminal states — only the first is PR-worthy: +Each optimization cycle ends in one of five recorded terminal states. Only the first is worth turning into a pull request: | State | Measured? | Meaning | |-------|-----------|---------| -| **KEEP** | yes | Significant improvement on the point's primary signal → Phase 8 drafts a PR | -| **INVESTIGATE** | yes | Measured but ambiguous → report recommends manual re-measurement, not a PR | -| **REVERT** | yes | A regression, or no detectable effect → not submitted (the conservative default) | -| **FAILED** | no | The change was incorrect (failed the gate) → branch never pushed, never counted as a regression | -| **INCOMPLETE** | no | Built and gated, but the stats couldn't be trusted → excluded, flagged for re-run | +| **KEEP** | yes | A genuine improvement on the point's primary signal, so Phase 8 drafts a pull request | +| **INVESTIGATE** | yes | Measured but ambiguous, so the report recommends a manual re-measurement rather than a pull request | +| **REVERT** | yes | A regression, or no detectable effect, so the change is not submitted (the conservative default) | +| **FAILED** | no | The change was incorrect and failed the gate, so its branch is never pushed and it is never counted as a regression | +| **INCOMPLETE** | no | Built and gated, but the statistics could not be trusted, so it is excluded and flagged for a re-run | -**→ Start here:** [Quickstart](#quickstart) · [Full per-phase docs](docs/index.md) · [Executive summary](executive-summary.md) · [How it works](#how-it-works) +**Start here:** [Quickstart](#quickstart), the [full per-phase docs](docs/index.md), the [executive summary](executive-summary.md), and [how it works](#how-it-works). --- ## Why CodeWeave exists -LLM coding agents are good at *proposing* performance changes and bad at *proving* them. Ask one to speed up a hot path and you get a confident diff and a confident claim — "~10% faster." Verifying that claim is the actual work: is the change even correct on the edge cases? Is the speedup real, or is it thermal drift and cache warmth? Would it survive being measured a second time? Multiply that by dozens of candidates and the verification cost swamps the generation cost. +Software has an energy bill, and most of it hides in a small number of hot paths. Shrinking that bill by hand is slow work: you have to find the paths that matter, propose a change, confirm the change is still correct, and then prove that it actually saved energy rather than merely appearing to. The last step is the hardest, because energy and timing measurements are noisy and a plausible-looking win is often just thermal drift or a warm cache. CodeWeave was built to carry that whole loop autonomously, from finding the hotspot to proving the fix, on any codebase you point it at. + +Coding agents make the first part easy and the last part dangerous. Ask one to speed up a hot path and you get a confident diff and a confident claim, for example that it is roughly ten percent faster. Verifying that claim is the real work. Is the change correct on the edge cases? Is the improvement real, or is it noise? Would it survive a second measurement? Multiply that by dozens of candidates and the cost of verification dwarfs the cost of generation, which is why agent-proposed optimizations so rarely reach production. The bottleneck was never generating ideas; it was trusting them. + +CodeWeave is built on a single conviction: the agent that writes a change must never be the thing that certifies it. Generators propose, and a deterministic, statistically disciplined pipeline disposes. The agent edits source and authors specifications, while the pipeline builds, fuzzes, measures, and rules on the result. The verdict tool `ab_compare.py` is the single source of truth for every number, and the signal that a phase is complete is always a file written by an independent check rather than a claim made by the generator. + +The long-term aim is greener software produced without a human babysitting every experiment. Energy and carbon are measured directly and treated as the ultimate objective, while per-iteration latency serves as the lever precise enough to resolve at the scale of one change. What emerges is an autonomous loop that can walk into an unfamiliar codebase, understand it, improve it under measurement, and leave behind an audit trail a reviewer can actually check. + +--- -So agent-proposed optimizations mostly don't ship. The bottleneck was never generating ideas — it was **trusting them.** +## What CodeWeave takes seriously -CodeWeave is built on one conviction: **the agent that writes a change must never be the thing that certifies it.** Generators propose; a deterministic, statistically rigorous pipeline disposes. The agent edits source and authors specs; the pipeline builds, fuzzes, measures, and rules. `ab_compare.py` is the single source of truth for every number. "Done" is always a file written by an independent check, never a claim. +Three commitments shape the design more than any single feature, and each is deliberate rather than incidental. -The long-term goal is **greener software**. Energy and carbon are measured directly (via CodeCarbon) and treated as the ultimate objective; per-iteration latency is the lever we can resolve precisely enough to act on. The result is an autonomous loop that can walk into an unfamiliar codebase, understand it, and improve it under measurement — leaving behind an audit trail a human reviewer can actually check. +**The statistical proof is embedded on purpose, not bolted on afterward.** A result cannot be reported unless it has survived the statistics, because the statistics are wired into the acceptance path itself. Every comparison runs Welch's t-test and must also clear a Minimum Detectable Effect, a noise floor computed from the baseline's own run-to-run variance, so that a difference which is significant but trivially small cannot pass. Because the workload is a fixed-time loop that pins the wall clock, every verdict uses per-iteration metrics rather than elapsed time. The base build is re-measured in the same cycle as each variant, back to back, so that slow machine drift cancels instead of masquerading as a result. And across a long campaign of many experiments, a Holm-Bonferroni correction demotes any winner that does not survive family-wise control. The point of all of this is that trust is designed in: the pipeline is built so a finding you did not statistically earn simply cannot reach the report. -**Design philosophy, in five lines:** +**The quality of the context it builds is what makes a change worth measuring.** Before it edits a single line, CodeWeave reads the system into an architecture book and a set of Architecture Decision Records, capturing subsystems, ownership boundaries, runtime behavior, and the paths that are sensitive to performance. Every later claim, including the choice of what to optimize and how, is grounded in those documents rather than in a keyhole view of one file. A generate-then-validate loop drives this understanding forward: a generation pass writes, an independent validation pass checks the result against an explicit checklist and records precise gaps, and the next generation pass closes those gaps before extending coverage, so the understanding deepens monotonically instead of churning. Good optimizations begin with a genuine model of the system, and building that model well is treated as part of the engineering, not a preamble to it. -- Generators propose; independent validators dispose. -- The pipeline owns truth and every commit. -- Correctness before performance — always. -- Measure the right thing, and only trust what you can resolve. -- Energy is the goal; latency is the lever. +**The integration tests are reverse-engineered from the real system, not assumed.** The measurement harness and the correctness gate are derived from how the target actually behaves, on the actual toolchain the target uses, discovered and pinned rather than guessed. The differential fuzz compares a changed build against a golden output captured from the base build of the same source tree, so that "correct" means "indistinguishable from the real system's own behavior on inputs that matter." Finding the right tests, the ones that truly exercise the hot path and would catch a regression there, is treated as a first-class problem: the harness specification is authored, validated, and only then turned into runnable code behind a smoke-test gate, so the tests that guard every optimization are ones that reflect the system as it really runs. --- -## Who it's for and what you'd use it for +## Who it is for and what you would use it for -CodeWeave targets people responsible for large, performance-sensitive systems where "make it faster" is a real, recurring job: +CodeWeave is aimed at the people responsible for large, compute-heavy systems where reducing the energy and time a program spends is a real and recurring job. It is designed to be independent of language and stack, reading the target's build and test toolchain from a manifest, so the same pipeline can be pointed at different kinds of codebase. -- **"Find and prove wins in a hot library."** Run the full pipeline against a compute-heavy codebase (the reference target is PyTorch on CPU) and get back a ranked set of PR-ready branches, each with a statistical verdict — plus honest REVERT/INVESTIGATE records for the ideas that didn't pan out. -- **"Understand a codebase I inherited."** Phases 1–4 alone produce an architecture book (+ PDF), per-area Architecture Decision Records committed alongside the code, and a runnable measurement harness — grounded in the actual source, not generic assumptions. -- **"Vet an agent's optimization before I trust it."** The correctness gate + drift-controlled A/B measurement is the review you'd otherwise do by hand for every candidate, run automatically and recorded. -- **"Stand up repeatable performance measurement."** Phases 3–6 give you a fixed-time hot-loop harness, a statistical baseline with a computed noise floor, and a hotspot profile — reusable infrastructure independent of the optimization stage. +The most direct use is to find and prove efficiency wins in a hot codebase. You run the full pipeline and receive a ranked set of pull-request-ready branches, each carrying a statistical verdict and a per-iteration energy and carbon figure, together with honest REVERT and INVESTIGATE records for the ideas that did not pan out. + +A second use is simply to understand a codebase you have inherited. The first four phases on their own produce an architecture book with a PDF, per-area Architecture Decision Records committed alongside the code they describe, and a runnable measurement harness, all grounded in the actual source rather than in generic assumptions about it. + +A third use is to vet an agent's optimization before you trust it. The correctness gate combined with drift-controlled A/B measurement is exactly the review you would otherwise perform by hand for every candidate, except that it runs automatically and records its reasoning. + +A fourth use is to stand up repeatable energy and performance measurement as reusable infrastructure. The measurement phases give you a fixed-time hot-loop harness, a statistical baseline with a computed noise floor, and a hotspot profile, all of which are useful independently of the optimization stage. --- ## Quickstart -CodeWeave runs as **GitHub Actions workflows**, not a local CLI. Phases 5–8 build and measure a native target, so they require a persistent machine. +CodeWeave runs as GitHub Actions workflows rather than as a local command-line tool. The later phases build and measure a native target, so they require a machine that persists state between phases. ### Prerequisites -- A **self-hosted GitHub Actions runner** (`[self-hosted, Linux, X64]`) that persists the built `src/` tree, the editable `integration-test/.venv`, and a warm ccache between phases. `[VERIFY SUPPORTED PLATFORMS — reference target is Linux/x64 + PyTorch CPU]` -- **GitHub Copilot CLI** access (installed automatically by the workflow via `npm i -g @github/copilot`). -- Two fine-grained PATs stored as repository secrets: - - `COPILOT_TOKEN` — authenticates the Copilot CLI. Needs **Copilot user requests: Read** (account permission; no repo permissions). - - `PUSH_TOKEN` — **Contents: Read and write** on the *target* repo only (Phase 2 pushes ADRs; Phase 7 pushes optimization branches). -- Toolchain for the target build. For the default PyTorch/CPU target that means **Python 3.12** specifically (see [`constraints/project.md`](constraints/project.md) for why — newer interpreters lack prebuilt wheels for the harness stack). - -### Steps +You will need a self-hosted GitHub Actions runner (labelled `self-hosted, Linux, X64`) that keeps the built `src/` tree, the editable build environment, and a warm build cache between phases. -1. **Fork this repository.** -2. **Configure the run** in [`.github/codeweave.config`](.github/codeweave.config) — target repo URL/branch, per-phase iteration caps, and per-phase model schedules. Nothing is hardcoded in the workflow logic. -3. **Replace the constraints.** [`constraints/project.md`](constraints/project.md) ships with a sample constraint — swap in your target's real constraints (toolchain versions, build env vars, scope limits). `constraints/harness.md` and `constraints/harness-context.md` are optional target-specific inputs for Phase 3. -4. **Add the two secrets** (`COPILOT_TOKEN`, `PUSH_TOKEN`). -5. **Smoke-test the structure first:** - ```bash - gh workflow run codeweave.yml -f dry_run=true - ``` - `dry_run` skips Copilot invocations and prerequisite checks, so you can confirm the eight-phase wiring and the Phase 7 auto-chain before spending model time. -6. **Run it for real:** - ```bash - gh workflow run codeweave.yml - ``` +You will need access to the GitHub Copilot CLI, which the workflow installs automatically through `npm i -g @github/copilot`. -### The "aha" moment +You will need two fine-grained personal access tokens stored as repository secrets. `COPILOT_TOKEN` authenticates the Copilot CLI and needs only the Copilot user requests Read account permission, with no repository permissions. `PUSH_TOKEN` needs Contents Read and write on the target repository alone, because Phase 2 pushes Architecture Decision Records and Phase 7 pushes optimization branches. -Watch a single dispatch cascade: Phases 1–6 document and baseline the target, then `codeweave.yml` automatically dispatches Phase 7, which optimizes one hotspot, gates it, measures a paired A/B, records a verdict — and dispatches the *next* cycle itself. The last cycle chains into Phase 8, which writes a ranked report and drafts a PR per PR-worthy win. You dispatched once; the pipeline ran an entire measurement campaign and handed you reviewable branches. +You will need the target's own build and test toolchain available on the runner. CodeWeave does not assume a particular language or package manager; you pin the toolchain your target needs in [`constraints/project.md`](constraints/project.md) and the optional [`constraints/harness.md`](constraints/harness.md), and the pipeline reads it from the generated manifest rather than hardcoding it. -Resume from any phase with `-f start_from_phase=N` (1–6). Full per-phase gates, inputs, and outputs: **[`docs/index.md`](docs/index.md)**. +### Steps ---- +Fork this repository. Configure the run in [`.github/codeweave.config`](.github/codeweave.config), which holds the target repository URL and branch, the per-phase iteration caps, and the per-phase model schedules; none of this is hardcoded in the workflow logic. Replace the sample constraint in [`constraints/project.md`](constraints/project.md) with your target's real constraints, such as toolchain versions, build environment variables, and scope limits, and note that `constraints/harness.md` and `constraints/harness-context.md` are optional target-specific inputs for the harness-design phase. Add the two secrets. Then smoke-test the structure before spending model time: -## Core capabilities +```bash +gh workflow run codeweave.yml -f dry_run=true +``` -### Understand an unfamiliar codebase (Phases 1–4) +The dry run skips the Copilot invocations and the prerequisite checks, so you can confirm that the eight phases are wired correctly and that the Phase 7 auto-chain fires. When that looks right, start the real run: -**What it enables:** an architecture book (+ PDF via Pandoc → Typst), per-area ADRs committed into the target's source tree, and a runnable measurement harness — all grounded in the real source. +```bash +gh workflow run codeweave.yml +``` -**Why it matters:** every later measurement decision traces back to a documented claim, so the harness reflects the actual system instead of the model's priors. +### What you should see -**How it stays honest:** each phase is a **generate → validate loop**. A generate pass edits artifacts; a *separate* validate pass checks them against an explicit checklist and is the **only** thing allowed to write the phase's completion marker. The validator writes a *Required Actions* list, the next generator burns that list down first, and durable state files record the high-water mark — so the manuscript expands and deepens across iterations instead of churning. +A single dispatch cascades through the whole system. The first six phases document and baseline the target, after which `codeweave.yml` automatically dispatches Phase 7. Phase 7 optimizes one hotspot, gates it, measures a paired A/B experiment, records a verdict with its energy and carbon figures, and then dispatches the next cycle itself. The final cycle chains into Phase 8, which writes a ranked report and drafts a pull request for each result worth submitting. You dispatched once, and the pipeline ran an entire measurement campaign and handed back reviewable branches. -**Limitation:** requires enough model budget for multiple passes per phase; iteration caps are set per phase in `codeweave.config`. +You can resume from any phase with `-f start_from_phase=N` for N from 1 to 6. The gates, inputs, and outputs of every phase are documented in [`docs/index.md`](docs/index.md). -### Measure with a trustworthy substrate (Phases 5–6) +--- -**What it enables:** the target built *from source* (ccache-backed), plus a statistical baseline (`baseline.json`) and a hotspot profile. +## Core capabilities -**Why it matters:** A/B comparisons compare two builds of the *same* source tree, and the baseline computes a **noise floor** — a coefficient of variation and a Minimum Detectable Effect — that every later verdict is held to. +### Understand an unfamiliar codebase (Phases 1 to 4) -**Key design decision:** the workload is a **fixed-time hot loop**, so wall-clock time carries no signal (a faster build just completes more iterations). **Every verdict uses per-iteration metrics** (`median_iter_ms`, iterations, joules/iter) — never wall clock. +The documentation phases produce an architecture book with a PDF rendered through Pandoc and Typst, a set of per-area Architecture Decision Records committed into the target's own source tree, and a runnable measurement harness, all grounded in the real source. This matters because every later measurement decision traces back to a documented claim, so the harness reflects the actual system rather than the model's priors. -### Optimize under a correctness-first gate (Phase 7) +The mechanism that keeps this honest is a generate-then-validate loop. A generate pass edits the artifacts, and a separate validate pass checks them against an explicit checklist and is the only step permitted to write the phase's completion marker. The validator writes a list of required actions, the next generator clears that list before doing anything else, and durable state files record the high-water mark, so the artifacts deepen across iterations instead of churning. The main cost is model budget: each phase runs several passes, and the iteration caps are set per phase in the configuration. -**What it enables:** one optimization per dispatch — generate the change, incrementally rebuild (with rebuild verification so a zero-compile edit can't slip through), run the **7a–7f correctness gate**, and only *then* measure a paired A/B. +### Measure on a trustworthy substrate (Phases 5 and 6) -**Why it matters:** the change must be provably *correct* — including a blocking differential fuzz against a base-build golden — before any timing run is spent. The A-side base is re-measured **every cycle**, back-to-back with the variant, to cancel machine drift over a long run. +Phase 5 builds the target from source with a warm build cache, and Phase 6 establishes a statistical baseline together with a hotspot profile. Building from source means that later A/B comparisons compare two builds of the same source tree, and the baseline computes a noise floor, expressed as a coefficient of variation and a Minimum Detectable Effect, that every later verdict is held to. -**Two measurement paths:** ops whose end-to-end effect falls below the noise floor by construction are judged on a per-op microbenchmark (`torch.utils.benchmark`); others on end-to-end latency. The verdict logic is path-aware and directional (regressions tested first). +A deliberate design choice sits underneath all of this. The workload is a fixed-time hot loop, so wall-clock time carries no signal: a faster build simply completes more iterations in the same budget. Every verdict therefore uses per-iteration metrics, such as median iteration latency, iteration count, and joules per iteration, and never raw wall clock. -**Limitation:** energy is measured and reported but **does not gate** — CodeCarbon's resolution is too coarse to judge a single optimization, so a latency win that regresses energy still KEEPs and is merely flagged. +### Optimize behind a correctness-first gate (Phase 7) -### Report with campaign-wide rigor (Phase 8) +Phase 7 handles one optimization per dispatch. It generates the change, rebuilds incrementally with a verification step so that a change which compiles nothing cannot slip through, runs the six-stage correctness gate, and only then measures a paired A/B experiment. The change must be provably correct, including a blocking differential fuzz against a golden captured on the base build, before any measurement run is spent on it. The base side is re-measured in every cycle, back to back with the variant, so that slow machine drift over a long run cancels out instead of polluting the comparison. -**What it enables:** a ranked report plus a drafted PR per PR-worthy optimization. +There are two measurement paths, chosen per optimization from the baseline. A point whose end-to-end effect falls below the noise floor by construction is judged on a per-operation microbenchmark using the target's native benchmark timer, while other points are judged on end-to-end latency. The verdict logic is aware of which path applies and tests for regressions first. Energy and carbon are recorded on every change, but they inform rather than gate the verdict, because their resolution is too coarse to arbitrate a single optimization; a latency win that appears to cost energy is kept and flagged rather than silently discarded. -**Why it matters:** running many cycles at `α = 0.05` inflates the odds of at least one false KEEP, so Phase 8 applies a **Holm–Bonferroni correction** across all cycles and demotes KEEPs that don't survive. +### Report with campaign-wide rigor (Phase 8) -**Limitation:** Phase 8 *drafts* PRs; opening them is a deliberate human step. `INVESTIGATE` is a classification, not an action — the system surfaces ambiguous candidates and the reason (`decision_signal`), but never re-measures or investigates on its own. +Phase 8 reads every per-cycle verdict, applies the family-wise correction, ranks the results, summarizes any environment drift, and drafts a pull request for each result worth submitting. Running many cycles at a per-cycle significance of 0.05 inflates the chance of at least one false winner, so the Holm-Bonferroni correction across all cycles demotes any KEEP that does not survive the campaign-wide test. Phase 8 drafts the pull requests; opening them remains a deliberate human step. An INVESTIGATE result is a classification rather than an action: the system surfaces the ambiguous candidate and records why it is ambiguous, but it never re-measures or investigates on its own. --- ## How it works -One `workflow_dispatch` runs eight gated phases. Phases 1–6 run inside `codeweave.yml`; Phases 7 and 8 are dedicated **auto-chaining** workflows — Phase 7 handles one optimization per dispatch and triggers the next, and the last chains into Phase 8. +A single `workflow_dispatch` runs eight gated phases. The first six run inside `codeweave.yml`, while Phases 7 and 8 are dedicated auto-chaining workflows: Phase 7 handles one optimization per dispatch and triggers the next, and the last chains into Phase 8. ```mermaid flowchart TD - start(["workflow_dispatch"]) --> clone["Clone target into src/ · create work branch"] + start(["workflow_dispatch"]) --> clone["Clone target into src/, create work branch"] clone --> P1 - subgraph gen["Phases 1-4 · generate → validate loops"] - P1["1 · Book"] --> P2["2 · ADRs"] --> P3["3 · Harness design"] --> P4["4 · Integration tests"] + subgraph gen["Phases 1 to 4: generate then validate loops"] + P1["1. Book"] --> P2["2. ADRs"] --> P3["3. Harness design"] --> P4["4. Integration tests"] end - P4 --> P5["5 · Build from source (ccache)"] - subgraph meas["Phases 5-8 · measure & optimize"] - P5 --> P6["6 · Statistical baseline → baseline.json"] - P6 --> P7["7 · Optimize one point
gate 7a–7f → paired A/B → verdict"] + P4 --> P5["5. Build from source"] + subgraph meas["Phases 5 to 8: measure and optimize"] + P5 --> P6["6. Statistical baseline to baseline.json"] + P6 --> P7["7. Optimize one point, gate, paired A/B, verdict"] P7 --> more{"more points?"} - more -- "yes · auto-chain N+1" --> P7 - more -- no --> P8["8 · Aggregate report + PR drafts"] + more -- "yes, auto-chain N+1" --> P7 + more -- no --> P8["8. Aggregate report and PR drafts"] end - P8 --> done(["ranked report + PR-worthy branches"]) + P8 --> done(["ranked report and PR-worthy branches"]) ``` -The generate → validate loop is the structural backbone of Phases 1–4: +The generate-then-validate loop is the structural backbone of the first four phases. ```mermaid flowchart TD enter(["enter phase"]) --> del["Delete completion marker"] - del --> genp["Generate pass · Copilot edits artifacts"] - genp --> smoke{"Phase 4 only:
smoke test?"} - smoke -- fail --> report["Write findings · skip validator"] --> iter - smoke -- "pass / N/A" --> val["Validate pass · writes marker on PASS"] + del --> genp["Generate pass: Copilot edits artifacts"] + genp --> smoke{"Phase 4 only: smoke test?"} + smoke -- fail --> report["Write findings, skip validator"] --> iter + smoke -- "pass or N/A" --> val["Validate pass: writes marker on PASS"] val --> marker{"marker present?"} marker -- yes --> exitok(["early exit"]) marker -- no --> iter{"iterations left?"} iter -- yes --> del - iter -- no --> stopmax(["stop — max iterations"]) + iter -- no --> stopmax(["stop, max iterations"]) ``` -**Operating rules that make it dependable:** +Three operating rules make the loop dependable. The pipeline owns every commit: Copilot runs non-interactively and is denied git in every phase except the Phase 7 optimization agent, which works on its own branch and reviews its own diff. Completion markers are deleted before each generate pass, so a stale marker can never short-circuit the next cycle. And the target's toolchain lives in a manifest rather than in the pipeline code: Phase 4 emits `integration-test/harness-manifest.json`, describing the build environment layout, smoke checks, profiler enable-environment, hotspot-report path, and gate commands, so the deterministic pipeline reads the toolchain instead of assuming a particular language or test runner. That manifest is the seam that lets the same engine run against different codebases. -- **The pipeline owns every commit.** Copilot runs non-interactively (`--no-ask-user`) and is denied git in every phase except the Phase 7 optimization agent (which works on its own branch and reviews its diff). -- **Markers are deleted before each generate pass**, so a stale "done" file can never short-circuit the next cycle. -- **Toolchain lives in a manifest, not the code.** Phase 4 emits `integration-test/harness-manifest.json` (venv layout, smoke checks, profiler enable-env, hotspot-report path, gate commands) so the deterministic pipeline reads the target's toolchain instead of hardcoding Python/pytest/venv — the seam intended to carry the engine beyond the PyTorch reference target. - -Full detail — every gate, input, and output per phase — is in **[`docs/`](docs/index.md)** and the **[executive summary](executive-summary.md)**. +The full detail, including every gate, input, and output per phase, lives in [`docs/`](docs/index.md) and in the [executive summary](executive-summary.md). --- -## How it's different +## How it is different -CodeWeave isn't a coding assistant and isn't a benchmark runner — it's the pipeline between them that makes an agent's performance claims trustworthy. +CodeWeave is neither a coding assistant nor a benchmark runner. It is the pipeline between them that turns an agent's efficiency claims into something you can trust. The table below compares it fairly with the two obvious alternatives, and the point is one of discipline rather than a criticism of coding agents; CodeWeave uses one, the GitHub Copilot CLI, as its generator. What differs is what happens to a change after it is written. -| | Ask an agent directly | Hand-roll a benchmark + review | **CodeWeave** | +| | Ask an agent directly | Hand-roll a benchmark and review | CodeWeave | |---|---|---|---| -| Correctness check before measuring | You do it, per change | You do it | Automated 6-stage gate incl. differential fuzz | -| Speedup vs. noise | Agent's word | Manual stats, if any | Welch's t-test **and** measured MDE floor | +| Energy and carbon measured | No | Rarely | On every change, per iteration | +| Correctness check before measuring | You do it, per change | You do it | Automated six-stage gate, including a differential fuzz | +| Improvement versus noise | The agent's word | Manual statistics, if any | Welch's t-test and a measured noise floor | | Machine drift over a long run | Ignored | Manual re-runs | Contemporaneous A/B, base re-measured each cycle | -| False positives across many changes | Unaddressed | Rarely corrected | Holm–Bonferroni across the campaign | -| Output | A diff + a claim | A number you produced | Ranked, gated PR branches + full `proof/` trail | -| Self-certification | The agent says "done" | — | Only an independent validator writes "done" | - -This is a positioning of *discipline*, not a knock on coding agents — CodeWeave uses one (GitHub Copilot CLI) as its generator. The difference is what happens to a change after it's written. +| False positives across many changes | Unaddressed | Rarely corrected | Holm-Bonferroni across the campaign | +| Output | A diff and a claim | A number you produced | Ranked, gated branches with a full `proof/` trail | +| Self-certification | The agent says it is done | Not applicable | Only an independent validator writes "done" | --- ## Project status -CodeWeave is **experimental**. It is a working, end-to-end pipeline that has been built and iterated against a real target (PyTorch, CPU), but it has not been hardened for arbitrary repositories or published with reproducible headline results. +CodeWeave is experimental. It is a working, end-to-end pipeline that has been built and iterated against a large real-world codebase, but it has not yet been exercised across a range of targets and it has not been published with reproducible headline results. + +What works today is the full eight-phase run: the generate-then-validate documentation loops, the source build, the statistical baseline, the correctness-gated optimization cycles with paired A/B verdicts, and the aggregate report with pull-request drafts. + +What is now tested is the verdict engine itself. The code that decides every KEEP, INVESTIGATE, and REVERT, `integration-test/_tools/ab_compare.py`, is covered by a unit and integration suite exercising the decision table, the noise-floor gate, the Holm-Bonferroni correction, and all three command-line modes. The continuous integration workflow at [`.github/workflows/ci.yml`](.github/workflows/ci.yml) runs that suite on every push and pull request and lints the pipeline's own workflow YAML and shell scripts. -- **Working today:** the full eight-phase run against the reference target — generate→validate documentation loops, source build, statistical baseline, correctness-gated optimization cycles with paired A/B verdicts, and the aggregate report with PR drafts. -- **Tested:** the verdict engine (`integration-test/_tools/ab_compare.py`) — the code that decides every KEEP / INVESTIGATE / REVERT — is covered by a unit + integration suite (decision table, noise-floor gate, Holm–Bonferroni correction, and all three CLI modes). CI ([`.github/workflows/ci.yml`](.github/workflows/ci.yml)) runs it on every push and PR and lints the pipeline's workflow YAML and shell scripts. -- **Experimental:** portability beyond the PyTorch/CPU reference target. The manifest seam is designed for it, but other toolchains are unverified. `[VERIFY portability on a second target]` -- **Planned / manual by design:** opening PRs (Phase 8 drafts them); acting on `INVESTIGATE` results (surfaced, never auto-investigated). -- **Known limitations:** - - Requires a **self-hosted, persistent runner** — Phases 6–8 reuse the build and venv in place; there is no ephemeral-runner path. - - Requires GitHub Copilot CLI access and two fine-grained PATs. - - Energy/carbon is **measured and reported but not gating** (resolution too coarse per change). - - Model and CI time cost scales with iteration caps and optimization count. - - No published benchmark results yet. `[ADD BENCHMARK — headline results from a real run]` +What remains experimental is portability across a wide range of targets. The manifest seam is designed to make the engine language- and stack-agnostic, but only a limited set of toolchains has been driven end to end so far. `[VERIFY portability on additional targets]` -> **Maturity warning:** treat CodeWeave as a research-grade automation harness. Review every drafted PR and read the `proof/` trail before shipping anything it produces. +Some behavior is manual by design. Phase 8 drafts pull requests but does not open them, and INVESTIGATE results are surfaced rather than acted on. + +Several limitations are worth stating plainly. The pipeline requires a self-hosted, persistent runner, because the later phases reuse the build and the build environment in place and there is no ephemeral-runner path. It requires GitHub Copilot CLI access and two fine-grained tokens. Energy and carbon are measured and reported but do not gate individual verdicts, because their per-change resolution is too coarse. The model and CI cost scales with the iteration caps and the number of optimizations. And there are no published benchmark results yet. `[ADD BENCHMARK: headline energy and latency results from a real run]` + +> **Maturity note.** Treat CodeWeave as a research-grade automation harness. Review every drafted pull request and read the `proof/` trail before shipping anything it produces. --- -## Documentation, community & trust +## Documentation, community, and trust + +The per-phase reference lives in [`docs/index.md`](docs/index.md), and the design rationale is in [`executive-summary.md`](executive-summary.md). Worked examples of the prompt and constraint files live under [`work/`](work) and [`constraints/`](constraints), and a sample run's artifacts appear in `proof/`, which is created automatically. `[ADD LINK TO A PUBLISHED EXAMPLE RUN]` -- **Documentation:** per-phase reference in [`docs/index.md`](docs/index.md); the design rationale in [`executive-summary.md`](executive-summary.md). -- **Examples:** prompt and constraint files under [`work/`](work) and [`constraints/`](constraints); a sample run's artifacts appear in `proof/` (auto-created). `[ADD LINK TO A PUBLISHED EXAMPLE RUN]` -- **Roadmap:** [`ROADMAP.md`](ROADMAP.md) — near-term focus is publishing a real run and verifying portability to a second target via the manifest seam. -- **Contributing:** [`CONTRIBUTING.md`](CONTRIBUTING.md) — issues and PRs welcome; please include the relevant `proof/` artifacts when reporting pipeline behavior. -- **Support:** open a [GitHub issue](../../issues). `[ADD DISCUSSIONS LINK IF ENABLED]` -- **Security:** the pipeline handles two PATs and pushes branches to a target repo; scope tokens minimally as described in [Prerequisites](#quickstart). Report vulnerabilities via [`SECURITY.md`](SECURITY.md). +The near-term direction is set out in [`ROADMAP.md`](ROADMAP.md), whose current focus is publishing a real run and verifying portability across additional targets through the manifest seam. Contributions are welcome; please read [`CONTRIBUTING.md`](CONTRIBUTING.md) first, and include the relevant `proof/` artifacts when you report pipeline behavior. For support, open a [GitHub issue](../../issues). `[ADD DISCUSSIONS LINK IF ENABLED]` Because the pipeline handles two access tokens and pushes branches to a target repository, please scope the tokens minimally as described under [Prerequisites](#quickstart) and report any vulnerability through [`SECURITY.md`](SECURITY.md). ## License Copyright © 2026 Hightech ICT B.V. -Licensed under the **GNU General Public License v3.0 or later**. See [`LICENSE`](LICENSE). +Licensed under the GNU General Public License v3.0 or later. See [`LICENSE`](LICENSE). diff --git a/ROADMAP.md b/ROADMAP.md index 1698d9f..e7aa20f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,53 +1,51 @@ # CodeWeave Roadmap -CodeWeave is **experimental**. It runs end to end against a PyTorch/CPU reference -target, but it is not yet hardened for arbitrary repositories and has no published -headline results. This roadmap describes the direction, not a dated commitment; -priorities and ordering may change. - -> Have an opinion on ordering, or want to pick something up? Open an issue or see -> [`CONTRIBUTING.md`](CONTRIBUTING.md). - -## Now — proving the core - -The immediate focus is turning "it works against one target" into "it demonstrably -works, and you can see the evidence." - -- **Publish a real run.** Link an example `proof/` trail and a Phase 8 report from a - full eight-phase run, including at least one `KEEP` verdict with its statistics. - _(Fills the `[ADD BENCHMARK]` / `[ADD LINK TO A PUBLISHED EXAMPLE RUN]` gaps in the - README.)_ -- **A demo asset.** A terminal recording or annotated report showing a cycle go - correctness-gate → paired A/B → verdict. - -## Next — portability beyond the reference target - -CodeWeave was built against PyTorch on CPU. The seam for going further already -exists — `integration-test/harness-manifest.json` lets the deterministic pipeline -read a target's toolchain instead of assuming Python/pytest/venv. - -- **Verify a second target end to end** through the manifest seam, and document what - a new target requires. _(Fills the `[VERIFY portability on a second target]` gap.)_ -- **Reduce runner assumptions.** Explore paths that don't require a single persistent - self-hosted runner to carry state across Phases 5–8. - -## Later — sharper science and less manual glue - -- **Act on `INVESTIGATE`.** Today it is a classification, not an action; the pipeline - surfaces ambiguous candidates and a `decision_signal` but never re-measures. A - guarded auto-re-measurement path is a candidate. -- **Optional PR opening.** Phase 8 currently *drafts* PRs; opening them is a - deliberate human step. An opt-in flag to open the drafted PRs directly. -- **Energy as more than a report.** Energy/carbon is measured but does not gate, - because CodeCarbon's per-change resolution is too coarse. Investigate measurement - approaches precise enough to make energy a first-class gating signal, in line with - the project's "energy is the goal" philosophy. -- **Broader statistical options.** Additional verdict models and configurable - multiple-comparison corrections beyond Holm–Bonferroni. - -## Non-goals (for now) - -- Becoming a general-purpose local coding assistant — CodeWeave is a measurement and - verification pipeline that *uses* a coding agent, not a replacement for one. -- Auto-merging changes. A human reviews every drafted PR and the `proof/` trail - before anything ships. +CodeWeave is experimental. It runs end to end against a large real-world codebase, +but it has not yet been exercised across a range of targets and it has no published +headline results. This roadmap describes the direction rather than a dated +commitment, and priorities may change. + +If you have an opinion on ordering, or want to pick something up, open an issue or +read [`CONTRIBUTING.md`](CONTRIBUTING.md). + +## Now: proving the core + +The immediate focus is turning "it works against one codebase" into "it demonstrably +works, and you can see the evidence." That means publishing a real run, with a +linked `proof/` trail and a Phase 8 report that includes at least one KEEP verdict +alongside its statistics and its energy and carbon figures, which fills the benchmark +and example-run gaps flagged in the README. It also means producing a demo asset: a +terminal recording, or an annotated report, showing a cycle move from correctness +gate to paired A/B measurement to verdict. + +## Next: portability across many targets + +CodeWeave is designed to be independent of language and stack. The seam that makes +this possible already exists, because `integration-test/harness-manifest.json` lets +the deterministic pipeline read a target's toolchain instead of assuming a +particular build system, test runner, or benchmark timer. The work here is to drive +additional and materially different targets end to end through that seam, and to +document what a new target requires. A related goal is to reduce the runner +assumptions, exploring paths that do not require a single persistent self-hosted +runner to carry state across the measurement phases. + +## Later: sharper science and less manual glue + +Several improvements would deepen the system once the core is proven. An INVESTIGATE +result is today a classification rather than an action, since the pipeline surfaces +an ambiguous candidate and a reason but never re-measures; a guarded automatic +re-measurement path is a natural extension. Phase 8 currently drafts pull requests +but leaves opening them as a human step, so an opt-in flag to open the drafts +directly is a candidate. Energy is measured but does not gate, because its +per-change resolution is too coarse to arbitrate a single optimization, and finding +a measurement approach precise enough to make energy a first-class gating signal +would align the mechanism fully with the project's stated goal. Finally, additional +verdict models and configurable multiple-comparison corrections beyond +Holm-Bonferroni would give users more control over the statistical treatment. + +## Non-goals for now + +CodeWeave is not trying to become a general-purpose local coding assistant. It is a +measurement and verification pipeline that uses a coding agent, rather than a +replacement for one. It also does not auto-merge changes: a human reviews every +drafted pull request and the `proof/` trail before anything ships. diff --git a/SECURITY.md b/SECURITY.md index 02cfe12..8066cf1 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -22,7 +22,7 @@ When reporting, please include: - The affected component (a workflow, a `work/` prompt, `ab_compare.py`, etc.). - Any suggested remediation. -**Please redact secrets** from anything you attach — never include the values of +**Please redact secrets** from anything you attach, and never include the values of `COPILOT_TOKEN`, `PUSH_TOKEN`, or any other credential. ### What to expect @@ -43,7 +43,7 @@ secrets and scoped as narrowly as possible: | Secret | Scope it needs | Used by | |--------|----------------|---------| -| `COPILOT_TOKEN` | **Copilot user requests: Read** (account permission only — no repository permissions) | Every Copilot phase, as `GH_TOKEN` | +| `COPILOT_TOKEN` | **Copilot user requests: Read** (account permission only, no repository permissions) | Every Copilot phase, as `GH_TOKEN` | | `PUSH_TOKEN` | **Contents: Read and write** on the *target* repository only | Phase 2 (push ADRs) and Phase 7 (push optimization branches) | Guidance: @@ -65,6 +65,6 @@ Guidance: This policy covers the CodeWeave pipeline in this repository (workflows, composite action, prompts, and tooling under `integration-test/_tools/`). Vulnerabilities in -the *target* repositories CodeWeave analyzes, or in third-party dependencies (the -GitHub Copilot CLI, PyTorch, CodeCarbon, etc.), should be reported to their -respective maintainers. +the *target* repositories CodeWeave analyzes, or in third-party dependencies such as +the GitHub Copilot CLI or CodeCarbon, should be reported to their respective +maintainers. From 8e0c7bd89f96bb611b23e60b2b5a42295cbe637c Mon Sep 17 00:00:00 2001 From: WBurggraaf Date: Thu, 23 Jul 2026 12:19:46 +0200 Subject: [PATCH 10/10] docs: add fighting technical debt as an explicit secondary goal The context CodeWeave builds to optimize a system safely (the architecture book, the committed ADRs, the grounded harness) is a durable artifact that outlives the run. Frame this as a second goal in the vision and reframe the 'inherited codebase' use case around recovering effectively-unmaintainable software and paying down technical debt. Co-Authored-By: Claude Opus 4.8 --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 884c64c..0608c47 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,8 @@ CodeWeave is built on a single conviction: the agent that writes a change must n The long-term aim is greener software produced without a human babysitting every experiment. Energy and carbon are measured directly and treated as the ultimate objective, while per-iteration latency serves as the lever precise enough to resolve at the scale of one change. What emerges is an autonomous loop that can walk into an unfamiliar codebase, understand it, improve it under measurement, and leave behind an audit trail a reviewer can actually check. +A second goal follows from the way that loop is built. To optimize a system safely, CodeWeave has to understand it first, and the understanding it produces does not evaporate once the run ends. The architecture book, the per-area Architecture Decision Records committed next to the code they describe, and the grounded measurement harness are durable artifacts that outlive the optimization campaign. Software that had drifted into being effectively unmaintainable, opaque to the people who own it and risky to touch, comes back documented, mapped, and safe to change again. In this sense CodeWeave fights technical debt as it works: the same context it builds to find energy wins is also the context a team needs to maintain the system for years afterward. + --- ## What CodeWeave takes seriously @@ -85,7 +87,7 @@ CodeWeave is aimed at the people responsible for large, compute-heavy systems wh The most direct use is to find and prove efficiency wins in a hot codebase. You run the full pipeline and receive a ranked set of pull-request-ready branches, each carrying a statistical verdict and a per-iteration energy and carbon figure, together with honest REVERT and INVESTIGATE records for the ideas that did not pan out. -A second use is simply to understand a codebase you have inherited. The first four phases on their own produce an architecture book with a PDF, per-area Architecture Decision Records committed alongside the code they describe, and a runnable measurement harness, all grounded in the actual source rather than in generic assumptions about it. +A second use is to recover a codebase that has become hard to maintain. The first four phases on their own produce an architecture book with a PDF, per-area Architecture Decision Records committed alongside the code they describe, and a runnable measurement harness, all grounded in the actual source rather than in generic assumptions about it. For software that had drifted into being effectively unmaintainable, this is a direct way to pay down technical debt: the system comes back documented, mapped, and safe to change, whether or not you go on to run the optimization phases. A third use is to vet an agent's optimization before you trust it. The correctness gate combined with drift-controlled A/B measurement is exactly the review you would otherwise perform by hand for every candidate, except that it runs automatically and records its reasoning.