A palette that inverted everything except the scrollbars, a deny list that read a quoted pipe as an operator, and a fourth submitter the prompt never credited

.github/workflows/ci.yml

@@ -67,10 +67,31 @@ jobs:
         # and therefore passes or fails on the time of day, not on the code. It
         # is deselected here rather than deleted so that it still runs locally,
         # where a human can tell the difference.
+        #
+        # `tests/property` runs in here with everything else -- it is inside
+        # `tests/` and needs no separate invocation. What it does *not* get from
+        # this leg is a different seed, which is the next step's job.
         run: |
           python -m pytest -q \
             --deselect tests/test_wakeup.py::test_the_deadline_is_reported_as_a_deadline
 
+      - name: Generated tests, with seeds a developer never runs
+        shell: bash
+        # The local profile is `derandomize=True` on purpose: a property suite
+        # that fails one time in five and passes when you re-run it teaches
+        # people to re-run it. That makes every local run explore the *same*
+        # examples, which is exactly the wrong trade for CI -- so this leg turns
+        # randomisation back on and raises the count, and a rare counterexample
+        # surfaces on a pull request rather than never.
+        #
+        # Separate from the step above rather than folded into it, because a
+        # failure here means something different: not "this push broke a rule"
+        # but "a rule was already breakable and this seed found it". Both are
+        # worth failing on; only one of them is a regression.
+        env:
+          HYPOTHESIS_PROFILE: ci
+        run: python -m pytest tests/property -q
+
   # --------------------------------------------------------------------------
   # What a user actually installs
   # --------------------------------------------------------------------------

.gitignore

@@ -4,6 +4,20 @@ __pycache__/
 venv/
 *.egg-info/
 .pytest_cache/
+.ruff_cache/
+
+# Hypothesis's example database: the counterexamples it has found, so a
+# property that failed once is re-checked with that input first on the next run.
+# Local and machine-specific -- the properties themselves are the artefact, and
+# `derandomize` is what makes a run reproducible without shipping this.
+# `.hypothesis/patches/` is the "here is the failing case as a diff" offer,
+# which is a suggestion rather than a result.
+.hypothesis/
+
+# mutmut copies the whole source tree here and runs pytest in it several hundred
+# times. Regenerated by `mutmut run`, and large.
+mutants/
+.mutmut-cache
 # Build outputs. Nothing here builds a wheel in the normal course of things --
 # the install is editable and `grad --update` moves the checkout rather than
 # reinstalling from an artifact -- but the packaging metadata is only really

CONTRIBUTING.md

@@ -75,6 +75,15 @@ What the CI checks, and why it is shaped the way it is:
   resolves whether or not the wheel contains it. If you change
   `[tool.setuptools]`, that job is the one to watch.
 
+There is a second suite inside the first. `tests/property/` generates its inputs
+with Hypothesis instead of listing them, and it is where a rule goes when the
+examples keep running out — `hooks._segments` had four bugs in four commits and a
+fifth that no example had reached. It runs with everything else and takes about a
+second; `HYPOTHESIS_PROFILE=deep` turns it up when you have just changed one of
+the modules it covers. Mutation testing is configured too, and is not in CI.
+Both are in [`docs/testing.md`](docs/testing.md), including the two ways a
+generated test can silently pass on the previous example's leftovers.
+
 Some conventions worth knowing before a larger change:
 
 - **Capability is a CLI, not a framework.** New agent-facing capability is a

README.md

@@ -290,15 +290,24 @@ shapes of the paths that have run, and they fail with actionable errors rather
 than tracebacks — but a real credential and a real run are what find the
 mismatches.
 
-The test suite is 56 files and runs offline; the network is stubbed by a fixture,
-because a suite that reaches the network does not fail, it hangs. The gate tests
-run against a real ledger in a temporary workspace, since a mock of a gate proves
-nothing about the gate.
+The test suite runs offline; the network is stubbed by a fixture, because a suite
+that reaches the network does not fail, it hangs. The gate tests run against a
+real ledger in a temporary workspace, since a mock of a gate proves nothing about
+the gate.
 
 ```bash
 python -m pytest -q
 ```
 
+Alongside the example-based tests, `tests/property/` generates inputs and checks
+rules rather than outputs — a mean lies between the extremes it was taken over, a
+rolling spend never falls when a run is submitted, and if the shell would run
+`ssh` then the deny list says so. That last one found three bypasses on its first
+run, including `( ssh gpu-box nvidia-smi )`: three tokens, no quoting, and the
+shortest hole the hook ever had. Mutation testing (`mutmut`) is configured for
+the same modules and run by hand rather than in CI. Both are described in
+[`docs/testing.md`](docs/testing.md).
+
 Three things to know before trusting it with a budget:
 
 - **Interfaces are not stable.** Ledger fields, exit codes and CLI flags still

core/stats.py

@@ -96,7 +96,25 @@ def summarise(values: Sequence[Any]) -> dict[str, Any]:
     if n == 0:
         return {"n": 0, "mean": None, "sd": None, "sem": None,
                 "ci95": None, "min": None, "max": None, "samples": []}
-    mean = math.fsum(samples) / n
+    low, high = min(samples), max(samples)
+    # Clamped into the range it was taken over, which the true mean of any set
+    # of reals is always inside. `fsum` gives the exactly-rounded sum and the
+    # division then rounds once more, and that last rounding can land outside --
+    # so `[3.05, 3.05, 3.05]` reported a mean of 3.0499999999999994, below its
+    # own minimum, and a standard deviation of 5e-16 for three runs that agreed
+    # exactly.
+    #
+    # Both halves of that matter and the second one more. This module exists
+    # because "`val_loss = 3.05` against a predicted `[2.9, 3.2]` was recorded as
+    # in-range with identical confidence whether the run-to-run spread was 0.001
+    # or 0.3" -- so a spread of *zero*, which is what three identical seeds
+    # measured, is precisely the reading it must not get wrong. Clamping fixes
+    # the deviation too rather than only the mean: with the mean exact, every
+    # `x - mean` is exactly 0 and the variance is exactly 0.
+    #
+    # The correction is never more than one unit in the last place. Found by
+    # `tests/property/test_prop_stats.py`, which asserts the identity directly.
+    mean = min(high, max(low, math.fsum(samples) / n))
     if n == 1:
         return {"n": 1, "mean": mean, "sd": None, "sem": None, "ci95": None,
                 "min": samples[0], "max": samples[0], "samples": samples}
@@ -112,8 +130,8 @@ def summarise(values: Sequence[Any]) -> dict[str, Any]:
         "sd": sd,
         "sem": sem,
         "ci95": [mean - half, mean + half],
-        "min": min(samples),
-        "max": max(samples),
+        "min": low,
+        "max": high,
         "samples": samples,
     }
 

docs/testing.md

@@ -0,0 +1,130 @@
+# Testing
+
+Three suites, answering three different questions. They are not tiers of the
+same thing and none of them replaces another.
+
+| Suite | Question | Cost | Runs in CI |
+| --- | --- | --- | --- |
+| `tests/*.py` | does this input produce that output? | ~3 min | yes, every push |
+| `tests/property/` | is there *any* input that breaks the rule? | ~5 s | yes, every push |
+| `mutmut` | would any test notice if this line were wrong? | CPU-hours | no, by hand |
+
+```bash
+python -m pytest -q
+```
+
+That is still the command. The generated suite is inside `tests/`, so it runs
+with everything else and needs no separate invocation.
+
+## Why the second suite exists
+
+The four commits before `tests/property/` was written were four bugs in one
+function — `hooks._segments`, the quote-aware splitter that decides whether a
+shell command reaches the deny list. Each was found by a person typing one more
+string, and each got an example-based test recording the string that found it.
+
+That is a good record and a bad search. `tests/test_hooks.py` now pins nineteen
+command lines; the shell accepts infinitely many, and the fifth bug was not in
+the nineteen. It was `( ssh gpu-box nvidia-smi )` — three tokens, no quoting, no
+substitution, and the shortest bypass the deny list ever had.
+
+So `tests/property/shellgrammar.py` builds command lines from a grammar instead,
+and carries the answer alongside the text: every node knows which heads the
+shell would execute in it. The property is then one line — if the shell runs
+`ssh`, the hook says so — and Hypothesis searches for a counterexample rather
+than waiting for one to be reported. It found three in the first run:
+
+- `( cmd )` and `{ cmd; }` were never read as starting a command;
+- `if`, `then`, `do`, `else`, `!` and `time` were read *as* the command they
+  introduce, so `for h in a b; do ssh $h; done` had a head of `do`;
+- `rm --recursive -f` matched none of the six alternations in the `rm -rf` rule,
+  which covered short-with-short and long-with-long and no mixed pair.
+
+The other five modules under `tests/property/` are chosen on the same basis:
+pure functions of their arguments, deciding something irreversible, where the
+answer is constrained by an identity rather than by an example. A mean lies
+between the extremes it was taken over; a rolling spend never falls when a run
+is submitted; a document hashes the same after a round trip through the archive.
+
+### Profiles
+
+```bash
+HYPOTHESIS_PROFILE=deep python -m pytest tests/property -q
+```
+
+- `dev` (default) — 50 examples, `derandomize=True`. About a second. A property
+  suite that fails one time in five and passes when you re-run it teaches people
+  to re-run it.
+- `ci` — 300 examples, random seeds, so CI explores what a developer never will.
+- `deep` — 2000 examples. Worth running deliberately against a module that has
+  just changed.
+
+### Writing one
+
+Two rules, both learned the hard way in this directory:
+
+**Fixtures come first in the signature.** `@given` binds positional strategies
+to the *trailing* parameters, so `def test(rows, tmp_path)` hands the strategy to
+`tmp_path` and asks pytest for a fixture called `rows`.
+
+**A function-scoped fixture is set up once and shared by every example.** The
+health check that says so is suppressed in `tests/property/conftest.py`, because
+most properties here are pure — but anything that writes needs its own isolation
+per example, or example 2 reads example 1's ledger. `test_prop_jsonl.py` uses a
+module-level counter for a fresh file; `test_prop_ceilings.py` uses the
+`fresh_workspace` fixture, which re-points `GRAD_ROOT` at a new directory. Both
+of those exist because the first version of each test silently passed on
+leftovers — a round trip that appended six records and read back 242.
+
+## Mutation testing
+
+Coverage says a line ran. It does not say anything would have failed if the line
+were different, and those turn out to be very different questions: a line
+executed by twenty tests that all assert on something else is covered and
+unprotected.
+
+`mutmut` changes one line at a time and runs the tests. A mutant that survives is
+a change to the source that no test objected to.
+
+```bash
+mutmut run
+mutmut results
+mutmut show <mutant>
+```
+
+Configured in `pyproject.toml` under `[tool.mutmut]`, deliberately narrow:
+`source_paths` is eight modules, not the project. Mutation testing costs about
+one test run per mutant, so the useful version of it is aimed rather than
+sprayed. These eight are the pure ones, deciding the irreversible things — what
+gets denied, what a run measured, what is written to the ledger — and they are
+the ones `tests/property/` already covers, which is what makes a surviving
+mutant a finding rather than a to-do. Widen it one module at a time, when that
+module is what changed.
+
+It is **not** in CI. Hours per run is a thing to spend on a module you are
+changing, not on every push.
+
+### On Windows
+
+mutmut refuses to run natively on Windows ([mutmut#397]) — it forks, and Windows
+has no fork. Use WSL:
+
+```bash
+wsl -d <distro>
+python3 -m venv ~/gradmut/.venv
+~/gradmut/.venv/bin/pip install -e ".[dev]"
+cd /path/to/checkout && ~/gradmut/.venv/bin/python -m mutmut run
+```
+
+Working from a copy inside the WSL filesystem rather than over `/mnt/d` is worth
+the `tar` — mutmut copies the whole tree into `mutants/` and then runs pytest in
+it several hundred times, and 9p is not the filesystem for that.
+
+[mutmut#397]: https://github.com/boxed/mutmut/issues/397
+
+## The three tests that fail for environmental reasons
+
+Unchanged, and documented in [`CONTRIBUTING.md`](../CONTRIBUTING.md): two lock
+tests in `tests/test_desktop_app.py` fail if a real Grad is running, and
+`tests/test_wakeup.py::test_the_deadline_is_reported_as_a_deadline` depends on
+the time of day. CI deselects the third and holds nothing.