test: property-based testing foundation (fast-check) over parse/print and geometry kernels - #1437
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Size Report
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. |
… kernels (#1413) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Blocking architecture gap: the new property domains duplicate their owners instead of deriving from them. property-arbitraries.ts manually lists selector keys even though parse.ts exports SELECTOR_KEY_NAMES, and REPLAY_SCRIPT_LINE_ARBS uses satisfies Record<string, …>, which accepts any subset. New selector keys or replay command forms can silently escape the suite; specialized production branches such as open, runtime, and record are already absent. Derive selector domains from the parser-owned export and classify the replay grammar from PUBLIC_COMMANDS or an owning replay registry with an exhaustive assertion plus valid templates/explicit waivers. The gesture/platform derivations, bounded runs, real pure boundaries, and lack of test-only seams are otherwise sound. Exact-head Coverage is also red on the inherited ios-system-ui benchmark gap, so this head is not ready. |
…gistries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
e2e4b5c to
7a8fccf
Compare
Fixed in the latest push, and the head is rebased onto latest
Local after rebase: |
thymikee
left a comment
There was a problem hiding this comment.
Re-reviewed at 7a8fccf0607580891ae0e48ec347566465381c0f: the prior domain-ownership gap is fixed. Selector keys derive from SELECTOR_KEY_NAMES with exhaustive value-shape classification, and replay plans are exhaustive over the derived public/internal command catalogs. The PR is mergeable (CLEAN) and all 25 reported checks are green. Ready to merge.
…op the bundled lane-health work (#1414) - Strip scripts/scheduled-lane/* and scheduled-lane-health.yml: that watcher is #1430's own deliverable and collides with PR #1439's implementation of the same lane. What this lane owes (a per-run envelope) moves into scripts/fuzz/envelope.ts. - Rebase onto #1437 and rebuild the generator layer on fast-check: cases come from arbitraries sharing SELECTOR_VALUE_HAZARDS with the property suite, and counterexamples are shrunk, so a failure names a minimal input plus fast-check's seed/path instead of a 20k-char random string. - Route harness.test.ts into the serialized subprocess-stub project. - Drop the AGENT_DEVICE_FUZZ_STARTUP_DELAY_MS test seam: the ready handshake is now proven by a case budget far below real worker startup. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…s, never hangs (#1414) (#1438) * test: nightly parser fuzz lane with typed-AppError invariant (#1414) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fuzz): run envelope, artifact promotion, and harness self-check tests (#1414) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fuzz): shared scheduled-lane envelope on every terminal path, watchdog after ready (#1414) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fuzz): envelope for malformed options; add scheduled-lane health consumer (#1414, #1430) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(lanes): actions:read scope, terminal error envelope, first-due grace (#1414, #1430) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(lanes): anchor first-run grace to schedule registration, use exec helper in tests (#1414, #1430) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(lanes): portable POSIX pickaxe pattern for schedule registration (#1414, #1430) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(fuzz): fast-check generators over the shared hazard list, drop the bundled lane-health work (#1414) - Strip scripts/scheduled-lane/* and scheduled-lane-health.yml: that watcher is #1430's own deliverable and collides with PR #1439's implementation of the same lane. What this lane owes (a per-run envelope) moves into scripts/fuzz/envelope.ts. - Rebase onto #1437 and rebuild the generator layer on fast-check: cases come from arbitraries sharing SELECTOR_VALUE_HAZARDS with the property suite, and counterexamples are shrunk, so a failure names a minimal input plus fast-check's seed/path instead of a 20k-char random string. - Route harness.test.ts into the serialized subprocess-stub project. - Drop the AGENT_DEVICE_FUZZ_STARTUP_DELAY_MS test seam: the ready handshake is now proven by a case budget far below real worker startup. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fuzz): replay the regression corpus through the worker watchdog (#1414) A promoted hang case used to wedge the unit job until the CI timeout, because corpus replay called checkCase in-process. It now goes through the same worker-backed watchdog the nightly lane uses, so such a case fails against a 5s per-case budget; the file moves to the serialized subprocess-stub project with the rest of the worker-driven fuzz tests. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fuzz): let the watchdog outlive vitest's default case timeout (#1414) A wedged parser was surfacing as a bare 'Test timed out in 5000ms' instead of the named hang: failure that says which input wedged, because the file's vitest timeout was shorter than the watchdog budget times the number of replayed cases. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fuzz): complete drift provenance in the lane envelope (#1414) configHash now covers every input that decides what a seed generates (generate.ts and the shared property arbitraries, not just the arbitraries/targets/invariant), and tool records fast-check's installed version. A generation-loop edit or a fast-check upgrade previously changed the case set while the envelope looked unchanged. A test recomputes the hash with each input omitted so a future omission fails. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Summary
Adds
fast-check(dev dep) and turns five hand-pinned example clusters into general properties over the pure parse/print and geometry kernels. Generators live in one shared module behind the test-utils barrel, so a new hazard (another quote shape, a new gesture kind, another.adcommand) is added once and inherited by every property instead of becoming another pinned case.src/__tests__/test-utils/property-arbitraries.ts(exported via the barrel):selectorChainArb— selector chains whose text values include the grammar's hazards:",',it's,say "hi", backslash runs,a || b,key=value, embedded newline/tab, unicode, empty and whitespace-only — plusformatSelectorChainExpression, the canonical printer the round-trip needs (the parser has no production serializer).refArb/formatRef—@eN~sM, with and without@and generation.viewportRectArb+COMPACT_VIEWPORTS(320×568, 375×667) andgestureInViewportArb, keyed byGESTURE_KINDSas aRecord<GestureKind, Arbitrary<GesturePayload>>so a new gesture kind fails to typecheck rather than silently escaping the geometry property.replayScriptArb—.adscripts over the command vocabulary, with comments, blank lines, flags and quoting hazards.rawSnapshotNodesArb— bounded snapshot trees for the diff properties.Properties, each in its owning module's test file:
src/selectors/parse.test.ts(new) — parsing a printed chain yields the terms it was printed from, andparse -> print -> parseis a fixed point.snapshot-ref-grammar.test.ts—splitRefGenerationSuffixand the~sMjoin are inverses for every ref.gesture-plan-viewport.test.ts— for every gesture kind × viewport × platform,buildGesturePlaneither keeps every synthesized point in bounds or refuses with a structuredINVALID_ARGS; never a silently out-of-bounds path. The two compact-iPhone viewports are kept as examples of that property (they areCOMPACT_VIEWPORTS, ready forsmoke-ios-simulator-coverage.test.tson the unmergedagent/ios-simulator-e2e-coveragebranch, which does not exist onmain).replay/__tests__/script.test.ts— re-serializing a parsed script is a fixed point, and command + positionals survive the rewrite (flag defaults are legitimately dropped, so the canonical form is the invariant, not the first parse).daemon/__tests__/snapshot-diff.test.ts—diff(a, a)yields no added/removed line, andsummaryis always exactly a count of the rendered lines:additions + unchanged === countSnapshotComparableLines(current)andremovals + unchanged === …(previous).docs/agents/testing.mdgains a "Properties over examples" section: pure parser or geometry change → extend a property, not another example, plus where the generators live, the boundedPROPERTY_RUNS/PROPERTY_RUNS_SMALLbudget, and how to replay a shrunk counterexample viafc.assert(..., { seed, path }).Closes #1413.
Validation
11 files touched, scope unchanged from the issue (no CLI-arg or exec-layer fuzzing). Local gates green:
format:check,lint,typecheck,check:layering,check:fallow --base origin/main,check:mcp-metadata. All property tests run inunit-coreand land well under the slow-test budget (each property file ≤ 45ms); no new entry in the slow-test report.Two
check:affectedsteps fail identically onorigin/mainin this environment and are unrelated to this change — verified on a cleanorigin/mainworktree:build:tsdowncannot loadtsdown.config.ts(Failed to import module "unrun", an optional peer that is not in the lockfile).scripts/__tests__/help-conformance-topic-coverage.test.ts: uncovered help topicios-system-ui.Everything else in
pnpm test:unitpasses (4557 tests). No device-facing behavior changes: this PR adds tests, generators and docs only, so no simulator/device evidence applies.Link to Devin session: https://app.devin.ai/sessions/fdcb06373f8f4e2fb9558de6f69c2601
Requested by: @thymikee