Skip to content

test: property-based testing foundation (fast-check) over parse/print and geometry kernels - #1437

Merged
thymikee merged 2 commits into
mainfrom
devin/1785158889-property-based-testing
Jul 27, 2026
Merged

test: property-based testing foundation (fast-check) over parse/print and geometry kernels#1437
thymikee merged 2 commits into
mainfrom
devin/1785158889-property-based-testing

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Summary

Adds fast-check (dev dep) and turns five hand-pinned example clusters into general properties over the pure parse/print and geometry kernels. Generators live in one shared module behind the test-utils barrel, so a new hazard (another quote shape, a new gesture kind, another .ad command) is added once and inherited by every property instead of becoming another pinned case.

src/__tests__/test-utils/property-arbitraries.ts (exported via the barrel):

  • selectorChainArb — selector chains whose text values include the grammar's hazards: ", ', it's, say "hi", backslash runs, a || b, key=value, embedded newline/tab, unicode, empty and whitespace-only — plus formatSelectorChainExpression, the canonical printer the round-trip needs (the parser has no production serializer).
  • refArb / formatRef@eN~sM, with and without @ and generation.
  • viewportRectArb + COMPACT_VIEWPORTS (320×568, 375×667) and gestureInViewportArb, keyed by GESTURE_KINDS as a Record<GestureKind, Arbitrary<GesturePayload>> so a new gesture kind fails to typecheck rather than silently escaping the geometry property.
  • replayScriptArb.ad scripts over the command vocabulary, with comments, blank lines, flags and quoting hazards.
  • rawSnapshotNodesArb — bounded snapshot trees for the diff properties.

Properties, each in its owning module's test file:

  1. src/selectors/parse.test.ts (new) — parsing a printed chain yields the terms it was printed from, and parse -> print -> parse is a fixed point.
  2. snapshot-ref-grammar.test.tssplitRefGenerationSuffix and the ~sM join are inverses for every ref.
  3. gesture-plan-viewport.test.ts — for every gesture kind × viewport × platform, buildGesturePlan either keeps every synthesized point in bounds or refuses with a structured INVALID_ARGS; never a silently out-of-bounds path. The two compact-iPhone viewports are kept as examples of that property (they are COMPACT_VIEWPORTS, ready for smoke-ios-simulator-coverage.test.ts on the unmerged agent/ios-simulator-e2e-coverage branch, which does not exist on main).
  4. replay/__tests__/script.test.ts — re-serializing a parsed script is a fixed point, and command + positionals survive the rewrite (flag defaults are legitimately dropped, so the canonical form is the invariant, not the first parse).
  5. daemon/__tests__/snapshot-diff.test.tsdiff(a, a) yields no added/removed line, and summary is always exactly a count of the rendered lines: additions + unchanged === countSnapshotComparableLines(current) and removals + unchanged === …(previous).

docs/agents/testing.md gains a "Properties over examples" section: pure parser or geometry change → extend a property, not another example, plus where the generators live, the bounded PROPERTY_RUNS / PROPERTY_RUNS_SMALL budget, and how to replay a shrunk counterexample via fc.assert(..., { seed, path }).

Closes #1413.

Validation

11 files touched, scope unchanged from the issue (no CLI-arg or exec-layer fuzzing). Local gates green: format:check, lint, typecheck, check:layering, check:fallow --base origin/main, check:mcp-metadata. All property tests run in unit-core and land well under the slow-test budget (each property file ≤ 45ms); no new entry in the slow-test report.

Two check:affected steps fail identically on origin/main in this environment and are unrelated to this change — verified on a clean origin/main worktree:

  • build: tsdown cannot load tsdown.config.ts (Failed to import module "unrun", an optional peer that is not in the lockfile).
  • scripts/__tests__/help-conformance-topic-coverage.test.ts: uncovered help topic ios-system-ui.

Everything else in pnpm test:unit passes (4557 tests). No device-facing behavior changes: this PR adds tests, generators and docs only, so no simulator/device evidence applies.

Link to Devin session: https://app.devin.ai/sessions/fdcb06373f8f4e2fb9558de6f69c2601
Requested by: @thymikee

@thymikee thymikee self-assigned this Jul 27, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-27 15:15 UTC

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.87 MB 1.87 MB 0 B
JS gzip 598.4 kB 598.4 kB 0 B
npm tarball 713.2 kB 713.2 kB +11 B
npm unpacked 2.49 MB 2.49 MB +28 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 29.7 ms 29.5 ms -0.2 ms
CLI --help 61.6 ms 60.7 ms -0.9 ms

Top changed chunks: no changes in the largest emitted chunks.

… kernels (#1413)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@thymikee

Copy link
Copy Markdown
Member

Blocking architecture gap: the new property domains duplicate their owners instead of deriving from them. property-arbitraries.ts manually lists selector keys even though parse.ts exports SELECTOR_KEY_NAMES, and REPLAY_SCRIPT_LINE_ARBS uses satisfies Record<string, …>, which accepts any subset. New selector keys or replay command forms can silently escape the suite; specialized production branches such as open, runtime, and record are already absent. Derive selector domains from the parser-owned export and classify the replay grammar from PUBLIC_COMMANDS or an owning replay registry with an exhaustive assertion plus valid templates/explicit waivers. The gesture/platform derivations, bounded runs, real pure boundaries, and lack of test-only seams are otherwise sound. Exact-head Coverage is also red on the inherited ios-system-ui benchmark gap, so this head is not ready.

…gistries

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot force-pushed the devin/1785158889-property-based-testing branch from e2e4b5c to 7a8fccf Compare July 27, 2026 14:19
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

the new property domains duplicate their owners instead of deriving from them

Fixed in the latest push, and the head is rebased onto latest main (the ios-system-ui benchmark gap is fixed there, so Coverage is no longer red for that reason).

  • Selectors: keys come from the parser-owned SELECTOR_KEY_NAMES; the generator only classifies the value shape per key via SELECTOR_KEY_VALUE_KINDS satisfies Record<SelectorKey, 'text' | 'boolean'>, so a new SelectorKey fails to typecheck here. A runtime guard also names the resolving file if a key in SELECTOR_KEY_NAMES has no shape.
  • Replay grammar: REPLAY_SCRIPT_LINE_ARBS is replaced by REPLAY_SCRIPT_LINE_PLANS satisfies Record<ReplayCommandName, ReplayLinePlan>, where ReplayCommandName is the union of PUBLIC_COMMANDS and INTERNAL_COMMANDS (runtime is internal, so public alone would have missed it) and each entry is either a valid line template or an explicit { waived: reason }. The specialized branches you named are now templated — open (positional + --relaunch + runtime hints), runtime (--platform), record (--fps, --hide-touches) — alongside snapshot/screenshot/click/press/longpress/wait/fill/type/get/swipe/gesture. Waived commands carry the reason "generic line shape, covered by the wait/longpress templates", i.e. they have no branch in script.ts/script-formatting.ts; a command that grows one has to move to a template. The line set is materialized by iterating the runtime catalogs, so a command present at runtime but absent from the plan throws a guard message naming REPLAY_SCRIPT_LINE_PLANS and its file.

Local after rebase: format:check, lint, typecheck, check:layering, check:fallow --base origin/main clean; pnpm test:unit 4563 passed / 0 failed, no new slow-test entries.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 27, 2026

@thymikee thymikee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 7a8fccf0607580891ae0e48ec347566465381c0f: the prior domain-ownership gap is fixed. Selector keys derive from SELECTOR_KEY_NAMES with exhaustive value-shape classification, and replay plans are exhaustive over the derived public/internal command catalogs. The PR is mergeable (CLEAN) and all 25 reported checks are green. Ready to merge.

@thymikee
thymikee merged commit 8670a88 into main Jul 27, 2026
25 checks passed
@thymikee
thymikee deleted the devin/1785158889-property-based-testing branch July 27, 2026 15:15
devin-ai-integration Bot added a commit that referenced this pull request Jul 28, 2026
…op the bundled lane-health work (#1414)

- Strip scripts/scheduled-lane/* and scheduled-lane-health.yml: that watcher is #1430's own
  deliverable and collides with PR #1439's implementation of the same lane. What this lane owes
  (a per-run envelope) moves into scripts/fuzz/envelope.ts.
- Rebase onto #1437 and rebuild the generator layer on fast-check: cases come from arbitraries
  sharing SELECTOR_VALUE_HAZARDS with the property suite, and counterexamples are shrunk, so a
  failure names a minimal input plus fast-check's seed/path instead of a 20k-char random string.
- Route harness.test.ts into the serialized subprocess-stub project.
- Drop the AGENT_DEVICE_FUZZ_STARTUP_DELAY_MS test seam: the ready handshake is now proven by a
  case budget far below real worker startup.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
thymikee added a commit that referenced this pull request Jul 28, 2026
…s, never hangs (#1414) (#1438)

* test: nightly parser fuzz lane with typed-AppError invariant (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): run envelope, artifact promotion, and harness self-check tests (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): shared scheduled-lane envelope on every terminal path, watchdog after ready (#1414)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): envelope for malformed options; add scheduled-lane health consumer (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): actions:read scope, terminal error envelope, first-due grace (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): anchor first-run grace to schedule registration, use exec helper in tests (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lanes): portable POSIX pickaxe pattern for schedule registration (#1414, #1430)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(fuzz): fast-check generators over the shared hazard list, drop the bundled lane-health work (#1414)

- Strip scripts/scheduled-lane/* and scheduled-lane-health.yml: that watcher is #1430's own
  deliverable and collides with PR #1439's implementation of the same lane. What this lane owes
  (a per-run envelope) moves into scripts/fuzz/envelope.ts.
- Rebase onto #1437 and rebuild the generator layer on fast-check: cases come from arbitraries
  sharing SELECTOR_VALUE_HAZARDS with the property suite, and counterexamples are shrunk, so a
  failure names a minimal input plus fast-check's seed/path instead of a 20k-char random string.
- Route harness.test.ts into the serialized subprocess-stub project.
- Drop the AGENT_DEVICE_FUZZ_STARTUP_DELAY_MS test seam: the ready handshake is now proven by a
  case budget far below real worker startup.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): replay the regression corpus through the worker watchdog (#1414)

A promoted hang case used to wedge the unit job until the CI timeout, because corpus replay called
checkCase in-process. It now goes through the same worker-backed watchdog the nightly lane uses, so
such a case fails against a 5s per-case budget; the file moves to the serialized subprocess-stub
project with the rest of the worker-driven fuzz tests.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): let the watchdog outlive vitest's default case timeout (#1414)

A wedged parser was surfacing as a bare 'Test timed out in 5000ms' instead of the named hang:
failure that says which input wedged, because the file's vitest timeout was shorter than the
watchdog budget times the number of replayed cases.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fuzz): complete drift provenance in the lane envelope (#1414)

configHash now covers every input that decides what a seed generates (generate.ts and the shared property arbitraries, not just the arbitraries/targets/invariant), and tool records fast-check's installed version. A generation-loop edit or a fast-check upgrade previously changed the case set while the envelope looked unchanged. A test recomputes the hash with each input omitted so a future omission fails.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: property-based testing foundation (fast-check) over parsing/geometry kernels

1 participant