Skip to content

πŸ—ΊοΈ Quest: Prove document behavior through <Execution>, and spawn xmd only for process claimsΒ #543

Description

@taras

Story

As a maintainer, I want the CLI suites that prove document behavior β€” props, targets, inline roots, discovery patterns, what a run prints β€” to prove it in process through <Execution host="run"> in checked-in Markdown documents, and to spawn xmd only for claims about the process itself, so the suite stops paying a process start for every document-level case and the evidence is a document a reviewer reads.

Today

The Deno suite is 22m07s standalone and 29m30s under the composability battery, which is what reddened main in #538. Test bodies sum to 1,169s; packages/cli is 476s of that across 41 tests (11.6s each), because the tests spawn xmd per case through runCli() β€” 218 spawn sites in all.

Four of those suites prove document behavior, not process behavior:

suite tier runCli sites s
cli/tests/test-target.test.ts DT 35 31
cli/tests/inline-cli.test.ts IE 34 25
cli/tests/targets-cli.test.ts CT 32 27
cli/tests/props-cli.test.ts PC 32 27

~110s at ~0.8s per spawn on Deno (~6.5s per spawn under Node). What they assert β€” which target expanded, which props reached the root, what an inline source rendered, which documents a pattern discovered β€” is observable from a <Test> that runs the child through <Execution host="run"> with <CollectOutput>/<CollectJournal>, against the same production assembly xmd run builds (packages/cli/src/testing-host.ts).

<Execution host="run"> is implemented and tested (testing-execution-host.test.ts, execution-harness.test.ts), but no checked-in .test.md uses it yet; every use is an inline source string inside a TypeScript test. This is its first real use.

Decision

  • Spawning is for process claims. A test spawns xmd only to prove something about the process: argv and option parsing (--pattern with no value, --pattern against a single document, --journal against a directory, --target with a directory), exit status, stdout/stderr routing, signals, the compiled binary relaunching itself, the npm bin.
  • Document claims move to nested-execution documents run by xmd test directly (the runners attach no host profile, so these cannot be hosted scenarios): one checked-in .test.md per tier, each <Test name> tagged with the row it proves, the fixture documents it targets checked in beside it.
  • One row, once. A row moved to Markdown is deleted from the TypeScript suite. A TypeScript case survives only when a mutation probe shows it catches a fault the document does not, and says which.

Scope

  • PC, CT, IE as nested-execution documents; DT reduced to its process rows, the rest moved.
  • The spec tables for those tiers updated so the documents' rows are rows.
  • Measured before and after: standalone test-deno wall clock and the battery's test line, recorded in the PR. Expected β‰ˆ βˆ’90s standalone.

Out of scope

  • The CLI workflow suites (workflow-crash 73s, workflow-fork 67s, workflow-cli 63s, workflow-inspection 47s, workflow-lifecycle-control 12s): their cost is git through exec, starting/killing/resuming xmd workflow processes and when() polling β€” not document behavior. They move in process only with <Execution host="workflow"> (Add <WorkflowRun> and <Execution host="workflow"> to Markdown testsΒ #547), or with a fixture-sharing design measured first.
  • packages/workflow git/PR and durability suites (389s): a real checkout and remote per test. Needs a measurement pass (repo creation vs hazard cost) before any change; hazards stay the unit, one per process death.
  • The placement and pruning rule itself (Quest: Ensure every specification requirement has exactly one testΒ #542). This ticket needs no policy change to start.

Things this will meet

  • Gap: <Execution> is attached only in cli.ts; a document that uses it runs under xmd test, not under a TypeScript runner.
  • Gap: xmd test reads its argument literally β€” a section is --target Section, never #Section (DT30); inside <Execution> a target is a reference and # selects.
  • Gap: colocated documents resolve their siblings first, so fixture documents beside the test are found with no --component-dir; a row about core-default resolution stays in TypeScript.
  • Gap: whether xmd test reports per-test duration β€” check first, since these documents will carry what the CLI suites carried.
  • Hazard: assert the execution outcome and a non-empty result count (Refuse incomplete root-testing activation before a <Test> runsΒ #523), never "no failures".

Acceptance

  • PC, CT, IE have no runCli sites left except rows that are process claims, named as such; DT keeps only process rows.
  • The checked-in documents run green under xmd test packages/cli/tests/<dir> and under the test-deno job; each moved row has a recorded mutation probe the document alone catches.
  • test-deno standalone wall clock and the battery's test line are lower, by the amount recorded in the PR.

Architecture readiness amendment β€” 2026-08-24

This section supersedes the stale performance and sequencing statements above where they conflict.

Owner decisions

Current verification topology

#546 removed the complete runtime suites from composability, so there is no battery test line to compare. #558 split test-deno into weighted shards and made the repository runtime corpus *.test.ts-only. The delivery evidence is therefore:

  1. before/after wall clock for the local whole Deno suite (deno task test);
  2. before/after total Deno runner spend, measured as the sum of the Deno test execution windows;
  3. before/after slowest Deno shard, reported as an observation rather than a gate; and
  4. a fresh hosted weight measurement and updated test-weights.json/partition evidence after the four files' costs move materially.

Evidence placement

Before implementation, the Planner freezes one disposition for every PC, CT, IE and DT row:

  • nested run document β€” observable from a child document through <Execution host="run">;
  • in-process CLI follow-up (Add in-process xmd command execution to Markdown testsΒ #581) β€” argv, explicit environment sources, help, or xmd test multi-document orchestration that the current run profile cannot construct; or
  • real subprocess β€” an OS-process claim such as signals, TTY/pipe behavior, compiled relaunch, executable lookup, or the npm bin.

A moved row appears once in a checked-in Markdown test and is removed from its TypeScript twin after its recorded mutation probe proves the document alone catches it. A retained TypeScript row names the distinct fault it catches.

One thin *.test.ts launcher per tier runs that tier's checked-in Markdown suite once so the documents remain in the repository runtime corpus under Deno, Node and Bun. The launcher is suite infrastructure, not a second proof of any row.

Testing output

Per-test duration is not added here. TestResult remains unchanged; timing is measured at the tier file, whole-suite and runtime-shard boundaries.

Amended acceptance

  • The frozen disposition matrix accounts for every PC, CT, IE and DT row exactly once.
  • PC, CT and IE document rows run through their tier Markdown documents; DT moves only the rows a run child can faithfully observe.
  • Parser, environment, help and directory-orchestration rows stay focused subprocess tests until Add in-process xmd command execution to Markdown testsΒ #581 supplies their supported in-process surface.
  • Each tier's thin TypeScript launcher runs its Markdown suite once and preserves Deno, Node and Bun coverage.
  • Every moved row has a recorded mutation probe showing the Markdown document alone fails for the fault.
  • The PR records the four current-topology measurements above and commits freshly measured weights/partition evidence.
  • This issue changes no public testing authority or TestResult persistence contract.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestquestCoordinating story with dependency-ordered sub-issues

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions