From aa81b805789702b6eeb9d5a35636bdd117e7e69d Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Wed, 2 Sep 2026 17:07:20 -0400 Subject: [PATCH 1/6] =?UTF-8?q?=E2=9C=A8=20Make=20``=20a=20text=20co?= =?UTF-8?q?mponent=20that=20produces=20approved=20source=20(#722)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `` no longer declares a return. Its rendering is the approved program source: bare, it emits those exact bytes where the component is written; with `as`, ordinary text capture binds them and nothing is emitted. Neither form evaluates the source. Exact bytes are not prose, so a declaring host may say that what its component renders is source. Segments such a component produced carry that fact to the emission loop, which is outside every scope the invocation owned, and the Output Api carries it to the presentation middleware — whitespace normalization and terminal formatting both pass exact writes through untouched while the prose around them is presented as before. A tenth draft that still fails its check after its repairs no longer opens a review. There is no decision left to offer, so the workflow asks the coding agent once, automatically, why the attempts did not work, treats that answer as inert text, and ends without a Plan. The authored exhaustion ending went with the review it belonged to. One Plan-artifact record retains the invocation identity, the instruction identity, the approved source, its digest and the successful admission together, replacing the per-candidate admission record. The instruction identity is retained again in the frozen inputs, so a continuation whose instructions render differently is refused there — as stale input, before a directory, a provider, a turn or a review exists. Claude-Session: https://claude.ai/code/session_01DNPFLq52HVSu3NH6WczYLf --- architecture.md | 2 +- packages/cli/src/documents/Plan.md | 94 ++++--- packages/cli/src/plan-component.ts | 198 +++++++++++-- .../tests/document-suites/plan/Plan.test.md | 163 +++++++++++ .../plan/agents/exhausted-plan.md | 169 +++++++++++ .../plan/agents/revised-plan.md | 36 +++ .../plan/agents/whitespace-plan.md | 17 ++ .../document-suites/plan/captures-plan.md | 11 + .../tests/document-suites/plan/emits-plan.md | 7 + packages/cli/tests/packaged-document.test.ts | 26 +- packages/cli/tests/plan-component.test.ts | 263 +++++++++++++++++- packages/cli/tests/plan.test.ts | 51 ++-- packages/core/src/api.ts | 11 +- .../core/src/components/declared-markdown.ts | 15 +- packages/core/src/execute.ts | 59 +++- packages/core/src/expand.ts | 21 ++ packages/core/src/output/normalize.ts | 16 +- packages/core/src/output/terminal.ts | 11 +- packages/core/src/types.ts | 18 ++ packages/core/tests/output-normalize.test.ts | 35 ++- specs/executable-mdx-spec.md | 66 ++++- specs/plan-command-spec.md | 46 +-- 22 files changed, 1173 insertions(+), 162 deletions(-) create mode 100644 packages/cli/tests/document-suites/plan/agents/exhausted-plan.md create mode 100644 packages/cli/tests/document-suites/plan/agents/revised-plan.md create mode 100644 packages/cli/tests/document-suites/plan/agents/whitespace-plan.md create mode 100644 packages/cli/tests/document-suites/plan/captures-plan.md create mode 100644 packages/cli/tests/document-suites/plan/emits-plan.md diff --git a/architecture.md b/architecture.md index 64b76421f..170f17c26 100644 --- a/architecture.md +++ b/architecture.md @@ -43,7 +43,7 @@ Existing documents and code get aligned to this section retroactively. | Prompt | a person's original request, in ordinary natural language. `xmd plan` takes exactly one | | Plan | the executable program produced from a Prompt: an Executable Markdown document combining readable prose that expresses the Prompt's intent with the components that carry it out, each placed beside the prose describing what it does. It begins with one descriptive level-one heading. A Plan is what `xmd plan` approves and then delivers: printed to stdout by default, written to an `--output` file, and run only under `--run`. It is not a synonym for a workflow, a policy document or any executable Markdown file | | plan command document | the one exact checked-in first-party Markdown value root `xmd plan` executes. It is the command's adapter and nothing else: it projects the request into ``, supplies the session, and returns the approved source. It is not itself a Plan. Internal: no command-line option selects another one, and no repository component search can answer for it | -| packaged `` Component | the one exact checked-in first-party Markdown value component that converts a Prompt into a Plan, `packages/cli/src/documents/Plan.md`, declared to every ordinary run as the public ``. It owns and implements the Plan authorship workflow — the Prompt wording, the draft and repair loops, the `` branches, human review, revision, approval, stopping, exhaustion and the final explanation turn — and returns the exact approved Plan source. Every Plan-producing turn in it states the complete Plan requirements for itself, so a replacement may add or correct a title rather than only carry one forward. Both surfaces expand these exact bytes under one origin and one digest; there is no generated TypeScript copy and no second Markdown implementation. Its four phase components are private to it, and it is not itself a Plan | +| packaged `` Component | the one exact checked-in first-party Markdown text component that converts a prompt into a Plan, `packages/cli/src/documents/Plan.md`, declared to every ordinary run as the public ``. It owns and implements the Plan authorship workflow — the Prompt wording, the draft and repair loops, the `` branches, human review, revision, approval, stopping, exhaustion and the automatic final explanation turn — and produces the exact approved Plan source as what it renders: written bare it emits those bytes, and `as` is ordinary text capture that binds them and emits nothing. Neither form evaluates the source. Every Plan-producing turn in it states the complete Plan requirements for itself, so a replacement may add or correct a title rather than only carry one forward. Both surfaces expand these exact bytes under one origin and one digest; there is no generated TypeScript copy and no second Markdown implementation. Its four phase components are private to it, and it is not itself a Plan | | authorship profile | the trusted-host assembly the packaged `` Component runs its authored turns under, installed by its own `` inside the invocation that owns it rather than around an execution — which is what makes it the same frame whether `xmd plan`, an ordinary document, or a configured `` run child asked. Which Agent context goes under it is a trusted-host capability the declaration carries — the production ACPX one built from the run's Agent stack, or the deterministic one a canonical `` child declaration produced — and a host that supplies none states the sentence a `` written there is refused with. The fixed policy is installed in one place for both, so a second provider cannot bring a weaker one: its fixed inputs, a constrained Agent provider, Elicitation, the fixed first-party components and the host-declared ``. It uses no repository component search and exposes no custom root, and the policy it installs is not readable from the command line. Its working directory is one host-owned directory dedicated to the logical session, keyed by the digest of that name, created empty and required to be empty on the way in. An explicitly named session's directory is durable, because continuation derives the same session identity from it; an invocation-unique default session's is scope-owned, claimed before it is created, and exactly one cleanup is attempted after profile teardown and before admission on every ending — the leaf removed non-recursively when it is still the empty directory that was handed over, and left as found with the command failing terminally when it has gained content or vanished. Where those directories live is a host dependency no caller or document selects | | upgrade command document | the one exact checked-in first-party Markdown streaming text root `xmd upgrade` executes to select and install a published release. It owns the exact-tag grammar, release selection, semantic-version comparison, consent, the status, already-current and installation branches, and the wording of every refusal and report; its rendered body is the command's output rather than a value it returns. Internal: no command-line option selects another one, and no repository component search can answer for it | | upgrade assembly | what one runtime-named entrypoint states about the `xmd` that is running: its provenance, reported version, invoked executable path, platform, architecture, release target when the release publishes one, and — for an eligible compiled macOS or Linux host alone — the factory for the four phases an installation needs. It describes how this `xmd` is running, never how its files arrived | diff --git a/packages/cli/src/documents/Plan.md b/packages/cli/src/documents/Plan.md index 6b35dc0fe..40b07851d 100644 --- a/packages/cli/src/documents/Plan.md +++ b/packages/cli/src/documents/Plan.md @@ -4,13 +4,11 @@ props: properties: session: { type: string, minLength: 1 } additionalProperties: false -returns: - type: string description: >- - Create an XMD program from a Prompt. `Ask for the user's - age.` expands its content normally to form the complete Prompt. -as: Required. The exact approved Plan source after teardown and structural admission. -context: The complete Prompt, expanded once before authorship. + Create an XMD program from a prompt. `Ask for the user's age.` + emits the approved program source. +as: Binds the exact approved program source and emits nothing. Omitted, that source is emitted where the component is written. +context: The complete prompt, expanded once before authorship. --- # Turning a Prompt into a Plan @@ -21,8 +19,9 @@ available to carry them out. A coding agent turns both into one document that explains and executes the sequence. A draft remains text while this workflow reviews it. Nothing in it runs before -you approve it. After approval, the exact source is checked once more as -structure and handed back unchanged. +you approve it, and nothing runs it afterwards either: what this workflow +produces is the approved program's source, byte for byte, for you to keep, +read, or hand to something that runs it. ## Read the Prompt @@ -37,7 +36,7 @@ nothing reaches no catalog, no session, no agent and no review. - + ## Say what this surface calls things @@ -48,13 +47,11 @@ of them is raised. - - @@ -162,18 +159,44 @@ explanation before or after it. +## Explain a tenth draft that could not be repaired + +Ten drafts is the limit, and the tenth cannot be revised into an eleventh. So a +tenth draft that still has problems after its repair attempts leaves nothing to +approve and nothing to ask for — there is no decision left for you to make, and +this workflow does not ask you to make one. + +Instead the coding agent is asked once, automatically, why the attempts did not +work and what would make a future prompt more likely to succeed. That answer is +explanation and nothing else: it creates no new draft, extends no limit, and +ends this Plan without a program. + + + +The final Plan still has these problems: + + + + +Explain briefly why the attempts did not resolve them and what the person should +clarify in their next Prompt. Do not create another Plan. + + + + + ## Review the draft -The workflow shows you the complete draft after its repair attempts. +Every draft you are shown has a decision left in it. The workflow shows you the +complete draft after its repair attempts. - Choose **Approve** to accept a draft that passed its check. - Choose **Request changes** to send feedback to the coding agent and create a new draft. - Choose **Stop** to end without returning anything. -A draft with remaining problems cannot be approved. You may review at most ten -drafts, and the tenth cannot be revised. If the tenth draft still has problems, -you may ask the coding agent to explain what went wrong or stop. +A draft with remaining problems cannot be approved, and the tenth draft cannot +be revised, so on either of those the choices you are offered are narrower. @@ -236,26 +255,7 @@ remaining choices are to ask the coding agent what went wrong, or to stop. - - - - - - - - - -The final Plan still has these problems: - - - - -Explain briefly why the attempts did not resolve them and what the person should -clarify in their next Prompt. Do not create another Plan. - - - @@ -286,17 +286,21 @@ explanation before or after it. -## Return the approved Plan +## Produce the approved Plan source Only an approved Plan leaves this workflow, and only after the whole authorship frame above has been taken down. Its exact source is checked once more as -structure — every declaration, resolution and form it uses — and then handed -back byte for byte. +structure — every declaration, resolution and form it uses — and retained +together with the prompt it was written for, so asking again restores these +exact bytes instead of writing a second Plan. - - + + +The source is what this workflow produces, byte for byte. Nothing here runs it. + +{admitted} diff --git a/packages/cli/src/plan-component.ts b/packages/cli/src/plan-component.ts index dece5e628..91d9c2d5a 100644 --- a/packages/cli/src/plan-component.ts +++ b/packages/cli/src/plan-component.ts @@ -40,7 +40,7 @@ import { createHash } from "node:crypto"; import { scoped } from "effection"; import type { Operation, Result, Scope } from "effection"; -import { createDurableOperation } from "@executablemd/durable-streams"; +import { createDurableOperation, StaleInputError } from "@executablemd/durable-streams"; import type { Json } from "@executablemd/durable-streams"; import { agentIdentityComponents, @@ -180,6 +180,23 @@ const INPUTS_RETURNS = { additionalProperties: false, }; +/** + * What the frozen inputs are given: the caller's optional session name, and the + * prompt this invocation is about. + * + * The prompt is here so that the first durable record of the invocation is + * about a question as well as a catalog. Only its digest is kept. + */ +const INPUTS_PROPS = { + type: "object", + properties: { + session: { type: "string", minLength: 1 }, + instruction: { type: "string" }, + }, + required: ["instruction"], + additionalProperties: false, +}; + const OPTIONAL_SESSION = { type: "object", properties: { session: { type: "string", minLength: 1 } }, @@ -204,6 +221,21 @@ const SOURCE_PROP = { additionalProperties: false, }; +/** + * What the final admission is given: the approved bytes, and the prompt they + * were written for. + * + * The prompt is here because the artifact this phase retains is an answer to a + * question, and a record that did not say which question would be restorable + * for one nobody asked. Only its digest is kept. + */ +const ADMIT_PROPS = { + type: "object", + properties: { source: { type: "string" }, instruction: { type: "string" } }, + required: ["source", "instruction"], + additionalProperties: false, +}; + const CHECK_RETURNS = { type: "object", properties: { valid: { type: "boolean" }, diagnostics: { type: "object" } }, @@ -241,6 +273,9 @@ export function* planComponentDeclaration( // empty rendering rather than on the spelling, which is the same answer for // a body that rendered to nothing. forms: ["paired"], + // What this Component renders is a program's source. Reflowing it or + // formatting it as Markdown would publish bytes nobody approved. + exact: true, privates: [ planInputs(assembly), planAuthorship(assembly), @@ -275,6 +310,7 @@ export function* planComponentDescription(): Operation[] = [ { name: "PlanInputs", - props: OPTIONAL_SESSION, + props: INPUTS_PROPS, returns: INPUTS_RETURNS, forms: ["self-closing"], }, { name: "PlanAuthorship", props: AUTHORSHIP_PROPS, forms: ["paired"] }, { name: "CheckDraft", props: SOURCE_PROP, returns: CHECK_RETURNS, forms: ["self-closing"] }, - { name: "AdmitPlan", props: SOURCE_PROP, returns: { type: "string" }, forms: ["self-closing"] }, + { name: "AdmitPlan", props: ADMIT_PROPS, returns: { type: "string" }, forms: ["self-closing"] }, ]; return described.map((component) => ({ ...component, @@ -326,17 +362,24 @@ function* uninvocable(): Operation { * * The catalog is an observation the first Agent turn is built from, so it is * journaled: a continuation restores what the run actually showed the agent - * rather than rebuilding one from a working tree that has moved. The session - * placement is derived here too, from the durable identity canonical execution - * minted for this exact expansion — which is what makes two `` sites, and - * two iterations of one site, distinct without either of them being nameable. + * rather than rebuilding one from a working tree that has moved. The instruction + * identity beside it is what makes a continuation answerable at all: this is the + * first durable record of the invocation, so comparing it here refuses a Plan + * asked for different instructions before a directory, a provider, a turn, a + * review or an admission exists — the only place that can refuse without having + * already done some of the work it would be refusing. + * + * The session placement is derived here too, from the durable identity canonical + * execution minted for this exact expansion — which is what makes two `` + * sites, and two iterations of one site, distinct without either of them being + * nameable. */ function planInputs(assembly: PlanComponentAssembly): IdentityComponent { return { name: "PlanInputs", origin: `${PLAN_ORIGIN}#PlanInputs`, forms: ["self-closing"], - props: OPTIONAL_SESSION, + props: INPUTS_PROPS, returns: INPUTS_RETURNS, factory: (claim: IdentityClaimant) => function* PlanInputs( @@ -351,10 +394,23 @@ function planInputs(assembly: PlanComponentAssembly): IdentityComponent { const authored = typeof props.session === "string" ? props.session : undefined; const session = placementFor(assembly, id, authored); - const syntax = yield* durablePlanOperation(`plan:inputs:${id}`, assembly.catalog); + const instruction = sourceDigest(String(props.instruction)); + + const frozen = yield* durablePlanOperation(`plan:inputs:${id}`, function* () { + return { syntax: yield* assembly.catalog(), instruction }; + }); + + // A history is input, so it is parsed rather than trusted. + const retained = readInputs(frozen); + if (retained === undefined) { + throw new Error(UNREADABLE_INPUTS); + } + if (retained.instruction !== instruction) { + throw new StaleInputError(STALE); + } return { - syntax, + syntax: retained.syntax, session, surface: assembly.surface, durable: durability(assembly, authored), @@ -570,6 +626,76 @@ function structuralAssessment( }; } +/** What the frozen inputs retained, or nothing when the record is not one. */ +interface RetainedInputs { + readonly syntax: string; + readonly instruction: string; +} + +function readInputs(value: Json): RetainedInputs | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return undefined; + } + const { syntax, instruction } = value; + if (typeof syntax !== "string" || typeof instruction !== "string") { + return undefined; + } + return { syntax, instruction }; +} + +const UNREADABLE_INPUTS = + "the retained Plan inputs cannot be read as Plan inputs, so no Plan source was produced."; + +/** + * What a continuation asking for a different Plan is refused with. + * + * A `StaleInputError` rather than an ordinary failure, because that is exactly + * what this is: the effect identity matches — same site, same occurrence — and + * what changed is outside the journal. Raising it as one also makes it the + * run's authoritative outcome, so the person reads why their Plan was refused + * instead of a count of unreached journal entries. + */ +const STALE = + "this Plan was written for different instructions, so the retained Plan is stale input. " + + "No Plan source was produced, and none was written for the new instructions. Re-run from " + + "the start rather than resuming a journal that answers a different question."; + +/** + * The record one approved Plan leaves behind. + * + * Five facts, retained together because they are one answer: which invocation + * asked, which prompt it asked about, the exact bytes that were approved, their + * digest, and that the structural admission succeeded. A continuation reads it + * back as hostile data — a history is input, not a value this process left in + * memory — and hands back those bytes without another turn, repair, check, + * review, explanation or admission. + */ +interface PlanArtifact { + readonly invocation: string; + readonly instruction: string; + readonly source: string; + readonly digest: string; + readonly admission: "valid"; +} + +/** What a record holds, or nothing when it is not one of these. */ +function readArtifact(value: Json): PlanArtifact | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return undefined; + } + const { invocation, instruction, source, digest, admission } = value; + if (typeof invocation !== "string" || typeof instruction !== "string") { + return undefined; + } + if (typeof source !== "string" || typeof digest !== "string" || admission !== "valid") { + return undefined; + } + return { invocation, instruction, source, digest, admission }; +} + +const UNREADABLE_ARTIFACT = + "the retained Plan artifact cannot be read as one, so no Plan source was produced."; + function admitPlan( assembly: PlanComponentAssembly, declared: readonly DeclaredMarkdownComponent[], @@ -578,7 +704,7 @@ function admitPlan( name: "AdmitPlan", origin: `${PLAN_ORIGIN}#AdmitPlan`, forms: ["self-closing"], - props: SOURCE_PROP, + props: ADMIT_PROPS, returns: { type: "string" }, factory: (claim: IdentityClaimant) => function* AdmitPlan( @@ -587,21 +713,41 @@ function admitPlan( ): Operation { const id = yield* claim(invocation); const candidate = String(props.source); - return yield* durablePlanOperation( - `plan:admit:${id}:${sourceDigest(candidate)}`, - function* () { - const validation = yield* structurally(assembly, declared, candidate); - if (validation.outcome === "invalid") { - throw new Error( - "the approved Plan does not validate:\n" + - JSON.stringify(validation.diagnostics, null, 2), - ); - } - // Byte for byte: what went in is what comes back, and the admission - // decided only whether it may. - return candidate; - }, - ); + const instruction = sourceDigest(String(props.instruction)); + // Named for the invocation alone. A name carrying the candidate's digest + // would be a different record for every draft, so a continuation whose + // instructions moved would author a second Plan rather than meet the one + // this site already has. + const retained = yield* durablePlanOperation(`plan:artifact:${id}`, function* () { + const validation = yield* structurally(assembly, declared, candidate); + if (validation.outcome === "invalid") { + throw new Error( + "the approved Plan does not validate:\n" + + JSON.stringify(validation.diagnostics, null, 2), + ); + } + return { + invocation: id, + instruction, + source: candidate, + digest: sourceDigest(candidate), + admission: "valid", + }; + }); + + const artifact = readArtifact(retained); + if (artifact === undefined) { + throw new Error(UNREADABLE_ARTIFACT); + } + if (artifact.invocation !== id || artifact.digest !== sourceDigest(artifact.source)) { + throw new Error(UNREADABLE_ARTIFACT); + } + if (artifact.instruction !== instruction) { + throw new StaleInputError(STALE); + } + // Byte for byte: what was approved is what comes back, and the + // admission decided only whether it may. + return artifact.source; }, }; } diff --git a/packages/cli/tests/document-suites/plan/Plan.test.md b/packages/cli/tests/document-suites/plan/Plan.test.md index cb82bfbe4..8fb3719e0 100644 --- a/packages/cli/tests/document-suites/plan/Plan.test.md +++ b/packages/cli/tests/document-suites/plan/Plan.test.md @@ -92,3 +92,166 @@ component does not exist. + +## The bare form emits the program's source + +Written without `as`, `` emits the approved program where the component +appears. The source is bytes rather than prose, so what reaches the run's output +is what the agent wrote — including a line ending in spaces and a run of blank +lines that the default output normalization rewrites in ordinary Markdown. + + + + + + + + + + + + + + + + +The trailing spaces and the four newlines survived, which is what says these +bytes were emitted as source rather than presented as prose. + + + +And nothing ran it: the file the program names is not here. + + + + + +## The captured form binds those same bytes and emits nothing + +`as` is ordinary text capture. The two forms differ only in where the source +goes, so the bytes a document binds are the bytes the bare form emits. + + + + + + + + + + + + + + + + +The run printed no program: capture emits nothing. + + + +What it bound is in the file it wrote, byte for byte — the same whitespace the +bare form emitted, read back rather than presented. + + + + + + + +Still nothing ran it. + + + + + +## A tenth draft that cannot be repaired is explained, not reviewed + +Ten drafts is the limit and the tenth cannot be revised, so a tenth draft that +still fails its check leaves no decision for a person to make. The workflow does +not ask for one. It asks the coding agent, once, why the attempts did not work, +and ends without a program. + +The only answer this test supplies is **Request changes**. That is what makes +the ending evidence: if a tenth review had opened, it could not have offered +that decision, and the run would have failed on the answer rather than on the +explanation below. + + + + + + + + + + + + + + + +The ending says ten drafts were reviewed and carries the coding agent's own +explanation of why none of them worked. + + + + + +No Plan came out of it, and the explanation was not mistaken for one. + + + + + + +## A draft with a revision left in it still reaches you + +The automatic explanation is only for a tenth draft with nothing left to decide. +Everywhere else the review is unchanged: a draft that still has problems can be +sent back, and the draft that comes back can be approved. + +The two answers below are told apart by what the review shows. A draft that +failed its check is presented with the problems that remain, so the matcher +naming that heading answers the first review; the second review has no such +heading and falls through to approval. + + + + + + + + + + + + + + + + +The revision was reviewed and approved, and its source is what came back. + + + + + + + + diff --git a/packages/cli/tests/document-suites/plan/agents/exhausted-plan.md b/packages/cli/tests/document-suites/plan/agents/exhausted-plan.md new file mode 100644 index 000000000..42e076f6a --- /dev/null +++ b/packages/cli/tests/document-suites/plan/agents/exhausted-plan.md @@ -0,0 +1,169 @@ +", ""].join("\n")} +/> + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +{draft} + + + +Every draft named a component this profile does not offer. Name a component that +exists, or describe the outcome without naming one. diff --git a/packages/cli/tests/document-suites/plan/agents/revised-plan.md b/packages/cli/tests/document-suites/plan/agents/revised-plan.md new file mode 100644 index 000000000..dea2c4491 --- /dev/null +++ b/packages/cli/tests/document-suites/plan/agents/revised-plan.md @@ -0,0 +1,36 @@ +", ""].join("\n")} +/> + +the approved Plan ran', + "", + ].join("\n")} +/> + + + +{broken} + + + +{broken} + + + +{broken} + + + +{broken} + + + +{fixed} diff --git a/packages/cli/tests/document-suites/plan/agents/whitespace-plan.md b/packages/cli/tests/document-suites/plan/agents/whitespace-plan.md new file mode 100644 index 000000000..bc684e3b4 --- /dev/null +++ b/packages/cli/tests/document-suites/plan/agents/whitespace-plan.md @@ -0,0 +1,17 @@ + + +the approved Plan ran', + "", + ].join("\n")} +/> + +{program} diff --git a/packages/cli/tests/document-suites/plan/captures-plan.md b/packages/cli/tests/document-suites/plan/captures-plan.md new file mode 100644 index 000000000..2d9972924 --- /dev/null +++ b/packages/cli/tests/document-suites/plan/captures-plan.md @@ -0,0 +1,11 @@ +# A document that captures its Plan and writes it down + +The captured form emits nothing, so a reader of this run sees no program. What +it binds is written to a file, which is how a test compares the captured bytes +with the ones the bare form emits. + +Write the release program. + +{approved} + +Captured the approved program. diff --git a/packages/cli/tests/document-suites/plan/emits-plan.md b/packages/cli/tests/document-suites/plan/emits-plan.md new file mode 100644 index 000000000..79b9edeff --- /dev/null +++ b/packages/cli/tests/document-suites/plan/emits-plan.md @@ -0,0 +1,7 @@ +# A document that emits its Plan + +This is an ordinary document. It writes `` with no `as`, which is the bare +form: the approved program's source is emitted here, where the component is +written. + +Write the release program. diff --git a/packages/cli/tests/packaged-document.test.ts b/packages/cli/tests/packaged-document.test.ts index b52df53dd..36fc40f6e 100644 --- a/packages/cli/tests/packaged-document.test.ts +++ b/packages/cli/tests/packaged-document.test.ts @@ -121,10 +121,13 @@ describe("packaged documents", () => { const committed = yield* readTextFile(fileURLToPath(packagedDocumentUrl(PLAN_DOCUMENT))); expect(source).toBe(committed); - // It is a Markdown value component a document invokes, not a root a command - // runs: one optional `session` prop, and a string return. - expect(source).toContain("returns:"); + // It is a Markdown text component a document invokes, not a root a command + // runs: one optional `session` prop, no declared return, and one top-level + // `` selecting the approved source as the whole of its rendering. + expect(source).not.toContain("returns:"); expect(source).toContain("session: { type: string, minLength: 1 }"); + expect(source).toContain("{admitted}"); + expect(source).not.toContain(" { ).toBe(2); // The worked example is itself a titled Plan. expect(source).toContain("# Ask for and save your age"); - // A final invalid review offers two ways out, and the prose says so. + // A tenth draft that could not be repaired is not a review at all, and the + // prose says why: there is no decision left to offer. expect(source).toContain( - "the two\nremaining choices are to ask the coding agent what went wrong, or to stop.", + "there is no decision left for you to make, and\nthis workflow does not ask you to make one.", ); // Every visible stage of the workflow is a heading somebody can audit. for (const heading of [ @@ -150,7 +154,8 @@ describe("packaged documents", () => { "## Check and repair the draft", "## Review the draft", "## Continue from your decision", - "## Return the approved Plan", + "## Explain a tenth draft that could not be repaired", + "## Produce the approved Plan source", ]) { expect(source).toContain(`${heading}\n`); } @@ -165,7 +170,10 @@ describe("packaged documents", () => { expect(source).toContain( '', ); - expect(source.split("reviewed ten drafts without an approved Plan").length - 1).toBe(4); + // Twice, once per surface: the automatic explanation is the only ending + // that says it. The authored exhaustion sentence went with the review a + // tenth invalid draft no longer reaches. + expect(source.split("reviewed ten drafts without an approved Plan").length - 1).toBe(2); // The closing branch is an unexpected-no-decision fallback, not a second // copy of exhaustion: exhaustion is decided inside review, and saying it // twice would make the two endings indistinguishable to a reader. @@ -173,7 +181,9 @@ describe("packaged documents", () => { // The choices are the words a person reads, with no internal spelling // behind them. expect(source).toContain('["Approve", "Request changes", "Stop"]'); - expect(source).toContain('["Explain what went wrong", "Stop"]'); + // A tenth draft that could not be repaired is explained automatically, so + // there is no decision to offer and no question naming one. + expect(source).not.toContain('"Explain what went wrong"'); expect(source).not.toContain('"revise"'); expect(source).not.toContain('"abort"'); expect(source).not.toContain(''); diff --git a/packages/cli/tests/plan-component.test.ts b/packages/cli/tests/plan-component.test.ts index d3bfcd226..b8d327f73 100644 --- a/packages/cli/tests/plan-component.test.ts +++ b/packages/cli/tests/plan-component.test.ts @@ -15,22 +15,24 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { ensure, scoped, until } from "effection"; +import { ensure, scoped, spawn, until } from "effection"; import type { Operation } from "effection"; import { ensureDir, rm, writeTextFile } from "@effectionx/fs"; import { randomUUID } from "node:crypto"; import { readdir } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { forEach } from "@effectionx/stream-helpers"; import { agentIdentityComponents, collect, installAgentComponents, retainedSource, + useNormalizedOutput, } from "@executablemd/core"; import type { Json } from "@executablemd/core"; import { validateDocument } from "@executablemd/core"; -import { executeInstalled } from "@executablemd/core/host"; +import { executeInstalled, sourceDigest } from "@executablemd/core/host"; import { InMemoryStream } from "@executablemd/durable-streams"; import type { DurableEvent } from "@executablemd/durable-streams"; @@ -52,6 +54,8 @@ const PLAN = ["# Say hello", "", "This document greets you.", "", "Hello.", ""]. /** What one document run produced, and every phase that was reached. */ interface Run { output: string; + /** What the Output Api actually emitted, through whatever middleware ran. */ + emitted: string; value: Json | undefined; failure: string | undefined; harness: PlanDeclarationHarness; @@ -92,6 +96,18 @@ function* runDocument(options: { } | null; assess?: (source: string) => Operation<{ valid: boolean; diagnostics: Json }>; session?: string; + props?: Record; + /** + * Install the output middleware an ordinary `xmd run` installs. + * + * Off by default, because most cases are about what a document rendered + * rather than about how it was presented. A case about exact bytes turns it + * on, so the normalizer that would rewrite them is actually in the way. The + * Markdown suite cannot do this: its host installs no presentation + * middleware, so an exactness assertion there would hold whether the bypass + * existed or not. + */ + normalized?: boolean; }): Operation { const root = options.root ?? (yield* authorshipRoot()); const stream = options.stream ?? new InMemoryStream(); @@ -114,14 +130,19 @@ function* runDocument(options: { let value: Json | undefined; let failure: string | undefined; let output = ""; + const chunks: string[] = []; yield* scoped(function* () { yield* installAgentComponents({ defaultAgent: AGENT, permissionMode: "deny-all" }); + if (options.normalized === true) { + yield* useNormalizedOutput(); + } try { const execution = yield* executeInstalled( { ...retainedSource(ROOT, options.source), stream, includes: [...(options.includes ?? [])], + ...(options.props === undefined ? {} : { props: options.props }), }, [ { @@ -130,15 +151,32 @@ function* runDocument(options: { }, ], ); + // Subscribed before the completion is awaited, so what is collected is + // what went out through the middleware — the close value is the + // document's own rendering and would show none of the presentation. + const draining = yield* spawn(function* () { + yield* forEach(function* (chunk: string) { + chunks.push(chunk); + }, execution.output); + }); value = yield* collect(execution); output = String(value); + yield* draining; } catch (error) { failure = error instanceof Error ? error.message : String(error); } }); const leftover = yield* until(readdir(root)); - return { output, value, failure, harness, stream, leftover: leftover.sort() }; + return { + output, + emitted: chunks.join(""), + value, + failure, + harness, + stream, + leftover: leftover.sort(), + }; } /** A partial continuation of one run: everything it recorded but the terminals. */ @@ -284,11 +322,14 @@ describe("Tier PC — in an ordinary document", () => { expect(plan).toBeDefined(); expect(plan?.sourceKind).toBe("declared-markdown"); expect(plan?.forms).toEqual(["paired"]); - expect(plan?.returnMode).toBe("value"); + // A text component: the approved source is what it renders, and `as` is + // ordinary text capture rather than a declared return. + expect(plan?.returnMode).toBe("text"); expect(Reflect.get(Object(plan?.origin), "origin")).toBe(PLAN_ORIGIN); // The description a document author reads is the packaged Component's own // frontmatter, so the asset and the entry describing it are one text. - expect(plan?.description).toContain("Create an XMD program from a Prompt."); + expect(plan?.description).toContain("Create an XMD program from a prompt."); + expect(plan?.description).toContain("emits the approved program source."); for (const category of catalog.categories) { const names = category.entries.map((entry) => entry.name); @@ -555,6 +596,218 @@ describe("Tier PC — in an ordinary document", () => { }); }); + /** + * The Plan artifact a run retained, or nothing when it retained none. + * + * Read from the journal rather than from the binding, because retention is what + * a continuation reads and an ending that produced no Plan must leave nothing + * there for one to find. + */ + function* retainedArtifact(stream: InMemoryStream): Operation { + for (const event of yield* stream.readAll()) { + if (event.type !== "yield" || !event.description.name.startsWith("plan:artifact:")) { + continue; + } + if (event.result.status === "ok") { + return event.result.value ?? null; + } + } + return undefined; + } + + it("PC19: the emitted source survives the CLI's own output normalization", function* () { + yield* useWorkingDirectory(function* (dir) { + // Whitespace the normalizer rewrites in prose: a line ending in spaces, + // and a run of four newlines. This case owns the presentation path + // because only a host can install that middleware — the Markdown suite's + // host installs none, so the same assertion there would hold whether the + // bypass existed or not. + const exact = [ + "# Approved program", + "", + "This program writes a file when something runs it. ", + "", + "", + "", + 'the approved Plan ran', + "", + ].join("\n"); + + const run = yield* runDocument({ + source: ["", "Write a program.", "", ""].join("\n"), + reply: exact, + normalized: true, + }); + + expect(run.failure).toBe(undefined); + expect(run.emitted).toContain(exact); + // And nothing ran it. + expect((yield* until(readdir(dir))).sort()).toEqual([]); + }); + }); + + it("PC20: the retained artifact carries the approved bytes and their digest", function* () { + yield* useWorkingDirectory(function* () { + const stream = new InMemoryStream(); + const run = yield* runDocument({ + source: ['Write a program.', "", "got: {approved}", ""].join( + "\n", + ), + reply: PLAN, + stream, + }); + + expect(run.failure).toBe(undefined); + // Retained before either form could publish the bytes. + const artifact = Object(yield* retainedArtifact(stream)); + expect(Reflect.get(artifact, "source")).toBe(PLAN); + expect(Reflect.get(artifact, "digest")).toBe(sourceDigest(PLAN)); + expect(Reflect.get(artifact, "admission")).toBe("valid"); + }); + }); + + it("PC21: a stopped Plan leaves no artifact for a later evaluation to begin from", function* () { + yield* useWorkingDirectory(function* () { + const stream = new InMemoryStream(); + const run = yield* runDocument({ + source: ['Write a program.', "", "got: {approved}", ""].join( + "\n", + ), + reply: PLAN, + reviews: ["Stop"], + stream, + }); + + expect(run.failure).toBe("Plan authorship stopped at your request. No Plan was returned."); + expect(run.output).not.toContain("got:"); + expect(run.emitted).not.toContain("# Say hello"); + // Nothing retained, so nothing a later evaluation could restore. + expect(yield* retainedArtifact(stream)).toBe(undefined); + }); + }); + + it("PC22: a completed Plan restores without authoring or admitting it again", function* () { + yield* useWorkingDirectory(function* (dir) { + // A component the first run can resolve and the second cannot. The + // approved Plan names it, so an admission that ran a second time would + // refuse these bytes instead of restoring them — which is how this tells + // restoration apart from a repeat. + yield* writeTextFile(join(dir, "Widget.md"), "a widget.\n"); + const withWidget = ["# Use the widget", "", "", ""].join("\n"); + const source = [ + 'Write a program.', + "", + "got: {approved}", + "", + ].join("\n"); + + const first = new InMemoryStream(); + const one = yield* runDocument({ source, reply: withWidget, includes: [dir], stream: first }); + expect(one.failure).toBe(undefined); + expect(one.output).toContain(`got: ${withWidget}`); + + const two = yield* runDocument({ + source, + reply: "# A different Plan\n\nnot this one.\n", + reviews: [], + includes: [], + stream: yield* continuing(first), + }); + + expect(two.failure).toBe(undefined); + expect(two.output).toContain(`got: ${withWidget}`); + expect(two.harness.fake.prompts).toEqual([]); + expect(two.harness.reviews).toEqual([]); + expect(two.harness.checked).toEqual([]); + }); + }); + + it("PC23: a continuation replays the retained root, so a changed body never runs", function* () { + yield* useWorkingDirectory(function* () { + // The retained-root negative control the story names. A continuation runs + // the root the journal kept, so a changed authored body is never expanded + // and is not the changed-instruction case below. + const first = new InMemoryStream(); + const one = yield* runDocument({ + source: ['Write a program.', "", "got: {approved}", ""].join( + "\n", + ), + reply: PLAN, + stream: first, + }); + expect(one.failure).toBe(undefined); + + const two = yield* runDocument({ + source: [ + 'Write a different program.', + "", + "second run: {approved}", + "", + ].join("\n"), + reply: PLAN, + reviews: [], + stream: yield* continuing(first), + }); + + expect(two.failure).toBe(undefined); + expect(two.output).not.toContain("second run:"); + expect(two.output).toContain(`got: ${PLAN}`); + }); + }); + + it("PC24: instructions that render differently refuse in the frozen inputs", function* () { + yield* useWorkingDirectory(function* () { + // The prompt arrives through props, which is what can actually differ on + // a continuation: the authored body cannot, per PC23. + const source = [ + "---", + "props:", + " type: object", + " properties:", + " request: { type: string }", + " required: [request]", + "---", + "", + '{props.request}', + "", + "got: {approved}", + "", + ].join("\n"); + + const root = yield* authorshipRoot(); + const first = new InMemoryStream(); + const one = yield* runDocument({ + source, + props: { request: "Write a program." }, + root, + reply: PLAN, + stream: first, + }); + expect(one.failure).toBe(undefined); + + const two = yield* runDocument({ + source, + props: { request: "Write a different program." }, + root, + reply: PLAN, + reviews: [], + stream: yield* continuing(first), + }); + + expect(two.failure).toContain("stale input"); + expect(two.failure).toContain("none was written for the new instructions"); + // Neither the retained Plan nor a newly authored one. + expect(two.output).not.toContain("# Say hello"); + expect(two.output).not.toContain("got:"); + // Refused in the frozen inputs: before a turn, a review, a check or a + // session directory existed. + expect(two.harness.fake.prompts).toEqual([]); + expect(two.harness.reviews).toEqual([]); + expect(two.harness.checked).toEqual([]); + expect(two.leftover).toEqual([]); + }); + }); + it("PC14: nothing the Component does reaches the caller's filesystem", function* () { yield* useWorkingDirectory(function* (dir) { const run = yield* runDocument({ diff --git a/packages/cli/tests/plan.test.ts b/packages/cli/tests/plan.test.ts index 96fa01b98..c1b2ceade 100644 --- a/packages/cli/tests/plan.test.ts +++ b/packages/cli/tests/plan.test.ts @@ -1107,10 +1107,10 @@ describe( expect(harness.executions).toHaveLength(0); }); - // Exhaustion is reachable, and it is a different ending. Ten presentations - // that never validated, the last offering nothing but a way out, and the - // sentence says there was never a Plan to approve rather than that - // somebody decided to stop. + // Exhaustion is reachable, and it is a different ending: ten drafts that + // never validated. The tenth is not a review at all — there is nothing to + // approve and nothing left to revise into — so what ends the command is + // the automatic explanation below rather than a decision somebody took. yield* useWorkingDirectory(function* (dir, authorshipRoot) { const harness = createPlanHarness({ authorshipRoot }); for (const round of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) { @@ -1118,46 +1118,45 @@ describe( for (const _draft of [0, 1, 2, 3]) { harness.fake.script({ reply: UNRESOLVED }); } - harness.script( - round < 10 - ? { decision: "Request changes", feedback: `round ${round}` } - : { decision: "Stop" }, - ); + if (round < 10) { + harness.script({ decision: "Request changes", feedback: `round ${round}` }); + } } + harness.fake.script({ + reply: "The drafts all named a component this profile does not have.", + }); const { value, lines } = yield* reported(() => runPlan(command(dir, [REQUEST], STACK), harness.deps), ); expect(value).toBe(1); - expect(harness.reviews).toHaveLength(10); - // Rounds one to nine could be sent back; the tenth had one choice, and - // taking it is what reaches the exhaustion ending. + // Nine reviews, not ten: the tenth invalid draft is never presented, + // because every decision it could have offered is unavailable. + expect(harness.reviews).toHaveLength(9); expect(decisions(harness.reviews[8])).toEqual(["Request changes", "Stop"]); - expect(decisions(harness.reviews[9])).toEqual(["Explain what went wrong", "Stop"]); expect(lines).toHaveLength(1); - expect(lines[0]).toBe( - "xmd plan reviewed ten drafts without an approved Plan. Nothing was output or run.", - ); - // Stopping asks the coding agent nothing: forty drafting turns and no - // forty-first. - expect(harness.fake.prompts).toHaveLength(40); + expect(lines[0]).toContain("reviewed ten drafts without an approved Plan"); + expect(lines[0]).toContain("Nothing was output or run."); + // Forty drafting turns and exactly one explanation turn after them. + expect(harness.fake.prompts).toHaveLength(41); expect(harness.executions).toHaveLength(0); }); }); - it("C3, C9: the last invalid draft can be explained rather than only stopped", function* () { + it("C3, C9: a tenth draft that cannot be repaired is explained automatically", function* () { yield* useWorkingDirectory(function* (dir, authorshipRoot) { const harness = createPlanHarness({ authorshipRoot }); for (const round of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) { for (const _draft of [0, 1, 2, 3]) { harness.fake.script({ reply: UNRESOLVED }); } - harness.script( - round < 10 - ? { decision: "Request changes", feedback: `round ${round}` } - : { decision: "Explain what went wrong" }, - ); + // No answer is scripted for a tenth review, because there is no + // tenth review: running out would be a test defect, so a run that + // asked for one fails here rather than passing quietly. + if (round < 10) { + harness.script({ decision: "Request changes", feedback: `round ${round}` }); + } } // The explanation the coding agent gives is prose, and stays prose. const explanation = [ @@ -1177,7 +1176,7 @@ describe( // conversation. It is not another draft: no eleventh review, and the // ten-draft limit is not reopened. expect(harness.fake.prompts).toHaveLength(41); - expect(harness.reviews).toHaveLength(10); + expect(harness.reviews).toHaveLength(9); expect(sessions(harness)).toHaveLength(1); // It carries the final problems, which were produced after the agent's diff --git a/packages/core/src/api.ts b/packages/core/src/api.ts index ada3ce829..edac3532b 100644 --- a/packages/core/src/api.ts +++ b/packages/core/src/api.ts @@ -12,7 +12,14 @@ import { type Api, createApi } from "@effectionx/context-api"; import type { Operation } from "effection"; export interface DocumentOutputApi { - output(text: string): Operation; + /** + * @param exact Whether this text is exact bytes rather than prose. The + * presentation middleware — whitespace normalization and terminal + * formatting — passes exact bytes through untouched, because their + * whitespace is part of what the run produced rather than a way of writing + * it. Absent means prose, which is what every ordinary emission is. + */ + output(text: string, exact?: boolean): Operation; } /** @@ -36,6 +43,6 @@ export interface DocumentOutputApi { export const DocumentOutput: Api = createApi( "DocumentOutput", { - *output(_text: string): Operation {}, + *output(_text: string, _exact?: boolean): Operation {}, }, ); diff --git a/packages/core/src/components/declared-markdown.ts b/packages/core/src/components/declared-markdown.ts index 2030b0347..777db0e6d 100644 --- a/packages/core/src/components/declared-markdown.ts +++ b/packages/core/src/components/declared-markdown.ts @@ -102,6 +102,17 @@ export interface DeclaredMarkdownComponent { readonly returns?: ReturnsSchema; /** Components only elements authored by these exact bytes may resolve. */ readonly privates?: readonly IdentityComponent[]; + /** + * Whether what this component renders is exact bytes rather than prose. + * + * A text component's rendering is ordinarily presentation: the whitespace + * middleware reflows it and the terminal middleware formats it as Markdown. + * A component whose rendering is a program's source is not presentation, and + * a host that ships such bytes says so here. Only a declaring host can: the + * Markdown itself cannot ask for it, so a repository file of the same name + * gets the presentation every other document gets. + */ + readonly exact?: boolean; } /** One declaration, admitted: what the host stated, checked against its bytes. */ @@ -181,7 +192,9 @@ export function* admitDeclaredMarkdown( ); } - const definition = yield* parseMarkdownDefinition(name, origin, source); + const parsed = yield* parseMarkdownDefinition(name, origin, source); + const definition: ComponentDefinition = + declaration.exact === true ? { ...parsed, exact: true } : parsed; const forms = declaration.forms ?? BOTH_FORMS; const badForms = formsRefusal(forms); diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index cb18b3e38..ef2f56952 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -1728,7 +1728,7 @@ function* runValueRoot( for (const resolved of expanded) { const text = renderSegment(resolved); if (text) { - yield* ephemeral(DocumentOutput.operations.output(text)); + yield* ephemeral(DocumentOutput.operations.output(text, exactly(resolved))); chunks.push(text); } } @@ -1755,6 +1755,44 @@ function* runValueRoot( * that exited nonzero failed the run is not theirs to decide, and this is where * the run says so (#441). */ +/** Whether one expanded segment carries exact bytes rather than prose. */ +function exactly(segment: Segment): boolean { + return segment.type === "text" && segment.exact === true; +} + +/** + * What a buffered region emits: consecutive segments of one exactness, joined. + * + * Buffering is what makes this necessary. A streaming root hands the Output Api + * one segment at a time and each write says what it is; a region that renders + * as a whole would otherwise join a program's approved source to the prose + * beside it and present the pair as one thing. Segments of the same kind still + * travel together, so a region holding no exact bytes emits exactly once, as it + * always has. + */ +interface Emission { + readonly text: string; + readonly exact: boolean; +} + +function emissions(segments: readonly Segment[]): Emission[] { + const runs: Emission[] = []; + for (const segment of segments) { + const text = renderSegment(segment); + if (!text) { + continue; + } + const exact = exactly(segment); + const last = runs[runs.length - 1]; + if (last !== undefined && last.exact === exact) { + runs[runs.length - 1] = { text: last.text + text, exact }; + continue; + } + runs.push({ text, exact }); + } + return runs; +} + function* refuseCheckedFailure(checkedFailures: CheckedFailures): Operation { const segment = checkedFailures.failure; if (segment !== undefined) { @@ -1880,11 +1918,12 @@ function* documentWorkflow( authority, undefined, ); - const text = selected.map(renderSegment).join(""); // An empty buffered root emits no output event. - if (text) { - yield* ephemeral(DocumentOutput.operations.output(text)); + const runs = emissions(selected); + for (const run of runs) { + yield* ephemeral(DocumentOutput.operations.output(run.text, run.exact)); } + const text = runs.map((run) => run.text).join(""); yield* refuseCheckedFailure(checkedFailures); return { status: "ok", output: text, value: text }; } @@ -1917,7 +1956,9 @@ function* documentWorkflow( // ephemeral() bridges from Workflow (durable) to Operation // (non-durable) — output emission is a derived side effect, // not journaled. - yield* ephemeral(DocumentOutput.operations.output(text)); + yield* ephemeral( + DocumentOutput.operations.output(text, resolved !== undefined && exactly(resolved)), + ); streamed.push(text); } } @@ -1941,9 +1982,10 @@ function* documentWorkflow( // chunks is who the preservation is for, and the completion path only emits // for a run that streamed nothing at all — which stops being true as soon // as an earlier segment went out. - const tail = produced.slice(emittedThrough).map(renderSegment).join(""); - if (tail) { - yield* ephemeral(DocumentOutput.operations.output(tail)); + const runs = emissions(produced.slice(emittedThrough)); + const tail = runs.map((run) => run.text).join(""); + for (const run of runs) { + yield* ephemeral(DocumentOutput.operations.output(run.text, run.exact)); } // A durability failure is not something the document did, so it never // becomes the document's own outcome (§6.11). @@ -2836,6 +2878,7 @@ function* invoke( ...(declaration.privates === undefined ? {} : { privates: [...declaration.privates].map(retainedIdentityComponent) }), + ...(declaration.exact === undefined ? {} : { exact: declaration.exact }), }), ), ), diff --git a/packages/core/src/expand.ts b/packages/core/src/expand.ts index a8359dd20..992f61988 100644 --- a/packages/core/src/expand.ts +++ b/packages/core/src/expand.ts @@ -2712,6 +2712,9 @@ function* expandComponent( } const bodyOwner = asBinding === undefined ? owner : undefined; + // Where this body's own segments start, so what it renders can be told apart + // from what the caller had already produced into the same array. + const renderedFrom = bodyOwner?.length ?? 0; const expanded = yield* withInvocation(function* (invocation) { yield* installInvocation(invocation); return yield* expandBody( @@ -2731,6 +2734,15 @@ function* expandComponent( ); }); + // What a component declared exact renders is exact wherever it renders: the + // caller's flow, its own returned region, or neither when the invocation + // binds instead. Marking it here — on the segments this body produced, after + // it produced them — is what carries the fact to the emission loop, which is + // outside every scope the invocation owned. + if (definition.exact === true) { + markExact(bodyOwner === undefined ? expanded : bodyOwner.slice(renderedFrom)); + } + if (asBinding) { // A capture never swallows an error. The body reported these where they // were created (§6.9) — including the ones a string projection rendered @@ -2764,6 +2776,15 @@ function* expandComponent( return bodyOwner === undefined ? expanded : []; } +/** Say of each text segment here that its bytes are exact rather than prose. */ +function markExact(segments: readonly Segment[]): void { + for (const segment of segments) { + if (segment.type === "text") { + segment.exact = true; + } + } +} + // Without `returns`, a function component's rendering is its return value, so // anything else is a contract violation rather than something to stringify. function asText(output: Json): string { diff --git a/packages/core/src/output/normalize.ts b/packages/core/src/output/normalize.ts index 8ba7042cf..aa4b8928b 100644 --- a/packages/core/src/output/normalize.ts +++ b/packages/core/src/output/normalize.ts @@ -17,7 +17,19 @@ export function* useNormalizedOutput(): Operation { let trailingNewlines = 0; yield* DocumentOutput.around({ - *output([text], next) { + *output([text, exact], next) { + // Exact bytes are not prose. Stripping a trailing space or collapsing a + // blank line inside a program's approved source would publish bytes + // nobody approved, so the write goes out as it arrived — while its own + // trailing newlines still count, because the prose after it is still + // prose. + if (exact === true) { + const trailing = text.match(/\n+$/); + trailingNewlines = trailing ? trailing[0].length : 0; + yield* next(text, true); + return; + } + let normalized = text; // Strip trailing whitespace on each line @@ -36,7 +48,7 @@ export function* useNormalizedOutput(): Operation { const match = normalized.match(/\n+$/); trailingNewlines = match ? match[0].length : 0; - yield* next(normalized); + yield* next(normalized, exact); }, }); } diff --git a/packages/core/src/output/terminal.ts b/packages/core/src/output/terminal.ts index 942143050..b09a1dcf8 100644 --- a/packages/core/src/output/terminal.ts +++ b/packages/core/src/output/terminal.ts @@ -15,9 +15,16 @@ export function* useTerminalOutput(): Operation { // markedTerminal() returns a marked extension object ({ renderer, useNewRenderer }) const marked = new Marked(markedTerminal()); yield* DocumentOutput.around({ - *output([text], next) { + *output([text, exact], next) { + // Exact bytes go out as they arrived. Rendering a program's approved + // source as Markdown would show its headings as headings and its fences + // as boxes, which is the one thing source must never be turned into. + if (exact === true) { + yield* next(text, true); + return; + } const formatted = marked.parse(text, { async: false }) as string; - yield* next(formatted); + yield* next(formatted, exact); }, }); } diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index 4dfe15bfe..57524b28a 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -24,6 +24,16 @@ export type Segment = export interface TextSegment { type: "text"; content: string; + /** + * Whether this text is exact bytes rather than prose. + * + * Prose is reflowed on the way out — trailing spaces go and runs of blank + * lines collapse — because that is presentation of something written to be + * read. Bytes a run produced as a program's source are not that: their + * whitespace is part of what was approved, so a segment carrying them says + * so and the emission loop hands them to the Output Api on their own. + */ + exact?: true; } export interface ComponentElement { @@ -199,6 +209,14 @@ export interface ComponentDefinition { /** Absent in text mode, where the component returns its rendered markdown. */ returns?: ReturnsSchema; bodySegments: Segment[]; + /** + * Whether what this component renders is exact bytes rather than prose. + * + * Stated by the trusted host that declares the component, never by the + * Markdown itself: a repository file cannot ask for its own rendering to + * bypass the presentation every other document is subject to. + */ + exact?: boolean; } /** diff --git a/packages/core/tests/output-normalize.test.ts b/packages/core/tests/output-normalize.test.ts index 74b8367e5..25e61355a 100644 --- a/packages/core/tests/output-normalize.test.ts +++ b/packages/core/tests/output-normalize.test.ts @@ -13,7 +13,7 @@ import { subscribe } from "../src/subscribe.ts"; * * Install order: normalize first (outermost), channel last (closest to core). */ -function* collectNormalized(texts: string[]): Operation { +function* collectNormalized(texts: (string | [string, boolean])[]): Operation { const channel = createChannel(); // First: normalization (runs first — outermost) yield* useNormalizedOutput(); @@ -29,7 +29,11 @@ function* collectNormalized(texts: string[]): Operation { yield* ready; for (const text of texts) { - yield* DocumentOutput.operations.output(text); + if (typeof text === "string") { + yield* DocumentOutput.operations.output(text); + continue; + } + yield* DocumentOutput.operations.output(text[0], text[1]); } yield* channel.close(); @@ -81,4 +85,31 @@ describe("Tier WN — Whitespace normalization", () => { const result = yield* collectNormalized(["text\t\n"]); expect(result).toEqual(["text\n"]); }); + + // WN8: An exact write is presentation-free + it("WN8: exact bytes pass through untouched", function* () { + // Every rewrite this middleware makes, in one write: a line ending in + // spaces, one ending in a tab, and a run of four newlines. + const exact = "one \ntwo\t\n\n\n\nthree"; + const result = yield* collectNormalized([[exact, true]]); + expect(result).toEqual([exact]); + }); + + // WN9: The bypass is per write, not a switch + it("WN9: prose around an exact write is still normalized", function* () { + const result = yield* collectNormalized([ + "before \n\n", + ["kept \n\n\n\n", true], + "\n\nafter \n\n\nend", + ]); + + // The prose before it is normalized as it always was. + expect(result[0]).toBe("before\n\n"); + // The exact write is not. + expect(result[1]).toBe("kept \n\n\n\n"); + // And the prose after it is normalized again — including the leading-newline + // collapse, which means the exact write's own trailing newlines were still + // counted rather than the tracking being abandoned. + expect(result[2]).toBe("\nafter\n\nend"); + }); }); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 211c4ae4e..72fe5f829 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2775,10 +2775,18 @@ middleware cannot answer, replace or change one of those. It says nothing about any other name: an unrelated import in the same execution is the ordinary open import it has always been. +**A declaring host may say its component renders source rather than prose.** A +text component's rendering is ordinarily presentation, and the output middleware +reflows it and formats it. One whose rendering is a program's source is not, so +what it renders is emitted exactly — the presentation middleware passes it +through untouched, and prose on either side of it is presented as before +(§9.4). Only the declaring host states this. The Markdown cannot ask for it, so +a repository file of the same name is presented like every other document. + **Every declaration is read once, before any installation runs.** The name, -origin, source, digest, forms, prose and each private declaration are captured -by the invocation and held by it, with the factory bound and every array and -schema copied. A schema is copied rather than referenced because it is a whole +origin, source, digest, forms, prose, whether it renders exact source, and each +private declaration are captured by the invocation and held by it, with the +factory bound and every array and schema copied. A schema is copied rather than referenced because it is a whole object graph: holding the caller's object would let a hook reach into it after capture and change the contract admission compiles, registration publishes and expansion validates each invocation against. So a host that hands over a @@ -2797,10 +2805,13 @@ element asking is inside the same declaration. **`` is the one of these the `run` profile has.** The CLI declares `packages/cli/src/documents/Plan.md` to every ordinary run under the origin -`@executablemd/cli/Plan.md`, paired-only, returning a string. Its body is the -Prompt, rendered once with the capabilities the calling document already has; -its `as` receives the exact approved Plan source after the authorship frame has -been dismantled and the bytes have been structurally admitted. Its four private +`@executablemd/cli/Plan.md`, paired-only, as a text component. Its body is the +prompt, rendered once with the capabilities the calling document already has; +what it renders is the exact approved Plan source, after the authorship frame +has been dismantled and the bytes have been structurally admitted. Written bare +it emits that source where the component is written, and `as` is ordinary text +capture: the same bytes are bound and nothing is emitted. Neither form +evaluates the source. Its four private capabilities — ``, ``, `` and `` — are the closure those exact bytes carry, and are syntax no document may write. [The plan command](./plan-command-spec.md) is the contract. @@ -9088,6 +9099,10 @@ A single Effection Api named `DocumentOutput` with one operation: `output`. The is the system's public surface — extensible to progress, printed errors, etc. as needs grow. +`exact` says whether the text is exact bytes rather than prose. Absent means +prose, which is what every ordinary emission is; the presentation middleware +(§9.4, §9.5) passes an exact write through untouched. + ```typescript // src/api.ts @@ -9095,11 +9110,11 @@ import type { Operation } from "effection"; import { createApi } from "./api.ts"; export interface DocumentOutputApi { - output(text: string): Operation; + output(text: string, exact?: boolean): Operation; } export const DocumentOutput = createApi("DocumentOutput", { - *output(_text: string): Operation {}, + *output(_text: string, _exact?: boolean): Operation {}, }); export const { output } = DocumentOutput.operations; @@ -9157,7 +9172,17 @@ export function* useNormalizedOutput(): Operation { const scope = yield* useScope(); scope.around(DocumentOutput, { - *output([text], next) { + *output([text, exact], next) { + // Exact bytes are not prose. Their whitespace is part of what the run + // produced, so the write goes out as it arrived — while its own trailing + // newlines still count, because the prose after it is still prose. + if (exact === true) { + const trailing = text.match(/\n+$/); + trailingNewlines = trailing ? trailing[0].length : 0; + yield* next(text, true); + return; + } + let normalized = text; // Strip trailing whitespace on each line @@ -9176,7 +9201,7 @@ export function* useNormalizedOutput(): Operation { const match = normalized.match(/\n+$/); trailingNewlines = match ? match[0].length : 0; - yield* next(normalized); + yield* next(normalized, exact); }, }); } @@ -9186,6 +9211,15 @@ Mutable closure state (`trailingNewlines`) is safe because the middleware is scoped per `useNormalizedOutput()` call — one instance per document run, not shared across concurrent scopes. +**An exact write is one write, not a mode.** `output()` carries whether the text +is exact bytes or prose, so the bypass applies to that write alone and the +middleware presents everything around it exactly as it did before. The terminal +formatter (§9.5) makes the same distinction, because rendering a program's +source as Markdown would turn its headings into headings and its fences into +boxes. Where a write comes from is §9.1: the emission loop reads it from the +segment, and only a component the host declared as rendering source produces +one (§5.3). + ### 9.5 Terminal ANSI formatting middleware **File:** `src/output/terminal.ts` @@ -9205,9 +9239,15 @@ export function* useTerminalOutput(): Operation { const scope = yield* useScope(); scope.around(DocumentOutput, { - *output([text], next) { + *output([text, exact], next) { + // Exact bytes go out as they arrived: a program's approved source must + // never be turned into rendered Markdown. + if (exact === true) { + yield* next(text, true); + return; + } const formatted = marked.parse(text, { async: false }) as string; - yield* next(formatted); + yield* next(formatted, exact); }, }); } diff --git a/specs/plan-command-spec.md b/specs/plan-command-spec.md index 8f784f9eb..829134ea9 100644 --- a/specs/plan-command-spec.md +++ b/specs/plan-command-spec.md @@ -251,8 +251,8 @@ They are not Plan root props and consume none of the approved Plan's property sources. **The root is an adapter, not the workflow.** It projects `props.request` into -`` without adding whitespace, supplies `props.session`, binds the returned -string and returns it. It contains no Prompt, no check, no review, no revision +`` without adding whitespace, supplies `props.session`, captures the exact +source the Component renders, and returns it. It contains no Prompt, no check, no review, no revision and no ending of its own. `props.syntax` is sealed by the host rather than forwarded: the catalog the agent is shown is the one this command rendered, and no prop on the adapter could supply another. @@ -260,7 +260,7 @@ no prop on the adapter could supply another. ## The packaged `` Component Everything the command used to hold is `packages/cli/src/documents/Plan.md`, the -one packaged Markdown value component the public `` name resolves to +one packaged Markdown text component the public `` name resolves to ([executable MDX](./executable-mdx-spec.md) §5.3). The command and an ordinary document invoking `` expand the same bytes, under the same origin `@executablemd/cli/Plan.md` and the same digest, in every distribution. There is @@ -292,12 +292,16 @@ run. TypeScript supplies neither the words nor the choice between them. **The four private capabilities.** The Component's phases are components only these exact bytes may write, declared by the host with the definition and revoked with -the execution: `` freezes the catalog, the session placement, the -surface and whether that placement outlives the invocation; paired +the execution: `` freezes the catalog, the instruction identity, the +session placement, the surface and whether that placement outlives the +invocation, and refuses a continuation whose instructions render differently — +as stale input, before a directory, a provider, a turn or a review exists; paired `` installs the constrained frame and does not return until every part of it has torn down; `` answers about one draft without executing it; and `` structurally admits the approved bytes after that -teardown. +teardown and retains them as one Plan artifact — the invocation identity, the +instruction identity, the approved source, its digest and that successful +admission — before the Component renders them. Whether the placement is durable is carried across that boundary rather than re-derived, because `` is the last thing that sees the public @@ -445,8 +449,8 @@ and prevents final validation and every way a Plan could leave the command. An Agent reply is an inert string while the Plan is being written. The command document may bind it, pass it to the validator, serialize its problems, present -it with `` and return it. It never evaluates the draft and never -dynamically imports it. Only after approval, teardown and final host validation +it with `` and produce it as source. It never evaluates the draft and +never dynamically imports it. Only after approval, teardown and final host validation may those exact bytes enter ordinary execution. ## Host-declared draft validation @@ -581,15 +585,16 @@ provider answers with — there is no internal spelling behind them: | 1–9 | passed its check | **Approve**, **Request changes**, **Stop** | | 1–9 | problems remain | **Request changes**, **Stop** | | 10 | passed its check | **Approve**, **Stop** | -| 10 | problems remain | **Explain what went wrong**, **Stop** | +| 10 | problems remain | *no review — see the explanation turn below* | **Request changes** requires non-empty feedback, sends one complete-replacement request through the same enclosing Session, and resets the three-turn repair budget. -**The explanation turn.** **Explain what went wrong** is offered only on a tenth -draft that still has problems, and makes exactly one more `` in the same -enclosing Session. The Session already holds the original Prompt, the catalog, +**The explanation turn.** A tenth draft that still has problems after its +repairs leaves nothing to approve and nothing left to revise into, so no review +opens for it: there is no decision to offer. The workflow instead makes exactly +one more `` in the same enclosing Session, automatically. The Session already holds the original Prompt, the catalog, every draft, every earlier diagnostic and every revision request, so nothing is resent: the turn carries only the final diagnostics, which were produced after the agent's last draft and have not appeared in the conversation. It asks for a @@ -612,16 +617,15 @@ returns its source unchanged. Stopping reaches `` with one of two authored messages, and which one depends on whether an approvable Plan ever existed: -- **Stop** on a tenth draft that still has problems is **exhaustion** — ten - drafts were reviewed and none was approved; -- **Explain what went wrong**, offered only there, makes one more turn (below) - and then ends the same way; -- every other **Stop**, including one on a tenth draft that could have been - approved, is the ordinary ending: you decided to stop. +- a tenth draft that still has problems is **exhaustion** — ten drafts were + written and none was approvable — and it ends through the automatic + explanation above rather than through a decision; +- every **Stop**, including one on a tenth draft that could have been approved, + is the ordinary ending: you decided to stop. The branch after the Session is only an unexpected-no-decision fallback and says -so; exhaustion is decided inside review, not duplicated there. Failure is authored in Markdown -rather than hidden in the host or represented by a missing-`` accident. +so; exhaustion is decided where the tenth draft's check is, not duplicated +there. Failure is authored in Markdown rather than hidden in the host. The command document's rendered output is not command output. Everything you see while a Plan is written reaches you through Elicitation, and that document's @@ -713,7 +717,7 @@ produced. | --- | --- | --- | | C1 | Fixed grammar and help | Prompt cardinality, individual-property ordering, aggregate props before the Prompt, `--session` including its empty-value refusal, run-only flags refused without `--run` before any authorship or filesystem effect, a first token of `prompt` refused in preflight rather than read as a document path — leaving `xmd run ./prompt` still able to execute a document of that name — and effect-free generic help that explains `--output` and `--run` | | C2 | Exact packaged adapter and Component | The command executes the checked-in Markdown value root under ``, and that root only projects the request into the packaged `` Component and returns what it approved; the turn text is that Component's own words, and no TypeScript authorship loop or custom root supplies them | -| C3 | Visible authorship workflow | Generation, three-turn repair, ten-round review, revision, approval, stopping, exhaustion and the explanation turn are present in the Component's Markdown under visible headings; the generation, repair and revision instructions each require the descriptive title and the steps-beside-components structure; `` remains one turn | +| C3 | Visible authorship workflow | Generation, three-turn repair, ten-round review, revision, approval, stopping, exhaustion and the automatic explanation turn are present in the Component's Markdown under visible headings; the generation, repair and revision instructions each require the descriptive title and the steps-beside-components structure; `` remains one turn | | C4 | One Session | One enclosing Session expansion carries every turn; two default invocations get different profile directories and session keys, two `--session` invocations get the same directory and key with the raw name absent from the path, and two named invocations sharing one ACPX store continue the established record rather than placing a second | | C5 | Authorship profile ceiling | This session's own host-owned directory, empty while the command document runs, no MCP servers, no native tools, strict private denial, no Files/command/network capability for the command document, and final-run permission flags that cannot widen any of it; pre-existing content in that directory refuses before any provider, session, turn, review, result or execution and is left untouched | | C6 | Draft inertness | A draft is only data while the Plan is written, and no draft effect occurs before the final execution | From 7698660cb56538fa1087c7337090daf9f0d5485a Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Wed, 2 Sep 2026 17:26:43 -0400 Subject: [PATCH 2/6] =?UTF-8?q?=F0=9F=94=92=20Close=20the=20exact-source?= =?UTF-8?q?=20disposition=20and=20both=20Plan=20record=20protocols=20(#722?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The trusted declaration's exact-source disposition is durable. It is retained in the declared-markdown selection as the closed shape `exact: true`, absence meaning ordinary prose, hostile-parsed as a fifth member the record may or may not carry, and compared against the current declaration including absence. A continuation whose origin, digest and bytes are unchanged but whose disposition moved is refused before it can publish those bytes — as prose where a program was recorded, or the reverse. Both Plan records are closed protocols. The frozen inputs hold exactly `syntax` and `instruction`; the artifact holds exactly `invocation`, `instruction`, `source`, `digest` and `admission`. A member missing, a member this version does not know, or a member of the wrong type is a record that cannot be read rather than one to fill a default into, and the invocation identity, the source digest and the successful admission are still checked on top of that shape. Both unreadable-record refusals are raised as durability failures, for the reason the changed-instruction refusal already was: they happen while the journal still holds entries the run will never reach, and an ordinary failure there is replaced by the completion's divergence report — so the person read a count of unreached entries instead of what was wrong with their history. Terminal formatting is held to the same bypass as whitespace normalization, and both directions are now proven: exact bytes keep their heading, fence, emphasis markers, trailing spaces and blank runs, while the prose on either side of the write is still formatted. Claude-Session: https://claude.ai/code/session_01DNPFLq52HVSu3NH6WczYLf --- packages/cli/src/plan-component.ts | 39 ++++- .../tests/document-suites/plan/Plan.test.md | 19 ++- packages/cli/tests/plan-component.test.ts | 139 +++++++++++++++++- packages/core/src/execute.ts | 45 +++++- .../tests/declared-markdown-component.test.ts | 49 ++++++ packages/core/tests/output-terminal.test.ts | 50 ++++++- 6 files changed, 324 insertions(+), 17 deletions(-) diff --git a/packages/cli/src/plan-component.ts b/packages/cli/src/plan-component.ts index 91d9c2d5a..042904592 100644 --- a/packages/cli/src/plan-component.ts +++ b/packages/cli/src/plan-component.ts @@ -403,7 +403,7 @@ function planInputs(assembly: PlanComponentAssembly): IdentityComponent { // A history is input, so it is parsed rather than trusted. const retained = readInputs(frozen); if (retained === undefined) { - throw new Error(UNREADABLE_INPUTS); + throw new StaleInputError(UNREADABLE_INPUTS); } if (retained.instruction !== instruction) { throw new StaleInputError(STALE); @@ -632,17 +632,37 @@ interface RetainedInputs { readonly instruction: string; } +/** + * The frozen inputs a record holds, read as a closed protocol. + * + * Exactly two members, both strings. A record missing one, carrying a member + * this version does not know, or holding one of the wrong type is a record this + * version cannot read — not one to fill in a default for, because every default + * here is a guess about what an earlier run actually asked. + */ function readInputs(value: Json): RetainedInputs | undefined { if (typeof value !== "object" || value === null || Array.isArray(value)) { return undefined; } const { syntax, instruction } = value; + if (Object.keys(value).length !== 2) { + return undefined; + } if (typeof syntax !== "string" || typeof instruction !== "string") { return undefined; } return { syntax, instruction }; } +/** + * What a record this version cannot read is refused with. + * + * A `StaleInputError`, like the changed-instruction refusal and for the same + * reason: it is raised while the journal still holds entries this run will now + * never reach, and an ordinary failure there is replaced by the completion's + * divergence report — so the person would read a count of unreached entries + * instead of what was actually wrong with their history. + */ const UNREADABLE_INPUTS = "the retained Plan inputs cannot be read as Plan inputs, so no Plan source was produced."; @@ -678,12 +698,23 @@ interface PlanArtifact { readonly admission: "valid"; } -/** What a record holds, or nothing when it is not one of these. */ +/** + * What a record holds, read as a closed protocol. + * + * Exactly five members. A record missing one, carrying a sixth this version does + * not know, or holding one of the wrong type is refused rather than read + * partially: this record is the only thing standing between a continuation and + * publishing bytes nobody approved, so a member it cannot account for is a + * reason to stop. + */ function readArtifact(value: Json): PlanArtifact | undefined { if (typeof value !== "object" || value === null || Array.isArray(value)) { return undefined; } const { invocation, instruction, source, digest, admission } = value; + if (Object.keys(value).length !== 5) { + return undefined; + } if (typeof invocation !== "string" || typeof instruction !== "string") { return undefined; } @@ -737,10 +768,10 @@ function admitPlan( const artifact = readArtifact(retained); if (artifact === undefined) { - throw new Error(UNREADABLE_ARTIFACT); + throw new StaleInputError(UNREADABLE_ARTIFACT); } if (artifact.invocation !== id || artifact.digest !== sourceDigest(artifact.source)) { - throw new Error(UNREADABLE_ARTIFACT); + throw new StaleInputError(UNREADABLE_ARTIFACT); } if (artifact.instruction !== instruction) { throw new StaleInputError(STALE); diff --git a/packages/cli/tests/document-suites/plan/Plan.test.md b/packages/cli/tests/document-suites/plan/Plan.test.md index 8fb3719e0..03b3664ae 100644 --- a/packages/cli/tests/document-suites/plan/Plan.test.md +++ b/packages/cli/tests/document-suites/plan/Plan.test.md @@ -96,9 +96,18 @@ component does not exist. ## The bare form emits the program's source Written without `as`, `` emits the approved program where the component -appears. The source is bytes rather than prose, so what reaches the run's output -is what the agent wrote — including a line ending in spaces and a run of blank -lines that the default output normalization rewrites in ordinary Markdown. +appears, and what a reader of the run sees is what the agent wrote. + +**This host installs no presentation middleware**, so what these two cases prove +is the raw author-facing path: which form emits, which binds, and that the bytes +arrive whole. They do not prove that the presentation an ordinary `xmd run` +installs leaves those bytes alone — nothing here would fail if it did not. +That discrimination is owned elsewhere, by cases that install the real +middleware and fail when the bypass is removed: `PC19` in +`packages/cli/tests/plan-component.test.ts` for whitespace normalization on the +emitted source, `WN8`/`WN9` in `packages/core/tests/output-normalize.test.ts` +for the normalizer itself, and `TF6`/`TF7` in +`packages/core/tests/output-terminal.test.ts` for terminal formatting. @@ -115,8 +124,8 @@ lines that the default output normalization rewrites in ordinary Markdown. -The trailing spaces and the four newlines survived, which is what says these -bytes were emitted as source rather than presented as prose. +The trailing spaces and the four newlines arrived intact, which is what says the +approved bytes reached the document's output whole. in an ordinary document", () => { return undefined; } + /** + * The same history, with one Plan record's retained value replaced. + * + * A continuation reads what the journal holds, and what it holds is not + * guaranteed to be what this version wrote — a record can be truncated, edited, + * or written by a build that knew a different protocol. Rewriting it here is how + * a case asks what the reader does with one it cannot account for. + */ + function* tampered( + stream: InMemoryStream, + prefix: string, + replace: (value: Json) => Json, + ): Operation { + const partial = new InMemoryStream(); + for (const event of yield* stream.readAll()) { + if (event.type === "close") { + continue; + } + if ( + event.type === "yield" && + event.description.name.startsWith(prefix) && + event.result.status === "ok" + ) { + yield* partial.append({ + ...event, + result: { status: "ok", value: replace(event.result.value ?? null) }, + }); + continue; + } + yield* partial.append(event); + } + return partial; + } + + /** One approved run, whose journal a hostile continuation then reads. */ + function* approvedRun(): Operation { + const stream = new InMemoryStream(); + const run = yield* runDocument({ + source: ['Write a program.', "", "got: {approved}", ""].join("\n"), + reply: PLAN, + stream, + }); + expect(run.failure).toBe(undefined); + return stream; + } + + /** What a continuation reading that history produced. */ + function* continued(stream: InMemoryStream): Operation { + return yield* runDocument({ + source: ['Write a program.', "", "got: {approved}", ""].join("\n"), + reply: PLAN, + reviews: [], + stream, + }); + } + it("PC19: the emitted source survives the CLI's own output normalization", function* () { yield* useWorkingDirectory(function* (dir) { // Whitespace the normalizer rewrites in prose: a line ending in spaces, @@ -658,11 +714,22 @@ describe("Tier PC — in an ordinary document", () => { }); expect(run.failure).toBe(undefined); - // Retained before either form could publish the bytes. + // Retained before either form could publish the bytes: all five members, + // and exactly those five. A sixth would be something a later reader has + // to account for, and a missing one is a record that cannot be read. const artifact = Object(yield* retainedArtifact(stream)); + expect(Object.keys(artifact).sort()).toEqual([ + "admission", + "digest", + "instruction", + "invocation", + "source", + ]); expect(Reflect.get(artifact, "source")).toBe(PLAN); expect(Reflect.get(artifact, "digest")).toBe(sourceDigest(PLAN)); expect(Reflect.get(artifact, "admission")).toBe("valid"); + expect(typeof Reflect.get(artifact, "invocation")).toBe("string"); + expect(typeof Reflect.get(artifact, "instruction")).toBe("string"); }); }); @@ -808,6 +875,76 @@ describe("Tier PC — in an ordinary document", () => { }); }); + it("PC25: a retained inputs record this version cannot read produces nothing", function* () { + yield* useWorkingDirectory(function* () { + // Three ways a record stops being one: a member gone, a member added, and + // a member of the wrong type. Each is refused with the same fixed + // sentence, and none of them produces source or a binding. + const cases: [string, (value: Json) => Json][] = [ + ["a member is missing", (value) => ({ syntax: Object(value).syntax })], + [ + "a member this version does not know was added", + (value) => ({ ...Object(value), extra: "surprise" }), + ], + ["a member has the wrong type", (value) => ({ ...Object(value), instruction: 7 })], + ]; + + for (const [, replace] of cases) { + const run = yield* continued( + yield* tampered(yield* approvedRun(), "plan:inputs:", replace), + ); + + expect(run.failure).toContain("cannot be read as Plan inputs"); + expect(run.output).not.toContain("got:"); + expect(run.output).not.toContain("# Say hello"); + expect(run.emitted).not.toContain("# Say hello"); + } + }); + }); + + it("PC26: a retained artifact this version cannot read produces nothing", function* () { + yield* useWorkingDirectory(function* () { + const cases: [string, (value: Json) => Json][] = [ + [ + "a member is missing", + (value) => { + const { digest: _digest, ...rest } = Object(value); + return rest; + }, + ], + [ + "a member this version does not know was added", + (value) => ({ ...Object(value), extra: "surprise" }), + ], + ["a member has the wrong type", (value) => ({ ...Object(value), source: 7 })], + [ + "the admission is not the one this version accepts", + (value) => ({ ...Object(value), admission: "invalid" }), + ], + [ + "the digest does not describe the source it sits beside", + (value) => ({ ...Object(value), digest: sourceDigest("something else") }), + ], + [ + "the record belongs to another invocation", + (value) => ({ ...Object(value), invocation: "somebody-else" }), + ], + ]; + + for (const [, replace] of cases) { + const run = yield* continued( + yield* tampered(yield* approvedRun(), "plan:artifact:", replace), + ); + + expect(run.failure).toContain("cannot be read as one"); + // No source reached the document either way it could have. + expect(run.output).not.toContain("got:"); + expect(run.output).not.toContain("# Say hello"); + expect(run.emitted).not.toContain("# Say hello"); + } + }); + }); + it("PC14: nothing the Component does reaches the caller's filesystem", function* () { yield* useWorkingDirectory(function* (dir) { const run = yield* runDocument({ diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index ef2f56952..7611eb899 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -221,7 +221,22 @@ type DurableSelection = * host that no longer declares it, refuses rather than continuing somebody * else's policy. */ - | { kind: "declared-markdown"; origin: string; digest: string; content: string } + | { + kind: "declared-markdown"; + origin: string; + digest: string; + content: string; + /** + * Whether the host declared this component's rendering to be source. + * + * Closed: present only as `true`, and absent means the ordinary prose + * disposition. It is retained because it decides how the bytes this + * component produces are published — a continuation that resumed under a + * different answer would present a program as prose, or prose as a + * program, from the same origin and the same digest. + */ + exact?: true; + } /** * A component only the declaration that carries it may write. * @@ -269,15 +284,28 @@ function readDurableSelection(value: unknown): DurableSelection | undefined { const origin = record["origin"]; const digest = record["digest"]; const declaredContent = record["content"]; + // Four members, or five when the optional disposition is present. Anything + // else — a member this version does not know, or `exact` written as + // anything but `true` — is a record this version cannot read rather than + // one it may guess at. + const exact = record["exact"]; + const withExact = Object.hasOwn(record, "exact"); if ( - members !== 4 || + members !== (withExact ? 5 : 4) || typeof origin !== "string" || typeof digest !== "string" || - typeof declaredContent !== "string" + typeof declaredContent !== "string" || + (withExact && exact !== true) ) { return undefined; } - return { kind: "declared-markdown", origin, digest, content: declaredContent }; + return { + kind: "declared-markdown", + origin, + digest, + content: declaredContent, + ...(withExact ? { exact: true } : {}), + }; } const path = record["path"]; @@ -450,6 +478,9 @@ function* durableImportComponent( origin: selected.origin, digest: selected.digest, content: selected.source, + // Recorded only when it holds, so an ordinary declaration's record + // is exactly what it always was. + ...(selected.definition.exact === true ? { exact: true } : {}), }; case "registered": return { @@ -517,11 +548,15 @@ function* durableImportComponent( if (selection.kind === "declared-markdown") { const declaration = declared?.component(name); + // The disposition compares like every other term, absence included: a host + // that added or removed it is publishing the same bytes a different way, + // which is a different answer to the question this run already recorded. if ( declaration === undefined || declaration.origin !== selection.origin || declaration.digest !== selection.digest || - declaration.source !== selection.content + declaration.source !== selection.content || + (declaration.definition.exact === true) !== (selection.exact === true) ) { throw new Error( `Component ${name} was recorded as the declared Markdown "${selection.origin}", which is ` + diff --git a/packages/core/tests/declared-markdown-component.test.ts b/packages/core/tests/declared-markdown-component.test.ts index d7fc0415d..7e871422e 100644 --- a/packages/core/tests/declared-markdown-component.test.ts +++ b/packages/core/tests/declared-markdown-component.test.ts @@ -1151,6 +1151,55 @@ describe("Tier DM — selection is journaled and replays", () => { expect(message).toContain("recorded as the declared Markdown"); }); + it("DM26b: the record carries the exact-source disposition when the host declares one", function* () { + const stream = new InMemoryStream(); + yield* run("\n", [declared(POLICY_SOURCE, { exact: true })], [], stream); + + const selection = (yield* stream.readAll()) + .filter(isImport) + .find((event) => event.type === "yield" && event.description.name === "Policy"); + + // Closed: `exact: true` beside the four members an ordinary record has. + expect(selection?.type === "yield" && selection.result).toEqual({ + status: "ok", + value: { + kind: "declared-markdown", + origin: ORIGIN, + digest: sourceDigest(POLICY_SOURCE), + content: POLICY_SOURCE, + exact: true, + }, + }); + }); + + it("DM26c: a continuation that dropped the exact disposition refuses", function* () { + const first = new InMemoryStream(); + yield* run("\n", [declared(POLICY_SOURCE, { exact: true })], [], first); + + // The same name, the same origin, the same digest and the same bytes. Only + // how they are published changed, which is enough: prose is not what this + // run recorded producing. + const message = yield* refusal( + run("\n", [declared(POLICY_SOURCE)], [], yield* continuing(first)), + ); + + expect(message).toContain("recorded as the declared Markdown"); + }); + + it("DM26d: a continuation that added the exact disposition refuses", function* () { + const first = new InMemoryStream(); + yield* run("\n", [declared(POLICY_SOURCE)], [], first); + + // The other direction, which absence has to be compared for rather than + // defaulted: a host that started calling these bytes source is answering + // differently from the run being continued. + const message = yield* refusal( + run("\n", [declared(POLICY_SOURCE, { exact: true })], [], yield* continuing(first)), + ); + + expect(message).toContain("recorded as the declared Markdown"); + }); + it("DM27: an ordinary execute() declares nothing and resolves no declared name", function* () { const message = yield* refusal( scoped(function* () { diff --git a/packages/core/tests/output-terminal.test.ts b/packages/core/tests/output-terminal.test.ts index 35f17914d..68166c5f3 100644 --- a/packages/core/tests/output-terminal.test.ts +++ b/packages/core/tests/output-terminal.test.ts @@ -16,7 +16,7 @@ import { subscribe } from "../src/subscribe.ts"; /** * Helper: install terminal middleware + capture handler, emit text, collect. */ -function* collectTerminal(texts: string[]): Operation { +function* collectTerminal(texts: (string | [string, boolean])[]): Operation { const channel = createChannel(); // First: terminal formatting (outermost) yield* useTerminalOutput(); @@ -32,7 +32,11 @@ function* collectTerminal(texts: string[]): Operation { yield* ready; for (const text of texts) { - yield* DocumentOutput.operations.output(text); + if (typeof text === "string") { + yield* DocumentOutput.operations.output(text); + continue; + } + yield* DocumentOutput.operations.output(text[0], text[1]); } yield* channel.close(); @@ -90,4 +94,46 @@ describe("Tier TF — Terminal ANSI formatting", () => { expect(captured[0]).not.toContain("**"); expect(captured[0]).toContain("bold"); }); + + // TF6: An exact write is source, and source is never rendered + it("TF6: exact bytes pass through byte for byte", function* () { + // Everything this formatter would otherwise transform: a heading, a fenced + // block, emphasis markers, a line ending in spaces, and a run of blank + // lines. As a program's source, all of it has to arrive unchanged — a + // heading turned into a styled heading is no longer source anybody can run. + const exact = [ + "# Approved program", + "", + "Writes **evidence** when something runs it. ", + "", + "", + "", + "```markdown", + ' ran', + "```", + "", + ].join("\n"); + + const result = yield* collectTerminal([[exact, true]]); + + expect(result).toEqual([exact]); + // The emphasis markers inside the source survived, which is the same fact + // TF2 proves this formatter removes from prose. + expect(result[0]).toContain("**evidence**"); + }); + + // TF7: The bypass is per write, not a switch + it("TF7: prose around an exact write is still formatted", function* () { + const exact = "keep **these** markers \n"; + const result = yield* collectTerminal(["**before**\n", [exact, true], "**after**\n"]); + + // The prose on either side went through marked-terminal as it always has. + expect(result[0]).not.toContain("**"); + expect(result[0]).toContain("before"); + expect(result[2]).not.toContain("**"); + expect(result[2]).toContain("after"); + // The write between them did not: its markers and its trailing spaces are + // exactly what was written. + expect(result[1]).toBe(exact); + }); }); From aee7ba8465abfe9d15ee182f6134d83d7f420e3c Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Wed, 2 Sep 2026 17:52:48 -0400 Subject: [PATCH 3/6] =?UTF-8?q?=E2=9C=85=20Hold=20the=20compiled=20binary?= =?UTF-8?q?=20to=20the=20text-mode=20``=20(#722)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The compiled-binary probe still expected a declared return from the packaged Component, and the `smoke` job is where that surfaced: import-based test selection cannot see a subprocess boundary, so no local affected-test run reaches it. The binary reports what it embedded, and what it embedded is now a text component. The origin and digest assertions above the failure had already passed, which is how we know the compile carried the new Component rather than an older copy — the expectation was stale, not the build. The scripted-review type loses the decision that went with the review a tenth unrepairable draft no longer reaches. Nothing scripts it any more. Claude-Session: https://claude.ai/code/session_01DNPFLq52HVSu3NH6WczYLf --- packages/cli/tests/support/plan-harness.ts | 2 +- scripts/tests/plan-component-compiled.test.ts | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/packages/cli/tests/support/plan-harness.ts b/packages/cli/tests/support/plan-harness.ts index 5090583e4..4bba55517 100644 --- a/packages/cli/tests/support/plan-harness.ts +++ b/packages/cli/tests/support/plan-harness.ts @@ -57,7 +57,7 @@ export const ADAPTERS: EmbeddedAdapters = createEmbeddedAdapters( * with: the workflow keeps no internal spelling behind them. */ export interface ScriptedReview { - decision: "Approve" | "Request changes" | "Stop" | "Explain what went wrong"; + decision: "Approve" | "Request changes" | "Stop"; feedback?: string; /** Answer with this instead, to drive a response the schema rejects. */ raw?: unknown; diff --git a/scripts/tests/plan-component-compiled.test.ts b/scripts/tests/plan-component-compiled.test.ts index faa2499fd..7ba812717 100644 --- a/scripts/tests/plan-component-compiled.test.ts +++ b/scripts/tests/plan-component-compiled.test.ts @@ -82,7 +82,10 @@ describe("compiled xmd", { sanitizeOps: false, sanitizeResources: false }, () => digest, }); expect(plan.forms).toEqual(["paired"]); - expect(plan.returnMode).toBe("value"); + // A text component: what it renders is the approved program source, so a + // build still reporting a declared return is one that embedded the bytes + // from before this stack. + expect(plan.returnMode).toBe("text"); // And the private capabilities are not syntax any build lets a document // write. From e474dff996273a7dc3aab81358b6cb0b7aeaf27d Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Wed, 2 Sep 2026 18:21:29 -0400 Subject: [PATCH 4/6] =?UTF-8?q?=F0=9F=94=92=20Make=20exact=20source=20a=20?= =?UTF-8?q?provenance=20rather=20than=20a=20field=20(#722)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exact-source presentation was read from `ComponentDefinition.exact` and from `TextSegment.exact` — public data on objects an import can produce. Ordinary `Component.importComponent` middleware could therefore answer an open name with a definition of its own carrying the disposition, and core would publish that handler's bytes unpresented. It is now derived from two facts no answer can write: that canonical execution authorized the import for a name this execution closed, and that this host's own admitted declaration for that name states exact source. The admission holds the disposition, because a declaration crosses on an ExecutionInstallation by value before any middleware exists, while a definition is an object a handler can copy and answer with. Which segments an expansion produced as source is recorded against those objects' identity rather than on them. A segment scanned from a document, parsed from a journal or written by a handler is simply not in that record — where a field would have been supplied, membership cannot be. Both public opt-ins are gone rather than replaced: there is no `exact` on `ComponentDefinition` or `TextSegment`, and no new one anywhere a document, package, frontmatter or middleware can reach. The durable protocol is unchanged and now stated as the two closed shapes it is, with every other spelling refused. The specification and architecture say what the implementation does, and the acceptance tables gain the rows this work introduced: a Tier DM table for declared-Markdown selection and exact source, and the WN and TF rows for the two presentation bypasses. Claude-Session: https://claude.ai/code/session_01DNPFLq52HVSu3NH6WczYLf --- architecture.md | 25 ++- .../core/src/components/declared-markdown.ts | 30 +++- packages/core/src/components/select.ts | 1 + packages/core/src/execute.ts | 7 +- packages/core/src/expand.ts | 39 +++-- packages/core/src/output/exact-source.ts | 45 +++++ packages/core/src/types.ts | 26 +-- .../tests/declared-markdown-component.test.ts | 156 +++++++++++++++++- specs/executable-mdx-spec.md | 64 +++++-- 9 files changed, 338 insertions(+), 55 deletions(-) create mode 100644 packages/core/src/output/exact-source.ts diff --git a/architecture.md b/architecture.md index 170f17c26..85f8649fa 100644 --- a/architecture.md +++ b/architecture.md @@ -3561,15 +3561,28 @@ registration can answer for it — one decision, shared by execution, inspection and validation, so a repository file under a private name is described by none of them and runs nowhere. -**The journal records the asset, not a lookup.** A declared import records -exactly `{ kind: "declared-markdown", origin, digest, content }`, and a private -one records exactly `{ kind: "declared-private", origin }`. A continuation reads -both as hostile data, verifies the recorded origin, digest and bytes against -what this run declares, and reads no file: a host that no longer declares the -name, or that declares different bytes under it, refuses rather than continuing +**The journal records the asset, not a lookup.** A declared import records one +of two closed shapes — `{ kind, origin, digest, content }`, and the same with +`exact: true` for a component the host declared as rendering a program's source +— and a private one records exactly `{ kind: "declared-private", origin }`. A +continuation reads all of it as hostile data, verifies the recorded origin, +digest, bytes and disposition against what this run declares, and reads no file: +a host that no longer declares the name, that declares different bytes under it, +or that publishes those bytes a different way, refuses rather than continuing somebody else's Markdown. A recorded private import refuses unless the element asking for it is inside the same declaration. +**Exact source is a provenance, not a property.** Whether bytes are published as +a program's source or presented as prose is decided from two things no answer +can write: that canonical execution authorized the import for a name this +execution closed, and that the host's own admitted declaration for that name +states it. A definition carrying the claim, a segment arriving already marked, a +frontmatter key and a middleware answer are each data that reached the engine +from somewhere, so none of them decides it — an ordinary +`Component.importComponent` handler answering an open name with a definition +claiming the disposition publishes prose, exactly as a repository file of that +name does. + ## The syntax catalog boundary `xmd syntax` answers what a document may write here, and answering must cost diff --git a/packages/core/src/components/declared-markdown.ts b/packages/core/src/components/declared-markdown.ts index 777db0e6d..12a202554 100644 --- a/packages/core/src/components/declared-markdown.ts +++ b/packages/core/src/components/declared-markdown.ts @@ -125,6 +125,16 @@ export interface AdmittedDeclaredMarkdown { /** The parse of `source`, produced once and shared by every reader. */ readonly definition: ComponentDefinition; readonly privates: readonly IdentityComponent[]; + /** + * Whether the host declared this component's rendering to be source. + * + * Captured from the declaration, which crossed on an `ExecutionInstallation` + * by value before any middleware existed. It lives on the admission rather + * than on the parsed definition because a definition is an object handlers + * can copy, write and answer with, and this is a fact about *what the host + * declared* rather than about any object. + */ + readonly exact: boolean; } /** The SHA-256 of exact UTF-8 bytes, lowercase hex. */ @@ -192,9 +202,7 @@ export function* admitDeclaredMarkdown( ); } - const parsed = yield* parseMarkdownDefinition(name, origin, source); - const definition: ComponentDefinition = - declaration.exact === true ? { ...parsed, exact: true } : parsed; + const definition = yield* parseMarkdownDefinition(name, origin, source); const forms = declaration.forms ?? BOTH_FORMS; const badForms = formsRefusal(forms); @@ -257,6 +265,7 @@ export function* admitDeclaredMarkdown( forms, definition, privates, + exact: declaration.exact === true, }); } @@ -483,6 +492,21 @@ export class DeclaredImports implements ImportTier { return this.#privates.has(name); } + /** + * Whether the host declared this name as a component that renders source. + * + * Answered from the admission — what the host stated before any middleware + * existed — rather than from the definition an import produced, so a handler + * that answers this name with a definition of its own gets whatever the host + * declared about the *name*, and a handler answering an undeclared name gets + * `false` because the host declared nothing about it. It is one half of the + * question; the caller supplies the other half by asking only for an import + * canonical execution authorized. + */ + declaresExact(name: string): boolean { + return this.#catalog.component(name)?.exact === true; + } + /** * Offer one private import, for the duration of one ask. * diff --git a/packages/core/src/components/select.ts b/packages/core/src/components/select.ts index fe132d630..e6970712c 100644 --- a/packages/core/src/components/select.ts +++ b/packages/core/src/components/select.ts @@ -146,6 +146,7 @@ export function* selectComponent( source: declared.source, forms: declared.forms, definition: declared.definition, + exact: declared.exact, }; } diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 7611eb899..f84b8366e 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -96,6 +96,7 @@ import { import { Component, importComponent, raise } from "./component-api.ts"; import { sourceDescription } from "./source-position.ts"; import { renderSegment } from "./render.ts"; +import { isExactSource } from "./output/exact-source.ts"; import { DocumentOutput } from "./api.ts"; import { composeBoundExecChain, @@ -480,7 +481,7 @@ function* durableImportComponent( content: selected.source, // Recorded only when it holds, so an ordinary declaration's record // is exactly what it always was. - ...(selected.definition.exact === true ? { exact: true } : {}), + ...(selected.exact ? { exact: true } : {}), }; case "registered": return { @@ -556,7 +557,7 @@ function* durableImportComponent( declaration.origin !== selection.origin || declaration.digest !== selection.digest || declaration.source !== selection.content || - (declaration.definition.exact === true) !== (selection.exact === true) + declaration.exact !== (selection.exact === true) ) { throw new Error( `Component ${name} was recorded as the declared Markdown "${selection.origin}", which is ` + @@ -1792,7 +1793,7 @@ function* runValueRoot( */ /** Whether one expanded segment carries exact bytes rather than prose. */ function exactly(segment: Segment): boolean { - return segment.type === "text" && segment.exact === true; + return isExactSource(segment); } /** diff --git a/packages/core/src/expand.ts b/packages/core/src/expand.ts index 992f61988..ebf412e43 100644 --- a/packages/core/src/expand.ts +++ b/packages/core/src/expand.ts @@ -123,6 +123,7 @@ import { declareChildAnswers, expandAnswers, strayAnswerError } from "./answers. import { DeclarationScan } from "./declaration-scan.ts"; import { RESERVED_STRUCTURAL } from "./structural.ts"; import { renderSegments } from "./render.ts"; +import { markExactSource } from "./output/exact-source.ts"; import { layerEnvironments, layerProjectedContentEnvironment, @@ -2357,6 +2358,14 @@ function* expandComponent( const selection = authority?.identities?.beginImport(name); let selected: IdentityDomain | undefined; let dispatcher: FunctionComponent | undefined; + /** + * Whether canonical execution answered this import for a name it closed. + * + * The provenance exact source is read from. An open import's answer — the + * chain's, unverified — never sets it, so nothing a handler writes into a + * definition can reach the presentation decision below. + */ + let authorizedCanonically = false; try { // The public chain answers, and canonical execution decides whether the // answer is one it produced. In a closed execution — a workflow holding a @@ -2398,6 +2407,11 @@ function* expandComponent( imported = answered; } else { imported = authority.imports.authorize(name, answered); + // This import is canonical execution's own answer for a name this + // execution closed, which is the only provenance exact source is read + // from. An open import — one no tier claims — never sets it, however its + // answer describes itself. + authorizedCanonically = true; // Closed authorization answers with core's retained copy rather than the // object the resolver recorded, and the copy is what this expansion // invokes — so the selection is recorded against it too. Only here: an @@ -2734,13 +2748,19 @@ function* expandComponent( ); }); + // Exact source is a provenance, and both halves of it are read here from + // things no answer can write: that canonical execution authorized this import + // for a name this execution closed, and that the *host's own admitted + // declaration* for that name states exact source. A definition claiming the + // disposition, or segments arriving already marked, decide nothing. + // // What a component declared exact renders is exact wherever it renders: the // caller's flow, its own returned region, or neither when the invocation - // binds instead. Marking it here — on the segments this body produced, after - // it produced them — is what carries the fact to the emission loop, which is - // outside every scope the invocation owned. - if (definition.exact === true) { - markExact(bodyOwner === undefined ? expanded : bodyOwner.slice(renderedFrom)); + // binds instead. Recording it here — against the segments this body produced, + // after it produced them — is what carries the fact to the emission loop, + // which is outside every scope the invocation owned. + if (authorizedCanonically && authority?.declared?.declaresExact(name) === true) { + markExactSource(bodyOwner === undefined ? expanded : bodyOwner.slice(renderedFrom)); } if (asBinding) { @@ -2776,15 +2796,6 @@ function* expandComponent( return bodyOwner === undefined ? expanded : []; } -/** Say of each text segment here that its bytes are exact rather than prose. */ -function markExact(segments: readonly Segment[]): void { - for (const segment of segments) { - if (segment.type === "text") { - segment.exact = true; - } - } -} - // Without `returns`, a function component's rendering is its return value, so // anything else is a contract violation rather than something to stringify. function asText(output: Json): string { diff --git a/packages/core/src/output/exact-source.ts b/packages/core/src/output/exact-source.ts new file mode 100644 index 000000000..173514dc8 --- /dev/null +++ b/packages/core/src/output/exact-source.ts @@ -0,0 +1,45 @@ +/** + * Which rendered segments carry a program's source rather than prose. + * + * Exact presentation is a *provenance*, not a property. The question it answers + * is "did canonical execution produce these bytes by expanding a declared + * Markdown component the trusted host said renders source?" — and no object can + * answer that about itself. A definition claiming the disposition, a segment + * arriving already marked, a frontmatter key, a prop: each is data that reached + * the engine from somewhere, and any of them would let ordinary + * `Component.importComponent` middleware publish unpresented bytes by writing a + * field. + * + * So the record lives here, keyed by the identity of the segment objects the + * engine marked, and it is written only where canonical expansion has already + * decided the provenance. It is deliberately not a field on `Segment`: a field + * can be supplied, and this cannot — a segment scanned from a document, parsed + * from a journal, or built by a handler is simply not in this set. + */ + +import type { Segment } from "../types.ts"; + +/** + * The segments this engine produced from exact-source components. + * + * Weak, so a segment is remembered exactly as long as something still holds it, + * and identity-keyed, so two segments with identical content are two different + * answers. Module-level because the producer and the emission loop are the same + * loaded copy of core; nothing crosses a copy boundary, and nothing a document + * or a package can reach ever writes to it. + */ +const EXACT = new WeakSet(); + +/** Record that canonical expansion produced these segments as source. */ +export function markExactSource(segments: readonly Segment[]): void { + for (const segment of segments) { + if (segment.type === "text") { + EXACT.add(segment); + } + } +} + +/** Whether this exact segment object is one canonical expansion marked. */ +export function isExactSource(segment: Segment): boolean { + return EXACT.has(segment); +} diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index 57524b28a..3dc9214ed 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -24,16 +24,6 @@ export type Segment = export interface TextSegment { type: "text"; content: string; - /** - * Whether this text is exact bytes rather than prose. - * - * Prose is reflowed on the way out — trailing spaces go and runs of blank - * lines collapse — because that is presentation of something written to be - * read. Bytes a run produced as a program's source are not that: their - * whitespace is part of what was approved, so a segment carrying them says - * so and the emission loop hands them to the Output Api on their own. - */ - exact?: true; } export interface ComponentElement { @@ -209,14 +199,6 @@ export interface ComponentDefinition { /** Absent in text mode, where the component returns its rendered markdown. */ returns?: ReturnsSchema; bodySegments: Segment[]; - /** - * Whether what this component renders is exact bytes rather than prose. - * - * Stated by the trusted host that declares the component, never by the - * Markdown itself: a repository file cannot ask for its own rendering to - * bypass the presentation every other document is subject to. - */ - exact?: boolean; } /** @@ -380,6 +362,14 @@ export type ComponentSelection = source: string; forms: readonly InvocationForm[]; definition: ComponentDefinition; + /** + * Whether the host declared this component's rendering to be source. + * + * Read from the admission rather than from `definition`, which is an + * object an import can copy and answer with. Selection reports what the + * host declared about the *name*. + */ + exact: boolean; } | { kind: "unresolved"; searched: string[]; registered: readonly ComponentOrigin[] }; diff --git a/packages/core/tests/declared-markdown-component.test.ts b/packages/core/tests/declared-markdown-component.test.ts index 7e871422e..62bc23660 100644 --- a/packages/core/tests/declared-markdown-component.test.ts +++ b/packages/core/tests/declared-markdown-component.test.ts @@ -52,9 +52,13 @@ import { inspectComponent, inspectSyntax } from "../src/inspect.ts"; import { validateDocument, validateDocumentStructure } from "../src/document-validation.ts"; import { registerComponents } from "../src/components/registration.ts"; import { retainedSource } from "../src/root-source.ts"; +import { DocumentOutput } from "../src/api.ts"; +import { useNormalizedOutput } from "../src/output/normalize.ts"; +import { useTerminalOutput } from "../src/output/terminal.ts"; +import { isExactSource } from "../src/output/exact-source.ts"; import type { ComponentInvocation } from "../src/invocation-identity.ts"; import type { ImportedDefinition } from "../src/components/import-authority.ts"; -import type { PropsSchema } from "../src/types.ts"; +import type { PropsSchema, Segment } from "../src/types.ts"; const ROOT_PATH = "documents/root.md"; const ORIGIN = "@executablemd/test/Policy.md"; @@ -1350,3 +1354,153 @@ describe("Tier DM — structural validation is the same walk without the run's v expect(yield* validateDocumentStructure(source)).toEqual(yield* validateDocument(source)); }); }); + +/** + * Bytes that say whether they were presented or emitted. + * + * A line ending in spaces, a run of four newlines and an emphasis marker: the + * whitespace middleware rewrites the first two and the terminal middleware + * removes the third, so text that still carries all three was published as + * source rather than as prose. + */ +const PRESENTABLE = "keep **these** markers \n\n\n\nand this run\n"; + +/** Whether what a run published still carries every mark of being unpresented. */ +function unpresented(published: string): boolean { + return published.includes("**these**") && published.includes("markers \n\n\n\n"); +} + +/** + * One run, published through the presentation an ordinary `xmd run` installs. + * + * Both middlewares, in the order the CLI installs them, so a case here asks the + * question a person's terminal asks: were these bytes presented, or emitted as + * source? + */ +function* published( + source: string, + declarations: readonly DeclaredMarkdownComponent[], + extra: readonly ExecutionInstallation[], +): Operation { + const chunks: string[] = []; + return yield* scoped(function* () { + yield* useNormalizedOutput(); + yield* useTerminalOutput(); + yield* DocumentOutput.around({ + // deno-lint-ignore require-yield + *output([text]) { + chunks.push(text); + }, + }); + yield* collect( + yield* executeInstalled( + { + ...retainedSource(ROOT_PATH, source), + stream: new InMemoryStream(), + includes: [], + }, + [installation(declarations), ...extra], + ), + ); + return chunks.join(""); + }); +} + +/** A handler that answers an open name with a definition it wrote itself. */ +function answeringOpenName(definition: ImportedDefinition): ExecutionInstallation { + return { + *install() { + yield* Component.around( + { + *importComponent([name, position], next) { + if (name === "Virtual") { + return definition; + } + return yield* next(name, position); + }, + }, + { at: "max" }, + ); + }, + }; +} + +describe("Tier DM — exact source is a provenance, not a field", () => { + it("DM49: a declared component the host called exact emits its bytes unpresented", function* () { + // The positive control. Without it the two refusals below would pass for a + // build where exact source never worked at all. + const output = yield* published("\n", [declared(PRESENTABLE, { exact: true })], []); + + expect(unpresented(output)).toBe(true); + }); + + it("DM50: an ordinary declared component is presented as prose", function* () { + const output = yield* published("\n", [declared(PRESENTABLE)], []); + + expect(unpresented(output)).toBe(false); + expect(output).not.toContain("**these**"); + }); + + it("DM51: middleware answering with `exact: true` gets prose", function* () { + // The definition is the handler's own, written to claim the disposition a + // trusted declaration states. Nothing admitted it, so nothing about it is + // exact — the claim is data on an object, and the answer is prose. + const output = yield* published( + "\n", + [declared(POLICY_SOURCE)], + [ + answeringOpenName({ + kind: "markdown", + name: "Virtual", + path: "Virtual.md", + meta: {}, + props: NO_PROPS, + exact: true, + bodySegments: [{ type: "text", content: PRESENTABLE }], + } as unknown as ImportedDefinition), + ], + ); + + expect(unpresented(output)).toBe(false); + expect(output).not.toContain("**these**"); + }); + + it("DM52: a segment this engine did not mark is prose, whatever it carries", function* () { + // The other half of the same attack: not the definition claiming the + // disposition, but segments arriving already wearing the mark. Two + // assertions, because they prove different things. + // + // First the seam itself. The record of what is exact is keyed by the + // identity of the segment objects canonical expansion marked, so an object + // carrying any field at all — including the one an earlier design used — + // answers false. This is the assertion that discriminates: a marker that + // consulted a field would pass it back. + expect( + isExactSource({ type: "text", content: PRESENTABLE, exact: true } as unknown as Segment), + ).toBe(false); + expect(isExactSource({ type: "text", content: PRESENTABLE } as unknown as Segment)).toBe(false); + + // Then the end-to-end shape, which records a second fact worth keeping: + // expansion rebuilds text segments, so a field a definition wrote onto its + // own body never reaches the emission loop to begin with. That is defence + // in depth rather than the defence — the assertion above is what holds if + // expansion ever starts passing segments through by reference. + const output = yield* published( + "\n", + [declared(POLICY_SOURCE)], + [ + answeringOpenName({ + kind: "markdown", + name: "Virtual", + path: "Virtual.md", + meta: {}, + props: NO_PROPS, + bodySegments: [{ type: "text", content: PRESENTABLE, exact: true }], + } as unknown as ImportedDefinition), + ], + ); + + expect(unpresented(output)).toBe(false); + expect(output).not.toContain("**these**"); + }); +}); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 72fe5f829..1cf361b42 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2780,8 +2780,21 @@ text component's rendering is ordinarily presentation, and the output middleware reflows it and formats it. One whose rendering is a program's source is not, so what it renders is emitted exactly — the presentation middleware passes it through untouched, and prose on either side of it is presented as before -(§9.4). Only the declaring host states this. The Markdown cannot ask for it, so -a repository file of the same name is presented like every other document. +(§9.4). + +**Exact source is a provenance, not a property.** The question is whether +canonical execution produced these bytes by expanding a declared component *this +host admitted* as rendering source, and no object answers that about itself. So +it is decided from two facts an answer cannot write: that canonical execution +authorized this import for a name the execution closed, and that the admitted +declaration for that name states it. A `ComponentDefinition` carrying the claim, +a `TextSegment` arriving already marked, a frontmatter key, a prop and a +middleware answer are each data that reached the engine from somewhere, and none +of them decides anything — an ordinary `Component.importComponent` handler that +answers an open name with a definition claiming the disposition gets prose, and +so does a repository file of the same name. Which segments an expansion produced +as source is recorded against those segment objects' identity, where nothing a +document or a package can supply is able to write. **Every declaration is read once, before any installation runs.** The name, origin, source, digest, forms, prose, whether it renders exact source, and each @@ -2794,14 +2807,23 @@ declaration and then replaces a member of it — or mutates a schema it still holds — has replaced nothing: what is admitted, registered and executed is what it declared at the moment of capture. -**The journal records the asset.** A declared import records exactly -`{ kind: "declared-markdown", origin, digest, content }` and a private one -records exactly `{ kind: "declared-private", origin }`. A continuation reads -both as hostile data and verifies the recorded origin, digest and bytes against -what this run declares, reading no file: a host that no longer declares the name, -or that declares different bytes under it, refuses rather than continuing -somebody else's Markdown, and a recorded private import refuses unless the -element asking is inside the same declaration. +**The journal records the asset.** A declared import records one of exactly two +closed shapes — `{ kind, origin, digest, content }` for an ordinary declaration, +and `{ kind, origin, digest, content, exact: true }` for one the host declared +as rendering source — and a private one records exactly +`{ kind: "declared-private", origin }`. Every other spelling of the disposition +is refused: `exact: false`, `exact: "true"`, a null and any other value are not +this protocol, and neither is a record missing a member or carrying one this +version does not know. + +A continuation reads all of it as hostile data and verifies the recorded origin, +digest, bytes and disposition against what this run declares, reading no file: a +host that no longer declares the name, or that declares different bytes under +it, refuses rather than continuing somebody else's Markdown. The disposition +compares like every other term, **absence included** — a run that added or +removed it publishes the same bytes a different way, which is a different answer +to the question the recorded run already settled — and a recorded private import +refuses unless the element asking is inside the same declaration. **`` is the one of these the `run` profile has.** The CLI declares `packages/cli/src/documents/Plan.md` to every ordinary run under the origin @@ -11774,6 +11796,8 @@ Identifiers match `packages/core/tests/loop.test.ts` one to one. | WN5 | Single newline preserved | `"a\nb"` → unchanged | | WN6 | Empty write | `""` → unchanged, trailing count preserved | | WN7 | Tab trailing whitespace | `"text\t\n"` → `"text\n"` | +| WN8 | Exact bytes pass through | A write marked exact keeps its trailing spaces, its trailing tab and its run of four newlines | +| WN9 | The bypass is per write | Prose written before and after an exact write is still stripped and collapsed, and the exact write's own trailing newlines still feed the cross-write count | ### Tier TF — Terminal ANSI formatting @@ -11784,6 +11808,8 @@ Identifiers match `packages/core/tests/loop.test.ts` one to one. | TF3 | Code block formatted | Fenced code block → syntax-highlighted output | | TF4 | `async: false` | `marked.parse()` called with `{ async: false }` — no promises | | TF5 | Middleware composes with normalize | Normalized text passes through terminal formatter | +| TF6 | Exact bytes pass through | A write marked exact keeps its heading, its fence, its emphasis markers, its trailing spaces and its blank runs byte for byte | +| TF7 | The bypass is per write | Prose written before and after an exact write is still formatted, and only the exact write is untouched | ### Tier SE — Streaming emission @@ -11962,6 +11988,24 @@ drives the `xmd plan` command. | VB9 | Host false is overridden to true for one lexical subtree | | VB10 | Host true is overridden to false, the body is skipped, and sibling state is restored | +### Tier DM — Declared Markdown selection and exact source (§5.3) + +The durable selection protocol and the boundary that decides which bytes are +published as a program's source rather than presented as prose. Every row here +runs against declarations a trusted host supplied on an `ExecutionInstallation`, +and the presentation rows install the real whitespace and terminal middleware so +what they observe is what a person's terminal would show. + +| # | Test | Verify | +|---|------|--------| +| DM26b | The record carries the disposition | A declaration the host called exact records `{ kind, origin, digest, content, exact: true }`, and an ordinary one records the four-member shape unchanged | +| DM26c | A dropped disposition refuses | A continuation whose origin, digest and bytes are identical but which no longer declares the component exact refuses before publishing | +| DM26d | An added disposition refuses | The same in the other direction, so absence is compared rather than defaulted | +| DM49 | Exact source reaches the terminal | A declared exact component's bytes keep their trailing spaces, blank runs and emphasis markers through both presentation middlewares | +| DM50 | An ordinary declaration is prose | The same bytes from a declaration the host did not call exact are stripped, collapsed and formatted | +| DM51 | A middleware answer cannot claim it | `Component.importComponent` middleware answering an open name with a definition carrying the disposition gets prose; nothing admitted that definition, so nothing about it is exact | +| DM52 | A mark this engine did not make is nothing | A segment carrying the disposition as a field is not exact, whoever supplied it; expansion also rebuilds text segments, so such a field never reaches emission in the first place | + ### Tier ORC — Repository composition under an ordinary run (§5.3, §8.1) Every case distinguishes what a *document* observed from what the *host* From 34e8c4801888fdaec0978c2472ed254874b01bc0 Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Wed, 2 Sep 2026 18:50:01 -0400 Subject: [PATCH 5/6] =?UTF-8?q?=F0=9F=90=9B=20Give=20the=20exact-source=20?= =?UTF-8?q?record=20to=20the=20run=20that=20owns=20it=20(#722)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The record of which segments an expansion produced as source was a module-scoped `WeakSet`, which `local/no-module-scoped-registry` refuses: one table shared by every execution the process performs, surviving the run that filled it and reachable from no caller's scope. CI caught it where a local lint run did not, because the failure was one error among a thousand warnings. It is now created inside the execution that owns it and handed down through context, beside the segment-cause table it sits next to — reclaimed with the run, answering nothing during the next. The security property is unchanged and is what it always was: membership is keyed by the identity of the segment objects canonical expansion marked, so nothing a document, a package or an import can supply is able to join it. The context is exported for the two core modules that produce and consume the mark and reaches neither public package entrypoint. The emission paths read that record once, where reading is an ordinary operation rather than a durable one. Claude-Session: https://claude.ai/code/session_01DNPFLq52HVSu3NH6WczYLf --- packages/core/src/execute.ts | 34 ++++++++--- packages/core/src/expand.ts | 2 +- packages/core/src/output/exact-source.ts | 61 +++++++++++++------ .../tests/declared-markdown-component.test.ts | 21 +++++-- 4 files changed, 86 insertions(+), 32 deletions(-) diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index f84b8366e..7992d4268 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -96,7 +96,8 @@ import { import { Component, importComponent, raise } from "./component-api.ts"; import { sourceDescription } from "./source-position.ts"; import { renderSegment } from "./render.ts"; -import { isExactSource } from "./output/exact-source.ts"; +import { ExactSource, useExactSource } from "./output/exact-source.ts"; +import type { ExactSource as ExactSourceRecord } from "./output/exact-source.ts"; import { DocumentOutput } from "./api.ts"; import { composeBoundExecChain, @@ -1761,10 +1762,11 @@ function* runValueRoot( authority, ownBody, ); + const exactRecord = yield* ExactSource.get(); for (const resolved of expanded) { const text = renderSegment(resolved); if (text) { - yield* ephemeral(DocumentOutput.operations.output(text, exactly(resolved))); + yield* ephemeral(DocumentOutput.operations.output(text, exactly(exactRecord, resolved))); chunks.push(text); } } @@ -1792,8 +1794,8 @@ function* runValueRoot( * the run says so (#441). */ /** Whether one expanded segment carries exact bytes rather than prose. */ -function exactly(segment: Segment): boolean { - return isExactSource(segment); +function exactly(exact: ExactSourceRecord | undefined, segment: Segment): boolean { + return exact !== undefined && exact.has(segment); } /** @@ -1811,14 +1813,17 @@ interface Emission { readonly exact: boolean; } -function emissions(segments: readonly Segment[]): Emission[] { +function emissions( + record: ExactSourceRecord | undefined, + segments: readonly Segment[], +): Emission[] { const runs: Emission[] = []; for (const segment of segments) { const text = renderSegment(segment); if (!text) { continue; } - const exact = exactly(segment); + const exact = exactly(record, segment); const last = runs[runs.length - 1]; if (last !== undefined && last.exact === exact) { runs[runs.length - 1] = { text: last.text + text, exact }; @@ -1870,6 +1875,11 @@ function* documentWorkflow( // per-segment expansion calls (see spec §6.1). const counter = createBlockCounter(); + // Which segments this run produced as source, read once: the emission paths + // below include the durable workflow's own, where an ordinary context read is + // not what `yield*` means. + const exactRecord = yield* ephemeral(ExactSource.get()); + // What the document rendered before it stopped, held outside the expansion // scope so a failure still leaves it here (§6.9 Partial output). The buffered // root fills `selected`; every streaming root has already emitted `streamed`. @@ -1955,7 +1965,7 @@ function* documentWorkflow( undefined, ); // An empty buffered root emits no output event. - const runs = emissions(selected); + const runs = emissions(exactRecord, selected); for (const run of runs) { yield* ephemeral(DocumentOutput.operations.output(run.text, run.exact)); } @@ -1993,7 +2003,10 @@ function* documentWorkflow( // (non-durable) — output emission is a derived side effect, // not journaled. yield* ephemeral( - DocumentOutput.operations.output(text, resolved !== undefined && exactly(resolved)), + DocumentOutput.operations.output( + text, + resolved !== undefined && exactly(exactRecord, resolved), + ), ); streamed.push(text); } @@ -2018,7 +2031,7 @@ function* documentWorkflow( // chunks is who the preservation is for, and the completion path only emits // for a run that streamed nothing at all — which stops being true as soon // as an earlier segment went out. - const runs = emissions(produced.slice(emittedThrough)); + const runs = emissions(exactRecord, produced.slice(emittedThrough)); const tail = runs.map((run) => run.text).join(""); for (const run of runs) { yield* ephemeral(DocumentOutput.operations.output(run.text, run.exact)); @@ -2175,6 +2188,9 @@ function* executeDocument( // failure from. All created here and reclaimed with this task, so nothing // a run decided outlives it. yield* useSegmentCauses(); + // Which segments this run produced as source. Scope-owned like the rest: + // reclaimed with the run, and answering nothing during the next. + yield* useExactSource(); yield* usePropsCompiler(); yield* useParseCompiler(); const liveFailure: LiveFailureSlot = {}; diff --git a/packages/core/src/expand.ts b/packages/core/src/expand.ts index ebf412e43..f303c9013 100644 --- a/packages/core/src/expand.ts +++ b/packages/core/src/expand.ts @@ -2760,7 +2760,7 @@ function* expandComponent( // after it produced them — is what carries the fact to the emission loop, // which is outside every scope the invocation owned. if (authorizedCanonically && authority?.declared?.declaresExact(name) === true) { - markExactSource(bodyOwner === undefined ? expanded : bodyOwner.slice(renderedFrom)); + yield* markExactSource(bodyOwner === undefined ? expanded : bodyOwner.slice(renderedFrom)); } if (asBinding) { diff --git a/packages/core/src/output/exact-source.ts b/packages/core/src/output/exact-source.ts index 173514dc8..f2990946e 100644 --- a/packages/core/src/output/exact-source.ts +++ b/packages/core/src/output/exact-source.ts @@ -10,36 +10,63 @@ * `Component.importComponent` middleware publish unpresented bytes by writing a * field. * - * So the record lives here, keyed by the identity of the segment objects the - * engine marked, and it is written only where canonical expansion has already - * decided the provenance. It is deliberately not a field on `Segment`: a field - * can be supplied, and this cannot — a segment scanned from a document, parsed - * from a journal, or built by a handler is simply not in this set. + * So the record is kept beside the segments rather than on them, keyed by the + * identity of the objects canonical expansion marked, and written only where + * that expansion has already decided the provenance. A segment scanned from a + * document, parsed from a journal, or built by a handler is simply not in it — + * where a field could be supplied, membership cannot be. + * + * It belongs to the run that made it. The table is created inside the execution + * that owns it and handed down through context, so what one document decided is + * reclaimed with it and answers nothing during the next. */ +import { createContext } from "effection"; +import type { Context, Operation } from "effection"; + import type { Segment } from "../types.ts"; +/** The segments one execution produced from exact-source components. */ +export type ExactSource = WeakSet; + /** - * The segments this engine produced from exact-source components. + * Where this execution keeps that record. * - * Weak, so a segment is remembered exactly as long as something still holds it, - * and identity-keyed, so two segments with identical content are two different - * answers. Module-level because the producer and the emission loop are the same - * loaded copy of core; nothing crosses a copy boundary, and nothing a document - * or a package can reach ever writes to it. + * Module-private: it is exported for the two core modules that produce and + * consume the mark, and reaches neither public package entrypoint, so nothing a + * document or a package can load is able to read or replace it. */ -const EXACT = new WeakSet(); +export const ExactSource: Context = createContext( + "xmd.exact-source", + undefined, +); + +/** Install this execution's own record, and hand it back. */ +export function* useExactSource(): Operation { + const exact: ExactSource = new WeakSet(); + yield* ExactSource.set(exact); + return exact; +} -/** Record that canonical expansion produced these segments as source. */ -export function markExactSource(segments: readonly Segment[]): void { +/** + * Record that canonical expansion produced these segments as source. + * + * With no run installed there is nothing to record into, and nothing is + * published as source — which is the safe answer rather than a lost one. + */ +export function* markExactSource(segments: readonly Segment[]): Operation { + const exact = yield* ExactSource.get(); + if (exact === undefined) { + return; + } for (const segment of segments) { if (segment.type === "text") { - EXACT.add(segment); + exact.add(segment); } } } /** Whether this exact segment object is one canonical expansion marked. */ -export function isExactSource(segment: Segment): boolean { - return EXACT.has(segment); +export function isExactSource(exact: ExactSource | undefined, segment: Segment): boolean { + return exact !== undefined && exact.has(segment); } diff --git a/packages/core/tests/declared-markdown-component.test.ts b/packages/core/tests/declared-markdown-component.test.ts index 62bc23660..676675585 100644 --- a/packages/core/tests/declared-markdown-component.test.ts +++ b/packages/core/tests/declared-markdown-component.test.ts @@ -55,7 +55,7 @@ import { retainedSource } from "../src/root-source.ts"; import { DocumentOutput } from "../src/api.ts"; import { useNormalizedOutput } from "../src/output/normalize.ts"; import { useTerminalOutput } from "../src/output/terminal.ts"; -import { isExactSource } from "../src/output/exact-source.ts"; +import { isExactSource, useExactSource } from "../src/output/exact-source.ts"; import type { ComponentInvocation } from "../src/invocation-identity.ts"; import type { ImportedDefinition } from "../src/components/import-authority.ts"; import type { PropsSchema, Segment } from "../src/types.ts"; @@ -1475,10 +1475,21 @@ describe("Tier DM — exact source is a provenance, not a field", () => { // carrying any field at all — including the one an earlier design used — // answers false. This is the assertion that discriminates: a marker that // consulted a field would pass it back. - expect( - isExactSource({ type: "text", content: PRESENTABLE, exact: true } as unknown as Segment), - ).toBe(false); - expect(isExactSource({ type: "text", content: PRESENTABLE } as unknown as Segment)).toBe(false); + yield* scoped(function* () { + // The record belongs to the scope that made it, so the questions are + // asked inside that scope rather than by carrying it out. + const record = yield* useExactSource(); + expect( + isExactSource(record, { + type: "text", + content: PRESENTABLE, + exact: true, + } as unknown as Segment), + ).toBe(false); + expect( + isExactSource(record, { type: "text", content: PRESENTABLE } as unknown as Segment), + ).toBe(false); + }); // Then the end-to-end shape, which records a second fact worth keeping: // expansion rebuilds text segments, so a field a definition wrote onto its From a3ffc53f16c758676533feceb427e0fca1fe5db0 Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Wed, 2 Sep 2026 19:01:38 -0400 Subject: [PATCH 6/6] =?UTF-8?q?=F0=9F=94=92=20Pass=20the=20exact-source=20?= =?UTF-8?q?record=20by=20value=20rather=20than=20through=20a=20context=20(?= =?UTF-8?q?#722)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An Effection context resolves by name, and a name is not a secret. Storing the exact-source record under `xmd.exact-source` therefore published it: an ordinary function component could build a context with that name, retrieve the execution's own WeakSet, replace the `has` it inherited, and have its prose emitted with no whitespace normalization and no terminal formatting. That is an authority escalation, and it is what AGENTS.md rule 15 means by refusing to let security enforcement trust replaceable context state. The record is now created inside the execution that owns it and handed down by value on the private `ExpansionAuthority` canonical core already carries its import authority and identity domains on. Marking and emission receive it explicitly. Nothing a document, a component, middleware or a separately loaded copy can name reaches that object, and it is reclaimed with the execution that made it. What the correction does not do is trade one hole for another: no module-scoped registry, no symbol or property brand on the segments, no secret context name — a name that must stay unguessed is still a name — and no new public opt-in. Eligibility is unchanged, and is still canonical authorization together with the host's admitted declaration. DM53 is the attack, end to end: middleware answers an open name with a function component that builds that context, corrupts whatever it finds, and returns prose carrying emphasis, trailing spaces and blank runs through both real presentation middlewares. It fails at 34e8c480 and passes here. Claude-Session: https://claude.ai/code/session_01DNPFLq52HVSu3NH6WczYLf --- architecture.md | 7 ++ .../core/src/components/import-authority.ts | 10 +++ packages/core/src/execute.ts | 19 ++--- packages/core/src/expand.ts | 5 +- packages/core/src/output/exact-source.ts | 53 +++++++------- .../tests/declared-markdown-component.test.ts | 72 ++++++++++++++----- specs/executable-mdx-spec.md | 13 +++- 7 files changed, 120 insertions(+), 59 deletions(-) diff --git a/architecture.md b/architecture.md index 85f8649fa..7f3df3918 100644 --- a/architecture.md +++ b/architecture.md @@ -3583,6 +3583,13 @@ from somewhere, so none of them decides it — an ordinary claiming the disposition publishes prose, exactly as a repository file of that name does. +The record of which segments an expansion produced as source belongs to the +execution that made it and is handed down **by value** on canonical core's +private expansion authority. Not through a context: a context resolves by name, +and a name is not a secret, so a component could build one, reach the record and +answer that everything is exact. It is reclaimed with its execution, reaches no +public entrypoint, and marks nothing on the segments themselves. + ## The syntax catalog boundary `xmd syntax` answers what a document may write here, and answering must cost diff --git a/packages/core/src/components/import-authority.ts b/packages/core/src/components/import-authority.ts index 32e34c62f..b747d600f 100644 --- a/packages/core/src/components/import-authority.ts +++ b/packages/core/src/components/import-authority.ts @@ -21,6 +21,7 @@ import type { ComponentDefinition, FunctionComponentDefinition } from "../types.ts"; import type { FormSelections, InvocationIdentities } from "../invocation-identity.ts"; import type { DeclaredImports, PrivateClosure } from "./declared-markdown.ts"; +import type { ExactSource } from "../output/exact-source.ts"; /** A definition an import may answer with. */ export type ImportedDefinition = ComponentDefinition | FunctionComponentDefinition; @@ -82,6 +83,15 @@ export interface ExpansionAuthority { readonly privates?: PrivateClosure; /** The domains this execution minted, for the components it gave one. */ readonly identities?: InvocationIdentities; + /** + * Which segments this execution produced as a program's source. + * + * Held by the execution and handed here by value, like everything else on + * this object. It is on the private authority rather than in a context + * because a context resolves by name, and a name is not a secret: a component + * could build one, reach the record and answer that everything is exact. + */ + readonly exact?: ExactSource; /** * What canonical resolution selected for each import, for the components * whose authored form selects an effect. diff --git a/packages/core/src/execute.ts b/packages/core/src/execute.ts index 7992d4268..7e516dfd4 100644 --- a/packages/core/src/execute.ts +++ b/packages/core/src/execute.ts @@ -96,7 +96,7 @@ import { import { Component, importComponent, raise } from "./component-api.ts"; import { sourceDescription } from "./source-position.ts"; import { renderSegment } from "./render.ts"; -import { ExactSource, useExactSource } from "./output/exact-source.ts"; +import { createExactSource } from "./output/exact-source.ts"; import type { ExactSource as ExactSourceRecord } from "./output/exact-source.ts"; import { DocumentOutput } from "./api.ts"; import { @@ -1762,7 +1762,7 @@ function* runValueRoot( authority, ownBody, ); - const exactRecord = yield* ExactSource.get(); + const exactRecord = authority.exact; for (const resolved of expanded) { const text = renderSegment(resolved); if (text) { @@ -1875,10 +1875,10 @@ function* documentWorkflow( // per-segment expansion calls (see spec §6.1). const counter = createBlockCounter(); - // Which segments this run produced as source, read once: the emission paths - // below include the durable workflow's own, where an ordinary context read is - // not what `yield*` means. - const exactRecord = yield* ephemeral(ExactSource.get()); + // Which segments this run produced as source. Read off the private authority + // this execution built, so the emission paths below reach it without a + // context — there is nothing here for a document to name. + const exactRecord = authority.exact; // What the document rendered before it stopped, held outside the expansion // scope so a failure still leaves it here (§6.9 Partial output). The buffered @@ -2188,9 +2188,6 @@ function* executeDocument( // failure from. All created here and reclaimed with this task, so nothing // a run decided outlives it. yield* useSegmentCauses(); - // Which segments this run produced as source. Scope-owned like the rest: - // reclaimed with the run, and answering nothing during the next. - yield* useExactSource(); yield* usePropsCompiler(); yield* useParseCompiler(); const liveFailure: LiveFailureSlot = {}; @@ -2272,6 +2269,10 @@ function* executeDocument( ...(declaredImports === undefined ? {} : { declared: declaredImports }), identities: identity.identities, forms, + // Created here, held here, and reclaimed with this execution. Nothing a + // document, a component, middleware or a separately loaded copy can + // name reaches this object. + exact: createExactSource(), }; // Install the document's runtime Component providers before durableRun diff --git a/packages/core/src/expand.ts b/packages/core/src/expand.ts index f303c9013..ffe05f103 100644 --- a/packages/core/src/expand.ts +++ b/packages/core/src/expand.ts @@ -2760,7 +2760,10 @@ function* expandComponent( // after it produced them — is what carries the fact to the emission loop, // which is outside every scope the invocation owned. if (authorizedCanonically && authority?.declared?.declaresExact(name) === true) { - yield* markExactSource(bodyOwner === undefined ? expanded : bodyOwner.slice(renderedFrom)); + markExactSource( + authority?.exact, + bodyOwner === undefined ? expanded : bodyOwner.slice(renderedFrom), + ); } if (asBinding) { diff --git a/packages/core/src/output/exact-source.ts b/packages/core/src/output/exact-source.ts index f2990946e..cc4bf3c5d 100644 --- a/packages/core/src/output/exact-source.ts +++ b/packages/core/src/output/exact-source.ts @@ -11,51 +11,46 @@ * field. * * So the record is kept beside the segments rather than on them, keyed by the - * identity of the objects canonical expansion marked, and written only where - * that expansion has already decided the provenance. A segment scanned from a + * identity of the objects canonical expansion marked. A segment scanned from a * document, parsed from a journal, or built by a handler is simply not in it — * where a field could be supplied, membership cannot be. * - * It belongs to the run that made it. The table is created inside the execution - * that owns it and handed down through context, so what one document decided is - * reclaimed with it and answers nothing during the next. + * ## It is passed, never published + * + * The record belongs to one execution, which creates it and hands it down by + * value on the private `ExpansionAuthority` canonical core already carries its + * import authority on. It is deliberately not in a context: an Effection + * context resolves by *name*, and a name is not a secret — anything that can + * run code can build a context with the same one and reach whatever is stored + * under it. A component doing that to this record could replace its `has` and + * have its own prose published as source. + * + * There is no module-scoped table either, so nothing outlives the run that made + * it, and no brand on the segments, because a property is exactly what an + * untrusted object can carry. */ -import { createContext } from "effection"; -import type { Context, Operation } from "effection"; - import type { Segment } from "../types.ts"; /** The segments one execution produced from exact-source components. */ export type ExactSource = WeakSet; -/** - * Where this execution keeps that record. - * - * Module-private: it is exported for the two core modules that produce and - * consume the mark, and reaches neither public package entrypoint, so nothing a - * document or a package can load is able to read or replace it. - */ -export const ExactSource: Context = createContext( - "xmd.exact-source", - undefined, -); - -/** Install this execution's own record, and hand it back. */ -export function* useExactSource(): Operation { - const exact: ExactSource = new WeakSet(); - yield* ExactSource.set(exact); - return exact; +/** A record for one execution, created where that execution begins. */ +export function createExactSource(): ExactSource { + return new WeakSet(); } /** * Record that canonical expansion produced these segments as source. * - * With no run installed there is nothing to record into, and nothing is - * published as source — which is the safe answer rather than a lost one. + * An expansion with no record — anything running outside an execution that made + * one — marks nothing, and nothing is published as source. That is the safe + * answer rather than a lost one. */ -export function* markExactSource(segments: readonly Segment[]): Operation { - const exact = yield* ExactSource.get(); +export function markExactSource( + exact: ExactSource | undefined, + segments: readonly Segment[], +): void { if (exact === undefined) { return; } diff --git a/packages/core/tests/declared-markdown-component.test.ts b/packages/core/tests/declared-markdown-component.test.ts index 676675585..a7fda350c 100644 --- a/packages/core/tests/declared-markdown-component.test.ts +++ b/packages/core/tests/declared-markdown-component.test.ts @@ -29,7 +29,7 @@ import { describe, it } from "@executablemd/test-support/bdd"; import { expect } from "@executablemd/test-support/expect"; -import { ensure, scoped, until } from "effection"; +import { createContext, ensure, scoped, until } from "effection"; import type { Operation } from "effection"; import { rm, writeTextFile } from "@effectionx/fs"; import { API } from "@executablemd/runtime"; @@ -55,7 +55,7 @@ import { retainedSource } from "../src/root-source.ts"; import { DocumentOutput } from "../src/api.ts"; import { useNormalizedOutput } from "../src/output/normalize.ts"; import { useTerminalOutput } from "../src/output/terminal.ts"; -import { isExactSource, useExactSource } from "../src/output/exact-source.ts"; +import { createExactSource, isExactSource } from "../src/output/exact-source.ts"; import type { ComponentInvocation } from "../src/invocation-identity.ts"; import type { ImportedDefinition } from "../src/components/import-authority.ts"; import type { PropsSchema, Segment } from "../src/types.ts"; @@ -1465,6 +1465,48 @@ describe("Tier DM — exact source is a provenance, not a field", () => { expect(output).not.toContain("**these**"); }); + it("DM53: a component that reaches for the record by context name gets prose", function* () { + // Effection contexts resolve by name, so a name is not a secret: anything + // that can run code can build a context with the same one. This is the + // attack that closes — a function component nobody trusts asks for the + // record, corrupts whatever it finds, and then returns prose that would be + // published unpresented if the corruption had worked. + let reached = false; + + const output = yield* published( + "\n", + [declared(POLICY_SOURCE)], + [ + answeringOpenName({ + kind: "function", + name: "Virtual", + path: "Virtual.ts", + props: NO_PROPS, + *fn() { + const stolen = createContext<{ has?: unknown } | undefined>( + "xmd.exact-source", + undefined, + ); + const record = yield* stolen.get(); + if (record !== undefined) { + reached = true; + // Every segment is exact, if anything asks this object. + record.has = () => true; + } + return PRESENTABLE; + }, + } as unknown as ImportedDefinition), + ], + ); + + // The record is not reachable by name at all, which is the property under + // test; the assertion below holds either way, because reaching it would + // still decide nothing. + expect(reached).toBe(false); + expect(unpresented(output)).toBe(false); + expect(output).not.toContain("**these**"); + }); + it("DM52: a segment this engine did not mark is prose, whatever it carries", function* () { // The other half of the same attack: not the definition claiming the // disposition, but segments arriving already wearing the mark. Two @@ -1475,21 +1517,17 @@ describe("Tier DM — exact source is a provenance, not a field", () => { // carrying any field at all — including the one an earlier design used — // answers false. This is the assertion that discriminates: a marker that // consulted a field would pass it back. - yield* scoped(function* () { - // The record belongs to the scope that made it, so the questions are - // asked inside that scope rather than by carrying it out. - const record = yield* useExactSource(); - expect( - isExactSource(record, { - type: "text", - content: PRESENTABLE, - exact: true, - } as unknown as Segment), - ).toBe(false); - expect( - isExactSource(record, { type: "text", content: PRESENTABLE } as unknown as Segment), - ).toBe(false); - }); + const record = createExactSource(); + expect( + isExactSource(record, { + type: "text", + content: PRESENTABLE, + exact: true, + } as unknown as Segment), + ).toBe(false); + expect( + isExactSource(record, { type: "text", content: PRESENTABLE } as unknown as Segment), + ).toBe(false); // Then the end-to-end shape, which records a second fact worth keeping: // expansion rebuilds text segments, so a field a definition wrote onto its diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 1cf361b42..84f6c97ed 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -2792,9 +2792,16 @@ a `TextSegment` arriving already marked, a frontmatter key, a prop and a middleware answer are each data that reached the engine from somewhere, and none of them decides anything — an ordinary `Component.importComponent` handler that answers an open name with a definition claiming the disposition gets prose, and -so does a repository file of the same name. Which segments an expansion produced -as source is recorded against those segment objects' identity, where nothing a -document or a package can supply is able to write. +so does a repository file of the same name. + +Which segments an expansion produced as source is recorded against those segment +objects' identity, in a record **the execution creates and hands down by value** +on the same private authority it carries its import authority on. It is reached +by no name: a context resolves by one, and a name is not a secret, so a record +stored in a context is one any component can build a context for, retrieve and +answer differently from. The record is reclaimed with the execution that made +it, appears in neither public package entrypoint, and is not a brand on the +segments — a property is exactly what an untrusted object can carry. **Every declaration is read once, before any installation runs.** The name, origin, source, digest, forms, prose, whether it renders exact source, and each