diff --git a/.gitignore b/.gitignore index 3992e21..e2d7c32 100644 --- a/.gitignore +++ b/.gitignore @@ -42,6 +42,21 @@ reports/**/*.out reports/**/*.fdb_latexmk reports/**/claims.tex +# Which folder this checkout's app was last pointed at (core/workspace.py). It +# sits beside the code rather than in data/ precisely so it survives switching +# away from a workspace -- which also means it is one machine's choice, not the +# repository's. +.grad-workspace.json + +# Per-machine UI state. `ui_storage_secret` signs the browser-id cookie each +# client's transcript is keyed by, so it is a credential in everything but name; +# the session files are the conversations themselves, and the layouts and kernel +# pid are one machine's, not the repository's. +data/ui_storage_secret +data/ui_session-*.jsonl +data/layouts/ +data/kernel/ + # Credentials never live in the workspace (HANDOFF §9). This is belt and braces. .env *.pem diff --git a/README.md b/README.md index 592cf4c..2ee1558 100644 --- a/README.md +++ b/README.md @@ -81,9 +81,40 @@ Store credentials once; they never enter the agent's environment: python -m tools.jobs credential set hf_token python -m tools.jobs credential set openrouter_key python -m tools.jobs credential set voyage_key -python -m tools.jobs credential set context7_key # optional; raises rate limits +python -m tools.jobs credential set claude_oauth_token +python -m tools.jobs credential set asta_api_key # optional; raises rate limits +python -m tools.jobs credential set context7_key # optional; raises rate limits ``` +Or store them from the app: the workspace menu (`project ▾`) has a credentials +panel, which is the same command with `--stdin` instead of the `getpass` prompt. +That exists because the prompt needs a terminal, and needing one for this was +the only thing that forced a shell open beside the app on a fresh machine. The +value goes down a pipe rather than in an argument — an argv is visible to +anything that can list processes. + +### Retrieval without an institutional email + +Tier 1 defaults to **Ai2's Asta** (`asta-tools.allen.ai`) rather than to the +Semantic Scholar REST API. It is the same corpus and it exposes the same +`snippet_search` the funnel is built around, but Semantic Scholar +[no longer issues API keys to free-domain email addresses][s2-keys], which +leaves a personal account on a shared anonymous pool that is rate limited often +enough that "no results" and "no key" are hard to tell apart. + +Asta is reached over streamable HTTP without adopting MCP as an architecture, +which is what §5 already said about that endpoint. Its key is optional and +raises limits rather than unlocking anything. Set `[retrieval] tier1` to `s2` +or `both` to change it, or pass `--tier1` per search. + +**The endpoint, the transport and the tool names are from Ai2's documentation; +the shape of each tool's result is not verified against the live service.** +`core/http.py:_rows` reads both plausible shapes and raises on anything else, +because a search that quietly returns nothing reads as "the literature has +nothing on this". + +[s2-keys]: https://www.semanticscholar.org/product/api + ## Run ```bash @@ -189,13 +220,15 @@ core/ the machinery the CLIs share, so no tool can forget a rule haiku.py funnel stages 0 and 3, via forced SDK tools http.py Semantic Scholar, rerank, embeddings, Context7 tools/ the CLIs -ui/ the NiceGUI workspace: a tiling shell over eleven windows +ui/ the NiceGUI workspace: a tiling shell over twelve windows tokens.py the design tokens; the stylesheet is generated from them layout.py the pane tree and the moves over it -- pure, tested models.py what each window shows, as plain data -- pure, tested registry.py the one list of windows the shell is derived from shell.py the chrome, and how a window survives a retile - windows/ eleven renderers, none of which read a ledger directly + tasks.py local commands run in the background, and how to stop one + sessions.py named chat sessions: a file each, listed by a glob + windows/ twelve renderers, none of which read a ledger directly jupyter_theme.py the same tokens, emitted as JupyterLab's custom.css config/jupyter/ the Lab server config: framing headers, overrides, theme skills/ loaded on demand, not into the default context @@ -211,14 +244,17 @@ Implemented and tested: the ledger, the submission hash, every gate, the smoke caps, the CLI contract, the hook, the persistent kernel, notebook verification, the project dimension and its three ceilings, HF organization namespaces, library-currency checking, the campaign loop and its budget gate, and the report -generator with all four of its rules. `pytest` covers these — 295 tests, no -network, no SDK required. +generator with all four of its rules, the background task runner and its stop +path, and named chat sessions. `pytest` covers these — no network, no SDK +required, and the "no network" half is now enforced rather than intended: an +autouse fixture in `tests/conftest.py` replaces `core.http._httpx`, because a +suite that reaches the network does not fail, it *hangs*. Implemented but not exercised against a live service: the HF Jobs backend, the -SSH backend, Semantic Scholar, the OpenRouter reranker, Voyage embeddings, the -two Haiku funnel stages, Context7, ShinkaEvolve, and RepoWiki. They are written -against the documented interfaces and fail with actionable errors rather than -tracebacks, but a real credential and a real run are what will find the +SSH backend, Asta, Semantic Scholar, the OpenRouter reranker, Voyage embeddings, +the two Haiku funnel stages, Context7, ShinkaEvolve, and RepoWiki. They are +written against the documented interfaces and fail with actionable errors rather +than tracebacks, but a real credential and a real run are what will find the mismatches. **Two of [HANDOFF-2 §23](HANDOFF-2.md)'s open questions are now closed:** diff --git a/agent.py b/agent.py index 891b20d..0abfcd0 100644 --- a/agent.py +++ b/agent.py @@ -57,7 +57,16 @@ def system_prompt() -> str: return (paths.root() / "prompts" / "system.md").read_text(encoding="utf-8") -def build_options(cfg: Any, *, permission_mode: str | None = None) -> Any: +def build_options(cfg: Any, *, permission_mode: str | None = None, resume: str | None = None) -> Any: + """The options one session runs under. + + `resume` is an SDK session id, and passing it is the difference between + reopening a transcript and reopening a *conversation*: without it the model + starts with no memory of the turns the window is showing above the composer, + which is a worse failure than not resuming at all because nothing on screen + says so. `ui/sessions.py` records the id and reports when it does not have + one. + """ sdk = _sdk() mode = permission_mode or str(cfg.get("agent", "permission_mode", "dontAsk")) hook_matchers = { @@ -65,6 +74,7 @@ def build_options(cfg: Any, *, permission_mode: str | None = None) -> Any: "Stop": [sdk.HookMatcher(hooks=[hooks.stop])], } return sdk.ClaudeAgentOptions( + resume=resume, model=cfg.model_for("research"), system_prompt=system_prompt(), allowed_tools=BUILTIN_TOOLS, @@ -72,6 +82,12 @@ def build_options(cfg: Any, *, permission_mode: str | None = None) -> Any: permission_mode=mode, cwd=str(paths.root()), hooks=hook_matchers, + # Off by default in the SDK, and the default is why an answer used to + # arrive in one lump: without it `receive_response` yields nothing until + # a whole `AssistantMessage` is finished. With it the same turn also + # emits `StreamEvent`s carrying token deltas. `TextStream` is what turns + # the two into one transcript -- see the warning in its docstring. + include_partial_messages=True, ) @@ -188,10 +204,14 @@ async def _turn(client: Any, prompt: str) -> bool: return False await client.query(prompt) + stream = TurnStream() async for message in client.receive_response(): - text = _text_of(message) - if text: - print(text, end="", flush=True) + # Whatever has not been printed yet -- a token as it arrives, the tail + # of a message that was never streamed, or a line naming a tool call. + # Never both halves of the same text. + chunk = stream.feed(message) + if chunk: + print(chunk, end="", flush=True) usage = getattr(message, "usage", None) if usage is not None: quota_log.from_sdk_usage( @@ -210,6 +230,320 @@ def _text_of(message: Any) -> str: return "" +def _delta_of(message: Any) -> str: + """The visible text a partial-message stream event carries, if any. + + `StreamEvent.event` is the raw Anthropic streaming event, so this is a + filter as much as an accessor: only `content_block_delta` carrying a + `text_delta` is answer text. Thinking deltas and tool-input deltas are + excluded deliberately, because `_text_of` excludes their finished blocks too + -- a `ThinkingBlock` has `.thinking`, not `.text`. Letting them through here + would make the stream say something the settled message does not. + """ + event = getattr(message, "event", None) + if not isinstance(event, dict) or event.get("type") != "content_block_delta": + return "" + delta = event.get("delta") + if not isinstance(delta, dict) or delta.get("type") != "text_delta": + return "" + text = delta.get("text") + return text if isinstance(text, str) else "" + + +class TextStream: + """One turn's visible text, assembled from deltas *and* finished messages. + + `include_partial_messages` makes the SDK emit both halves of the same text: + a run of `text_delta` events, and then the `AssistantMessage` that contains + all of it. **Appending both is the bug this class exists to prevent** -- it + is the obvious way to write the loop, and it makes every answer appear + twice. + + So a finished message *replaces* the deltas that built it rather than + following them. That ordering also makes the finished message authoritative: + if the two ever disagree -- a dropped event, a turn resumed from cache, a + message the SDK never streamed -- what stays on screen is the message, not + the reconstruction. A turn is many messages, so this repeats per message, + which is why `_streamed` is reset each time rather than once at the end. + + `feed` returns only the text that has not been shown yet, so a CLI can print + its return value directly; `text` is the whole answer so far, for a UI that + re-renders from it. + """ + + def __init__(self) -> None: + self.text = "" + #: The tail of `text` contributed by deltas since the last finished + #: message -- the part a finished message is entitled to overwrite. + self._streamed = "" + + def feed(self, message: Any) -> str: + delta = _delta_of(message) + if delta: + self.text += delta + self._streamed += delta + return delta + + text = _text_of(message) + # A message with no text at all -- a tool result, a system message, the + # final result -- must leave a half-streamed block alone. + if not text: + return "" + + if text.startswith(self._streamed): + unseen = text[len(self._streamed) :] + self.text += unseen + else: + self.text = self.text[: len(self.text) - len(self._streamed)] + text + unseen = "" + self._streamed = "" + return unseen + + +# --------------------------------------------------------------------------- +# tool calls +# --------------------------------------------------------------------------- +#: What one call contributes to a transcript, at most. A `Read` of a long file +#: or a training log is tens of thousands of characters, and every one of them +#: would be held for the life of the session, written to the transcript file on +#: settle, and drawn again on restore. A card is a record that the call happened +#: and how it went, not a second copy of its output. +RESULT_CHARS = 2000 +RESULT_LINES = 40 + +#: Which input key says what a call was *on*. Anything not listed here falls +#: back to `SUBJECT_KEYS`, then to the first short string in the input -- an +#: `Edit` carries its whole replacement text, and a card head that is a wall of +#: source is worse than one that is empty. +TOOL_SUBJECT = { + "Bash": "command", + "Read": "file_path", + "Write": "file_path", + "Edit": "file_path", + "Glob": "pattern", + "Grep": "pattern", +} +SUBJECT_KEYS = ("command", "file_path", "path", "pattern", "query", "url", "prompt") + +#: Per-row limits for the rest of a call's input. Six rows of two lines is a +#: card you can read at a glance; the whole input is not. +ROW_CHARS = 200 +ROW_LINES = 2 +MAX_ROWS = 6 + + +def clip(text: str, *, chars: int = RESULT_CHARS, lines: int = RESULT_LINES) -> str: + """`text`, bounded -- and saying what it dropped rather than trailing off. + + ASCII on purpose: this can reach a Windows console, where a stray `…` is a + `UnicodeEncodeError` that would take the turn down. + """ + if not text: + return "" + split = text.splitlines() + dropped_lines = max(0, len(split) - lines) + out = "\n".join(split[:lines]) + dropped_chars = max(0, len(out) - chars) + out = out[:chars] + if dropped_chars: + out += f"\n... +{dropped_chars:,} more characters" + if dropped_lines: + out += f"\n... +{dropped_lines:,} more lines" + return out + + +def _one_line(text: str, limit: int = 120) -> str: + """A subject collapsed onto one line, for a card head.""" + flattened = " ".join(str(text).split()) + return flattened if len(flattened) <= limit else flattened[: limit - 3] + "..." + + +def describe_tool(name: str, tool_input: dict[str, Any]) -> tuple[str, str]: + """`(subject_key, subject)` -- what this call was on, and under which key.""" + keys = (TOOL_SUBJECT[name],) if name in TOOL_SUBJECT else SUBJECT_KEYS + for key in keys: + value = tool_input.get(key) + if isinstance(value, str) and value.strip(): + return key, value + for key, value in tool_input.items(): + if isinstance(value, str) and value.strip() and len(value) <= ROW_CHARS: + return key, value + return "", "" + + +def _content_blocks(message: Any) -> list[Any]: + content = getattr(message, "content", None) + return content if isinstance(content, list) else [] + + +def _tool_uses(message: Any) -> list[dict[str, Any]]: + """Every `ToolUseBlock` in a finished message, as plain data. + + Duck-typed rather than isinstance-checked so `ServerToolUseBlock` -- a tool + the API runs on the model's behalf -- draws the same card, and so this file + keeps working if the SDK renames a class. + """ + uses: list[dict[str, Any]] = [] + for block in _content_blocks(message): + name = getattr(block, "name", None) + tool_input = getattr(block, "input", None) + identifier = getattr(block, "id", None) + if isinstance(name, str) and isinstance(tool_input, dict) and isinstance(identifier, str): + uses.append({"id": identifier, "name": name, "input": tool_input}) + return uses + + +def _tool_results(message: Any) -> list[dict[str, Any]]: + """Every `ToolResultBlock`. These arrive on a `UserMessage`, not on the + assistant's -- the tool ran on this side of the wire and is reporting back.""" + results: list[dict[str, Any]] = [] + for block in _content_blocks(message): + identifier = getattr(block, "tool_use_id", None) + if not isinstance(identifier, str): + continue + results.append( + { + "id": identifier, + "text": _result_text(getattr(block, "content", None)), + "is_error": bool(getattr(block, "is_error", False)), + } + ) + return results + + +def _result_text(content: Any) -> str: + """A result's content, which is a string, a list of blocks, or nothing.""" + if content is None: + return "" + if isinstance(content, str): + return content + if isinstance(content, list): + parts: list[str] = [] + for item in content: + if isinstance(item, dict): + text = item.get("text") + parts.append(text if isinstance(text, str) else json.dumps(item, default=str)) + else: + parts.append(str(item)) + return "\n".join(p for p in parts if p) + return str(content) + + +def tool_block(use: dict[str, Any]) -> dict[str, Any]: + """One tool card, before its result lands.""" + tool_input = use["input"] + subject_key, subject = describe_tool(use["name"], tool_input) + rows = [ + (key, clip(str(value), chars=ROW_CHARS, lines=ROW_LINES)) + for key, value in tool_input.items() + if key != subject_key + ] + return { + "kind": "tool", + "id": use["id"], + "name": use["name"], + "title": _one_line(subject), + "text": clip(subject, chars=600, lines=12), + "rows": rows[:MAX_ROWS], + "status": "running", + "result": "", + } + + +class TurnStream: + """One turn as an ordered list of blocks: prose, and the tool calls between. + + `TextStream` answers what the agent *said*; this answers what it *did*, and + that was the larger half of most turns here -- every capability in this + project is reached by a `Bash` into `tools/`, and none of it was visible. A + turn that ran six commands and then summarised them arrived as the summary + alone, which is exactly the part you cannot check. + + A turn is kept as blocks rather than as one string because **the order is + the information**: which command ran before which claim. So a run of prose + is cut at each tool call, and `blocks` reads top to bottom as the turn + happened. Text assembly is delegated to `TextStream` unchanged, including + its rule that a finished message replaces the deltas that built it. + + `feed` returns what a CLI should print next; a UI re-renders from `blocks`. + """ + + def __init__(self) -> None: + self.blocks: list[dict[str, Any]] = [] + self._text = TextStream() + #: The block the current run of prose is accumulating into, if open. + self._open: dict[str, Any] | None = None + #: Tool blocks by call id, so a result can find the call it answers. + self._calls: dict[str, dict[str, Any]] = {} + + @property + def text(self) -> str: + """The turn's prose, tool cards left out -- what a plain transcript says.""" + return "".join(b["text"] for b in self.blocks if b["kind"] == "text") + + def feed(self, message: Any) -> str: + printed = self._feed_text(message) + for use in _tool_uses(message): + block = tool_block(use) + self.blocks.append(block) + self._calls[block["id"]] = block + # A tool call ends the run of prose above it: whatever the agent says + # next belongs *below* the card, because that is when it said it. + self._text = TextStream() + self._open = None + printed += _tool_line(block) + for result in _tool_results(message): + block = self._calls.get(result["id"]) + if block is None: + # A result for a call this stream never saw -- a turn resumed + # from cache, a subagent's tool. Nothing to attach it to, and + # inventing a card for it would claim an order we do not know. + continue + block["result"] = clip(result["text"]) + block["status"] = "error" if result["is_error"] else "ok" + printed += _result_line(block) + return printed + + def note(self, text: str) -> None: + """Append text the session itself has to say -- that a turn died, say.""" + if not text: + return + self._text = TextStream() + self._open = {"kind": "text", "text": text} + self.blocks.append(self._open) + + def active(self) -> dict[str, Any] | None: + """The call currently in flight, for a status line to name.""" + for block in reversed(self.blocks): + if block["kind"] == "tool" and block["status"] == "running": + return block + return None + + def _feed_text(self, message: Any) -> str: + chunk = self._text.feed(message) + # Synced from `TextStream.text`, never `+=`: a finished message may + # rewrite the tail its own deltas built, and the block has to follow. + if self._text.text: + if self._open is None: + self._open = {"kind": "text", "text": ""} + self.blocks.append(self._open) + self._open["text"] = self._text.text + return chunk + + +def _tool_line(block: dict[str, Any]) -> str: + subject = f" {block['title']}" if block["title"] else "" + return f"\n[tool] {block['name']}{subject}\n" + + +def _result_line(block: dict[str, Any]) -> str: + if block["status"] == "error": + first = next((line for line in block["result"].splitlines() if line.strip()), "failed") + return f"[tool] {block['name']} failed: {_one_line(first, 160)}\n" + lines = len(block["result"].splitlines()) + return f"[tool] {block['name']} ok ({lines} line{'' if lines == 1 else 's'})\n" + + # --------------------------------------------------------------------------- # the deny probe (§9, §12 step 1) # --------------------------------------------------------------------------- @@ -278,6 +612,12 @@ def main() -> None: parser.add_argument("--once", action="store_true", help="exit after the first response") parser.add_argument("--probe", action="store_true", help="run the §9 permission deny probe and exit") parser.add_argument("--ui", action="store_true", help="launch the NiceGUI desktop app instead") + parser.add_argument( + "--port", + type=int, + default=8080, + help="port for --ui; move it when something else already holds 8080", + ) parser.add_argument("--check", action="store_true", help="report environment and auth posture, then exit") args = parser.parse_args() @@ -289,7 +629,10 @@ def main() -> None: if args.ui: from ui.app import run as run_ui # noqa: PLC0415 - run_ui() + # A non-default port also moves the app's origin, and the embedded Lab + # scopes its `frame-ancestors` to that origin -- so `tools.lab` needs + # `--ui-origin http://127.0.0.1:` to match, or the iframe is blocked. + run_ui(port=args.port) return prompt = " ".join(args.prompt) if args.prompt else None diff --git a/core/config.py b/core/config.py index 4b14382..9a966b6 100644 --- a/core/config.py +++ b/core/config.py @@ -57,6 +57,16 @@ "cache_ttl_s": 604800, "request_timeout_s": 60, "min_request_interval_s": 1.1, # unauthenticated S2 is ~1 req/s + # Which tier-1 client does discovery: "asta", "s2", or "both". + # + # Asta by default because it is the one that *works*. Both reach the + # same Semantic Scholar corpus and both expose snippet search, but S2's + # own API stopped issuing keys to free-domain email addresses, leaving a + # personal account on the shared anonymous pool -- which is rate limited + # often enough that "no results" and "no key" are hard to tell apart. + # Asta's key is optional. "both" is for comparing them, and it doubles + # the request count for candidates that mostly fuse back together. + "tier1": "asta", }, # HANDOFF-2 §18 listed the REST paths as unverified (§23 item 2). They are # now verified against the live API: `/api/v2/libs/search` returns diff --git a/core/credentials.py b/core/credentials.py index 8c1779a..aca7fe2 100644 --- a/core/credentials.py +++ b/core/credentials.py @@ -31,6 +31,42 @@ # optional and says so when it is missing. CONTEXT7_KEY = "context7_key" +# The sixth. The funnel's Haiku stages (`core/haiku.py`) are Agent SDK clients +# in their own right, and they are reached the way every capability here is +# reached: the agent runs the CLI over Bash. That hop strips +# CLAUDE_CODE_OAUTH_TOKEN from the child environment -- deliberately, and only +# that variable; everything else in the environment survives it. So a token that +# lives in the environment authenticates the funnel from a terminal and leaves +# it unauthenticated under the agent, which is the only way it actually runs. +# The answer is the one §9 already gives for every other credential: keep it in +# the credential store and fetch it at the moment of use. +CLAUDE_TOKEN = "claude_oauth_token" + +# The seventh, and the one that exists because the fourth cannot be obtained. +# Semantic Scholar stopped issuing API keys to free-domain email addresses, so +# `S2_KEY` is unreachable from a personal account and the anonymous pool is +# shared with everyone else in that position. Ai2 serve the same corpus over MCP +# at `asta-tools.allen.ai`, where a key is optional and raises rate limits +# rather than unlocking anything -- so this is treated like `CONTEXT7_KEY`: its +# absence is a note, not an error. +ASTA_KEY = "asta_api_key" + +#: Every credential this project knows, in one tuple so nothing derived from it +#: can be added to and then forgotten. `status()` reports these, +#: `tools/jobs.py` accepts these, and `scrub_environment` removes the `GRAD_*` +#: fallback of each -- and it was that last one that drifted: two credentials +#: were added to the lookup and not to the scrub, leaving the agent's own +#: environment holding tokens §9 says must not be in it. +ALL: tuple[str, ...] = ( + HF_TOKEN, + OPENROUTER_KEY, + VOYAGE_KEY, + S2_KEY, + CONTEXT7_KEY, + CLAUDE_TOKEN, + ASTA_KEY, +) + def _keyring() -> Any: try: @@ -97,10 +133,7 @@ def present(name: str) -> bool: def status() -> dict[str, bool]: """Which credentials exist. Values are never returned.""" - return { - n: present(n) - for n in (HF_TOKEN, OPENROUTER_KEY, VOYAGE_KEY, S2_KEY, CONTEXT7_KEY) - } + return {n: present(n) for n in ALL} def _env_fallback_allowed() -> bool: @@ -115,6 +148,15 @@ def scrub_environment() -> list[str]: export silently bills the API instead of the subscription (HANDOFF §2). """ removed = [] + # The GRAD_* fallbacks are derived from `ALL` rather than listed, and that + # is the fix for a real gap: the list used to be written out by hand, so + # `claude_oauth_token` and `asta_api_key` were added to `get()`'s lookup and + # not to this one. The agent inherits its environment, so each omission + # handed it exactly the environment-resident credential §9 argues must not + # exist -- and `GRAD_CLAUDE_OAUTH_TOKEN` is the worst of them, because + # `core/haiku.py` passes that token to subprocesses on purpose and the + # scrub is what bounds who else can read it. Deriving it means adding a + # credential cannot silently widen the boundary again. for var in ( "ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", @@ -123,16 +165,7 @@ def scrub_environment() -> list[str]: "OPENROUTER_API_KEY", "VOYAGE_API_KEY", "CONTEXT7_API_KEY", - # The GRAD_* fallbacks too. They exist for CI and first-run bootstrap, - # where no agent is running; leaving them in place under the agent would - # hand it exactly the environment-resident credentials §9 argues must - # not exist, and with them the ability to reach a remote without going - # through the submitters that hold the spend ceilings. - f"GRAD_{HF_TOKEN.upper()}", - f"GRAD_{OPENROUTER_KEY.upper()}", - f"GRAD_{VOYAGE_KEY.upper()}", - f"GRAD_{S2_KEY.upper()}", - f"GRAD_{CONTEXT7_KEY.upper()}", + *(f"GRAD_{name.upper()}" for name in ALL), ): if os.environ.pop(var, None) is not None: removed.append(var) diff --git a/core/haiku.py b/core/haiku.py index 50b6156..ba57080 100644 --- a/core/haiku.py +++ b/core/haiku.py @@ -27,9 +27,10 @@ import asyncio import json +import os from typing import Any, Callable -from core import paths, quota_log +from core import credentials, paths, quota_log from core.errors import ConfigError, UpstreamError from core.ledger_store import now_iso @@ -79,6 +80,37 @@ def _sdk() -> Any: """ +NOT_AUTHENTICATED = ( + "the funnel's Haiku stages have no subscription credentials, so the model was " + "never reached. The agent runs this CLI over Bash, and that hop strips " + "CLAUDE_CODE_OAUTH_TOKEN from the environment -- so the token has to come from " + "the credential store, not from the environment." +) +AUTH_FIX = ( + "claude setup-token # mint a token, then store it where the hop cannot strip it:\n" + "python -m tools.jobs credential set claude_oauth_token" +) + + +def _credentials_env() -> dict[str, str]: + """Subscription credentials for the CLI this stage spawns. + + The ambient variable comes first, so running a funnel stage by hand in a + terminal keeps working with no setup at all. The credential store is the + fallback that makes the same command work when the *agent* is the one + running it -- see `credentials.CLAUDE_TOKEN` for why the two differ. + + `ClaudeAgentOptions.env` merges over the inherited environment rather than + replacing it, so this adds one variable and takes nothing away. + """ + token = os.environ.get("CLAUDE_CODE_OAUTH_TOKEN") + if not token: + token = credentials.get(credentials.CLAUDE_TOKEN, required=False) + if not token: + raise ConfigError(NOT_AUTHENTICATED, fix=AUTH_FIX) + return {"CLAUDE_CODE_OAUTH_TOKEN": token} + + def _validate_expansion(args: dict[str, Any]) -> str | None: queries = args.get("queries") hyde = args.get("hyde") @@ -139,15 +171,32 @@ async def _submit(args: dict[str, Any]) -> dict[str, Any]: mcp_servers={"funnel": sdk.create_sdk_mcp_server("funnel", tools=[_submit])}, allowed_tools=[f"mcp__funnel__{tool_name}"], disallowed_tools=["Read", "Write", "Edit", "Bash", "Glob", "Grep", "WebSearch", "WebFetch"], + env=_credentials_env(), ) transcript: list[str] = [] usage: Any = None - async for message in sdk.query(prompt=user_prompt, options=options): - text = _text_of(message) - if text: - transcript.append(text) - usage = getattr(message, "usage", None) or usage + unauthenticated = False + try: + async for message in sdk.query(prompt=user_prompt, options=options): + # An unauthenticated CLI answers with a synthetic "Not logged in" + # turn and then exits non-zero. The SDK reports the exit as + # "Claude Code returned an error result: success" -- the CLI sends no + # `errors` array, so the SDK falls back to printing the result + # *subtype*, which is `success`. That message is worse than useless + # here, so the reason is taken from the message that carries it. + if getattr(message, "error", None) == "authentication_failed": + unauthenticated = True + text = _text_of(message) + if text: + transcript.append(text) + usage = getattr(message, "usage", None) or usage + except Exception as exc: # noqa: BLE001 - re-raised unless we know better + if unauthenticated: + raise ConfigError(NOT_AUTHENTICATED, fix=AUTH_FIX) from exc + raise + if unauthenticated: + raise ConfigError(NOT_AUTHENTICATED, fix=AUTH_FIX) quota_log.from_sdk_usage( stage, usage, model=model, role=role, diff --git a/core/http.py b/core/http.py index efb2d08..0ba2902 100644 --- a/core/http.py +++ b/core/http.py @@ -24,6 +24,61 @@ _last_request: dict[str, float] = {} +#: Where a paper's identity is read from, **in this order, everywhere**. +#: +#: The order is the point. `/snippet/search` returns `corpusId` and `paperId`; +#: `/paper/search` is asked for fields that include neither corpus id nor +#: anything but `paperId`; Asta returns whichever its own shape carries. Reading +#: them in different orders in different methods -- corpus id first in one, SHA +#: only in another -- gave the *same paper* two ids, and `corpus.rrf` fuses by +#: id, so it ranked twice and took a slot from something else. `cmd_search` +#: calls both endpoints for every expanded query, so that was the ordinary path +#: and not a corner of it. +#: +#: `paperId` leads because it is the one field every endpoint returns, and +#: because `paper_id` -- the seed `neighbours` expands from -- is read from it +#: too. One field decides both, so a candidate and its citation expansion cannot +#: disagree about which paper they are. +IDENTITY_KEYS = ("paperId", "paper_id", "corpusId", "corpus_id", "id") + + +def identifier_of(paper: dict[str, Any]) -> Any: + """A paper's identity, by `IDENTITY_KEYS`. None when it carries none.""" + for key in IDENTITY_KEYS: + value = paper.get(key) + if value not in (None, ""): + return value + return None + + +def candidate_id(identifier: Any, title: Any = "", text: Any = "") -> str | None: + """The key the funnel fuses candidates on, or None when there is not one. + + Both tier-1 clients mint ids in one `s2:` namespace, deliberately: it is one + corpus, so a paper found through both has to fuse to a single candidate + rather than rank twice under two names. + + That sharing is also why an id-less hit cannot be given a shared literal. + Formatting a missing identifier produced `"s2:None"`, and since `corpus.rrf` + fuses by id, *every* hit without one -- from either client, across every + query in the run -- collapsed into a single phantom candidate. Distinct + papers vanished into each other with nothing on screen to say so, which is + the same class of failure as an empty result that reads as "the literature + has nothing on this". + + So: the real id when there is one; a digest of what the reranker would read + when there is not, which fuses genuine duplicates and separates genuine + distinctions; and None when there is neither, because a hit with no id, no + title and no text has nothing to rank and nothing to cite. + """ + if identifier not in (None, ""): + return f"s2:{identifier}" + material = f"{title or ''}\n{str(text or '')[:400]}".strip() + if not material: + return None + return "s2:t-" + hashlib.sha256(material.encode("utf-8")).hexdigest()[:16] + + def _httpx() -> Any: try: import httpx # noqa: PLC0415 @@ -111,9 +166,14 @@ def snippet_search(self, query: str, limit: int = 20) -> list[dict[str, Any]]: for item in data.get("data", []): snippet = item.get("snippet", {}) paper = item.get("paper", {}) + key = candidate_id( + identifier_of(paper), paper.get("title"), snippet.get("text", "") + ) + if key is None: + continue out.append( { - "id": f"s2:{paper.get('corpusId') or paper.get('paperId')}", + "id": key, "paper_id": paper.get("paperId"), "title": paper.get("title"), "year": (paper.get("publicationDate") or "")[:4] or None, @@ -128,19 +188,24 @@ def snippet_search(self, query: str, limit: int = 20) -> list[dict[str, Any]]: def paper_search(self, query: str, limit: int = 20) -> list[dict[str, Any]]: fields = "title,abstract,year,externalIds,citationCount,authors" data = self._get("/paper/search", {"query": query, "limit": limit, "fields": fields}) - return [ - { - "id": f"s2:{p.get('paperId')}", - "paper_id": p.get("paperId"), - "title": p.get("title"), - "year": p.get("year"), - "abstract": p.get("abstract") or "", - "citations": p.get("citationCount"), - "source": "s2.paper", - "external": p.get("externalIds", {}), - } - for p in data.get("data", []) - ] + out = [] + for p in data.get("data", []): + key = candidate_id(identifier_of(p), p.get("title"), p.get("abstract")) + if key is None: + continue + out.append( + { + "id": key, + "paper_id": p.get("paperId"), + "title": p.get("title"), + "year": p.get("year"), + "abstract": p.get("abstract") or "", + "citations": p.get("citationCount"), + "source": "s2.paper", + "external": p.get("externalIds", {}), + } + ) + return out def neighbours(self, paper_id: str, *, direction: str = "citations", limit: int = 20) -> list[dict[str, Any]]: """Citation-graph expansion. @@ -171,6 +236,380 @@ def neighbours(self, paper_id: str, *, direction: str = "citations", limit: int return out +# --------------------------------------------------------------------------- +# Asta -- the same corpus, through a door that opens +# --------------------------------------------------------------------------- +#: The MCP protocol version this client speaks. Sent on `initialize` and echoed +#: on every request after it; a server that wants another version says so in its +#: `initialize` result and this follows it. +MCP_PROTOCOL_VERSION = "2025-06-18" + + +class Asta: + """Ai2's scientific corpus over MCP, at `asta-tools.allen.ai`. + + **Why this exists.** `SemanticScholar` above is the better-documented client + and it is not reachable: Ai2 stopped accepting API key requests from + free-domain email addresses, so a personal account cannot get one, and the + anonymous pool is shared with every other unauthenticated caller and is + near-permanently rate limited. Asta is the *same index* -- Ai2 describe the + MCP tool as an extension of the Semantic Scholar API -- and it exposes + `snippet_search`, which is the endpoint §5's funnel is actually built around: + ~500-word excerpts from full text are what make triage possible without + downloading anything. A key is optional here and raises limits rather than + unlocking anything. + + **Why it is not an MCP integration.** §5 already settles this: Asta's + endpoint is reached "over streamable HTTP without adopting MCP as an + architecture". Streamable HTTP is a POST with a JSON-RPC body; the parts of + MCP that would be an architecture -- a client runtime, a tool registry, a + server lifecycle -- buy nothing when the whole surface is three calls. So + this is `httpx` and the same disk cache, rate limiter and usage log as + everything else in this module. + + **What is unverified.** The endpoint, the transport and the tool names are + from Ai2's published documentation. The *shape of each tool's result* is + not, because that needs a live call. So `_rows` reads both the shape S2's + REST API uses (`{"data": [{"snippet": …, "paper": …}]}`) and a flattened + one, and an unrecognised payload becomes an `UpstreamError` naming what came + back rather than an empty list that reads as "the literature has nothing". + """ + + def __init__(self, cfg: Config) -> None: + self.base = str(cfg.get("retrieval", "asta_base")).rstrip("/") + self.timeout = float(cfg.get("retrieval", "request_timeout_s", 60)) + self.ttl = float(cfg.get("retrieval", "cache_ttl_s", 604800)) + self.interval = float(cfg.get("retrieval", "min_request_interval_s", 1.1)) + try: + self.key = credentials.get(credentials.ASTA_KEY, required=False) + except ConfigError: + # Same reasoning as Context7: an optional credential whose *store* is + # unreachable must not make an anonymous call impossible. + self.key = None + self._session: str | None = None + self._protocol = MCP_PROTOCOL_VERSION + self._id = 0 + #: Set only once `initialize` *and* the notification after it have + #: succeeded. See `_handshake` for why this is not inferred. + self._ready = False + + @property + def authenticated(self) -> bool: + return bool(self.key) + + # -- transport ---------------------------------------------------------- + def _headers(self) -> dict[str, str]: + headers = { + "Content-Type": "application/json", + # Both, because streamable HTTP lets the server answer either way + # for the same request and does not tell you which in advance. + "Accept": "application/json, text/event-stream", + "MCP-Protocol-Version": self._protocol, + } + if self.key: + headers["x-api-key"] = self.key + if self._session: + headers["Mcp-Session-Id"] = self._session + return headers + + def _post(self, body: dict[str, Any]) -> Any: + _throttle("asta", self.interval) + httpx = _httpx() + try: + resp = httpx.post(self.base, json=body, headers=self._headers(), timeout=self.timeout) + except Exception as exc: # noqa: BLE001 + raise UpstreamError( + f"Asta request failed: {exc}", + fix="retry, or run with --local-only to search papers already ingested", + ) from exc + + # Assigned on `initialize` and echoed from then on. A server that does + # not use sessions simply never sends it. + session = resp.headers.get("mcp-session-id") + if session: + self._session = session + + if resp.status_code == 401 or resp.status_code == 403: + raise UpstreamError( + f"Asta rejected the request ({resp.status_code})", + fix=( + "the corpus tool is usable anonymously; if a key is stored it may be " + f"wrong: python -m tools.jobs credential set {credentials.ASTA_KEY}" + ), + ) + if resp.status_code == 429: + raise UpstreamError( + "Asta rate-limited the request", + fix=( + "wait, or store a key to raise the limit -- it is requested from a form " + "rather than reviewed, so a personal address is fine: " + f"python -m tools.jobs credential set {credentials.ASTA_KEY}" + ), + ) + if resp.status_code >= 400: + raise UpstreamError( + f"Asta returned {resp.status_code}: {resp.text[:200]}", + fix=( + "the endpoint may have moved: check allenai.org/asta/resources/mcp and " + "set [retrieval] asta_base in config/grad.toml" + ), + ) + # A notification gets 202 Accepted and an empty body; there is nothing + # to parse and nothing to wait for. + if resp.status_code == 202 or not (resp.content or b"").strip(): + return None + return _mcp_payload(resp) + + def _call(self, method: str, params: dict[str, Any] | None = None) -> Any: + self._id += 1 + payload = self._post( + {"jsonrpc": "2.0", "id": self._id, "method": method, "params": params or {}} + ) + if payload is None: + raise UpstreamError( + f"Asta returned no body for {method}", + fix="retry; if it persists the endpoint may have changed transport", + ) + error = payload.get("error") if isinstance(payload, dict) else None + if error: + raise UpstreamError( + f"Asta refused {method}: {error.get('message') or error}", + fix="check the tool name and its arguments against allenai.org/asta/resources/mcp", + ) + return (payload or {}).get("result") + + def _handshake(self) -> None: + """`initialize`, then the notification that says the client is ready. + + Once per client, and `_ready` is an explicit flag rather than something + inferred from `_session` or `_id`. Inferring it from the request counter + was wrong in the case that matters: `_call` increments the counter before + it sends, so an `initialize` that *failed* -- a timeout, a 429 on the + very first call -- left the counter non-zero and every later request + skipped the handshake and went straight to `tools/call` on a connection + that was never initialised. One transient failure poisoned the client for + the life of the process. + """ + if self._ready: + return + result = self._call( + "initialize", + { + "protocolVersion": MCP_PROTOCOL_VERSION, + "capabilities": {}, + "clientInfo": {"name": "grad", "version": "1"}, + }, + ) + negotiated = (result or {}).get("protocolVersion") + if isinstance(negotiated, str) and negotiated: + self._protocol = negotiated + # A notification: no id, so no reply is expected and none is waited for. + self._post({"jsonrpc": "2.0", "method": "notifications/initialized", "params": {}}) + self._ready = True + + def tool(self, name: str, arguments: dict[str, Any]) -> Any: + """Call one MCP tool, through the disk cache. + + The cache key deliberately excludes the session: a session is a + transport detail and two of them asking the same question of the same + corpus should not cost two requests. + """ + key = f"asta:{self.base}:{name}:{json.dumps(arguments, sort_keys=True)}" + hit = _cached(key, self.ttl) + if hit is not None: + return hit + self._handshake() + result = self._call("tools/call", {"name": name, "arguments": arguments}) + if isinstance(result, dict) and result.get("isError"): + raise UpstreamError( + f"Asta's {name} failed: {_mcp_text(result)[:200]}", + fix="check the arguments; the tool ran and reported an error", + ) + data = _mcp_result(result) + _store(key, data) + return data + + # -- the three calls the funnel makes ------------------------------------ + def snippet_search(self, query: str, limit: int = 20) -> list[dict[str, Any]]: + """Full-text excerpts. The reason to prefer this over a metadata search: + ~500 words of the paper itself is what stage 3 triages on.""" + data = self.tool("snippet_search", {"query": query, "limit": limit}) + return _normalise(_rows(data, "snippet_search"), "asta.snippet") + + def paper_search(self, query: str, limit: int = 20) -> list[dict[str, Any]]: + data = self.tool("search_papers_by_relevance", {"query": query, "limit": limit}) + return _normalise(_rows(data, "search_papers_by_relevance"), "asta.paper") + + def neighbours( + self, paper_id: str, *, direction: str = "citations", limit: int = 20 + ) -> list[dict[str, Any]]: + """Citation-graph expansion. + + §5: worth more for recall than any reranker upgrade, because the + retriever sets the ceiling and the graph reaches papers no query string + does. Asta publishes `get_citations` and no references counterpart, so + the backward direction is refused here rather than silently answered + with the forward one -- which would quietly double-count one direction. + """ + if direction != "citations": + return [] + data = self.tool("get_citations", {"paper_id": paper_id, "limit": limit}) + return _normalise(_rows(data, "get_citations"), "asta.citations") + + +def _mcp_payload(resp: Any) -> Any: + """One JSON-RPC message out of a streamable-HTTP response. + + The same request may be answered with `application/json` or with an SSE + stream, at the server's discretion, so both are handled. For a stream the + *last* `data:` frame carrying a result is taken: progress notifications + share the channel with the answer. + """ + content_type = (resp.headers.get("content-type") or "").lower() + if "text/event-stream" not in content_type: + try: + return resp.json() + except Exception as exc: # noqa: BLE001 + raise UpstreamError( + f"Asta returned a body that is not JSON: {resp.text[:200]}", + fix="retry; if it persists the endpoint may no longer speak streamable HTTP", + ) from exc + + answer: Any = None + for line in resp.text.splitlines(): + if not line.startswith("data:"): + continue + try: + frame = json.loads(line[5:].strip()) + except json.JSONDecodeError: + continue + if isinstance(frame, dict) and ("result" in frame or "error" in frame): + answer = frame + if answer is None: + raise UpstreamError( + "Asta's event stream carried no result", + fix="retry; the stream held only notifications", + ) + return answer + + +def _mcp_text(result: Any) -> str: + """The text content blocks of a `tools/call` result, joined.""" + if not isinstance(result, dict): + return "" + parts = [] + for block in result.get("content") or []: + if isinstance(block, dict) and isinstance(block.get("text"), str): + parts.append(block["text"]) + return "\n".join(parts) + + +def _mcp_result(result: Any) -> Any: + """The data a tool returned, whichever way it chose to return it. + + `structuredContent` is the typed channel and is preferred. Failing that the + text block is usually JSON; failing *that* it is prose, and it is handed back + as-is rather than discarded -- `_rows` is where an unusable shape becomes an + error that says what arrived. + """ + if isinstance(result, dict) and result.get("structuredContent") is not None: + return result["structuredContent"] + text = _mcp_text(result) + try: + return json.loads(text) + except (json.JSONDecodeError, TypeError): + return text + + +def _rows(data: Any, tool_name: str) -> list[dict[str, Any]]: + """The list of hits inside a tool's payload, whatever it is wrapped in. + + Unverified against the live service, so this reads the S2 REST shape and the + obvious flattenings of it. An unrecognised payload raises: a search that + quietly returns nothing reads as "the literature has nothing on this", and + that is a conclusion nobody should draw from a schema change. + """ + if isinstance(data, list): + candidates = data + elif isinstance(data, dict): + for key in ("data", "results", "snippets", "papers", "citations", "items"): + if isinstance(data.get(key), list): + candidates = data[key] + break + else: + raise UpstreamError( + f"Asta's {tool_name} returned no recognisable list of hits: " + f"keys were {sorted(data)[:8]}", + fix=( + "the tool's result shape has changed -- compare it against " + "allenai.org/asta/resources/mcp and update core/http.py:_rows" + ), + ) + else: + raise UpstreamError( + f"Asta's {tool_name} returned {type(data).__name__}, not a result set: " + f"{str(data)[:200]}", + fix="retry; if it persists the tool name or its arguments have changed", + ) + return [c for c in candidates if isinstance(c, dict)] + + +def _normalise(items: list[dict[str, Any]], source: str) -> list[dict[str, Any]]: + """Hits in the funnel's vocabulary, dropping any that cannot be fused. + + A hit with no identifier, no title and no text has nothing for the reranker + to read and nothing to cite. Dropping it costs no recall; keeping it under a + shared placeholder id cost real recall, because fusion is by id -- see + `candidate_id`. + """ + out = [] + for item in items: + row = _row(item, source=source) + if row is not None: + out.append(row) + return out + + +def _row(item: dict[str, Any], *, source: str) -> dict[str, Any] | None: + """One hit, in the shape the funnel already fuses and reranks. + + `paper_search.py` does not know which tier a candidate came from, and it + must not have to: RRF fuses rankings by id, and the reranker reads title and + snippet. So the two clients in this module answer in one vocabulary. + """ + # The `s2:` prefix is shared with `SemanticScholar` on purpose, not by + # accident of copying: it is the same corpus and the same corpus ids, so a + # paper found through both tiers has to fuse to one candidate rather than + # rank twice under two names. + # + # The S2 shape nests the paper under the snippet; the flat one does not. + paper = item.get("paper") if isinstance(item.get("paper"), dict) else item + snippet = item.get("snippet") if isinstance(item.get("snippet"), dict) else item + + text = snippet.get("text") or item.get("text") or "" + key = candidate_id(identifier_of(paper), paper.get("title"), text) + if key is None: + return None + year = paper.get("year") + if year is None: + year = (str(paper.get("publicationDate") or "")[:4]) or None + external = paper.get("externalIds") or paper.get("external_ids") or {} + return { + "id": key, + # The same field the id came from, by the same order -- so the seed + # `neighbours` expands from cannot name a different paper than the + # candidate it was taken from. + "paper_id": paper.get("paperId") or paper.get("paper_id") or paper.get("id"), + "title": paper.get("title"), + "year": year, + "snippet": text, + "abstract": paper.get("abstract") or "", + "section": snippet.get("snippetKind") or snippet.get("section") or "", + "source": source, + "external": external if isinstance(external, dict) else {}, + } + + # --------------------------------------------------------------------------- # Context7 (HANDOFF-2 §18) -- what is *current*, as opposed to what is installed # --------------------------------------------------------------------------- diff --git a/core/jsonl.py b/core/jsonl.py index 14e2f96..af0ff9f 100644 --- a/core/jsonl.py +++ b/core/jsonl.py @@ -12,9 +12,12 @@ * readers tolerate a torn final line, because a reader may open the file between a partial write and its flush. -`portalocker` is used when installed; otherwise we fall back to `msvcrt` on -Windows and `fcntl` elsewhere. The fallback is real, not decorative -- the -ledger must not depend on an optional package to stay uncorrupted. +Windows locks by hand, POSIX through `portalocker` when it is installed and +`fcntl` when it is not. The reason that split is not a preference is spelled out +above `_lock`; the short version is that a Windows lock denies reads, so it has +to be taken somewhere other than over the data. The `fcntl` fallback is real, +not decorative -- the ledger must not depend on an optional package to stay +uncorrupted. """ from __future__ import annotations @@ -50,48 +53,62 @@ def _thread_lock(path: Path) -> threading.Lock: return lock -try: # pragma: no cover - exercised by whichever branch the machine has - import portalocker - +# Which backend locks which platform is decided by one asymmetry: **Windows +# byte-range locks are mandatory and POSIX ones are advisory.** A locked region +# on Windows denies reads as well as writes, to every handle including another +# one in this process; `fcntl.flock` denies nothing and only excludes other +# lockers. So on Windows *where* the lock is taken is load-bearing, and on POSIX +# it is not. +# +# On Windows all writers therefore contend on one fixed sentinel byte positioned +# far past any real ledger, never over the data. A lock over live data would make +# concurrent readers -- and a `precondition` that consults the file it is being +# appended to -- fail with PermissionError. +# +# `portalocker` cannot express that, which is why it is not used here: its +# `MsvcrtLocker` normalises the file position to 0 and locks 64 KiB from there, +# which is exactly over the data. Preferring it on Windows is what made +# `campaign.request_halt` and `ledger_store`'s uniqueness check fail against +# their own ledgers, and it denied the UI's two-second poll for the length of +# every append. +if os.name == "nt": # pragma: no cover - one branch per platform + import msvcrt + + # msvcrt locks a byte range at the *current file position*, and a handle + # opened in append mode starts at EOF -- so left alone, every writer would + # lock a different byte as the file grows and none would exclude any other. + # The position is set with os.lseek on the descriptor, which is what + # msvcrt.locking reads; O_APPEND still sends every write to EOF. def _lock(fh) -> None: - portalocker.lock(fh, portalocker.LOCK_EX) + deadline = time.monotonic() + _LOCK_TIMEOUT_S + while True: + try: + os.lseek(fh.fileno(), _LOCK_OFFSET, os.SEEK_SET) + msvcrt.locking(fh.fileno(), msvcrt.LK_NBLCK, 1) + return + except OSError: + if time.monotonic() > deadline: + raise + time.sleep(_LOCK_POLL_S) def _unlock(fh) -> None: - portalocker.unlock(fh) - -except ImportError: # pragma: no cover - if os.name == "nt": - import msvcrt - - # msvcrt locks a byte range at the *current file position*, and a handle - # opened in append mode starts at EOF -- so left alone, every writer - # locks a different byte as the file grows and none excludes any other. - # All writers therefore contend on one fixed sentinel byte, positioned - # far past any real ledger: a Windows lock denies reads as well as - # writes, so a lock over live data would make readers (and a - # precondition that consults the file) fail with PermissionError. - # The position is set with os.lseek on the descriptor, which is what - # msvcrt.locking reads; O_APPEND still sends every write to EOF. + try: + os.lseek(fh.fileno(), _LOCK_OFFSET, os.SEEK_SET) + msvcrt.locking(fh.fileno(), msvcrt.LK_UNLCK, 1) + except OSError: + pass + +else: # pragma: no cover - one branch per platform + try: + import portalocker + def _lock(fh) -> None: - deadline = time.monotonic() + _LOCK_TIMEOUT_S - while True: - try: - os.lseek(fh.fileno(), _LOCK_OFFSET, os.SEEK_SET) - msvcrt.locking(fh.fileno(), msvcrt.LK_NBLCK, 1) - return - except OSError: - if time.monotonic() > deadline: - raise - time.sleep(_LOCK_POLL_S) + portalocker.lock(fh, portalocker.LOCK_EX) def _unlock(fh) -> None: - try: - os.lseek(fh.fileno(), _LOCK_OFFSET, os.SEEK_SET) - msvcrt.locking(fh.fileno(), msvcrt.LK_UNLCK, 1) - except OSError: - pass + portalocker.unlock(fh) - else: + except ImportError: import fcntl def _lock(fh) -> None: @@ -189,12 +206,18 @@ def damaged_lines(path: Path | str) -> list[int]: def write_json(path: Path | str, obj: Any) -> None: """Atomic whole-file JSON write, for the preflight records in §6. - These are single-writer and replaced wholesale, so a temp file plus - os.replace is the right tool rather than the append lock. + These are replaced wholesale, so a temp file plus `os.replace` is the right + tool rather than the append lock. + + The temp name carries the *thread* as well as the process. The UI persists a + layout through here, one `Workspace` per connected client in one process -- + so two windows open on the same project are two threads writing the same + path, and a pid-only name would give them the same temp file to interleave + into. `os.replace` would then publish whichever half won. """ path = Path(path) path.parent.mkdir(parents=True, exist_ok=True) - tmp = path.with_suffix(path.suffix + f".tmp{os.getpid()}") + tmp = path.with_suffix(path.suffix + f".tmp{os.getpid()}.{threading.get_ident():x}") tmp.write_text(json.dumps(obj, indent=2, ensure_ascii=False, default=str), encoding="utf-8") os.replace(tmp, path) diff --git a/core/paths.py b/core/paths.py index 2d6578c..a6072be 100644 --- a/core/paths.py +++ b/core/paths.py @@ -11,10 +11,24 @@ def root() -> Path: - """Workspace root. GRAD_ROOT overrides, otherwise the repo directory.""" + """Workspace root: GRAD_ROOT, then the remembered choice, then the repo. + + The middle rule is what makes the app's folder chooser survive a restart; + `core/workspace.py` holds the pointer and explains why it is stored beside + the code rather than inside the workspace it names. GRAD_ROOT still wins, so + an explicit override -- the test suite's, or one typed on a command line -- + is never quietly beaten by a remembered one. + """ env = os.environ.get("GRAD_ROOT") if env: return Path(env).resolve() + # Imported here rather than at module scope: `workspace` raises the CLI's + # error type, and this module is imported by almost everything. + from core import workspace # noqa: PLC0415 + + chosen = workspace.remembered() + if chosen is not None: + return chosen return Path(__file__).resolve().parent.parent diff --git a/core/workspace.py b/core/workspace.py new file mode 100644 index 0000000..b5ab1be --- /dev/null +++ b/core/workspace.py @@ -0,0 +1,205 @@ +"""Which directory the workspace *is*, and remembering the answer. + +Every path in the system derives from `paths.root()`, which used to be GRAD_ROOT +or the directory the code sits in. The app can now be pointed at another +workspace from its own menu, and that needs two things this module owns: a +validation step, because the value arrives from a text field, and somewhere to +remember the choice. + +**Precedence, highest first:** + +1. `GRAD_ROOT` in the environment. An explicit override stays explicit -- the + test suite sets it, and a remembered choice must never quietly beat someone + who typed it on the command line. +2. The pointer file, written by the app's folder chooser. +3. The directory the code is installed in, which is what a fresh checkout gets. + +**The pointer lives beside the code, not in `data/`.** That is the whole reason +it works: a pointer stored inside the workspace it points away from is +unreadable the moment you leave, so the app could never find its way back. + +Switching is applied by setting `GRAD_ROOT` in this process's environment, which +is also how it reaches the CLIs -- they run as subprocesses and inherit it, so +the agent's Bash tools and the UI cannot end up reading different ledgers. +""" + +from __future__ import annotations + +import json +import logging +import os +from pathlib import Path +from typing import Any + +from core.errors import UsageError + +log = logging.getLogger("grad.workspace") + +#: How many previous roots the chooser offers. Long enough to switch back and +#: forth between a couple of projects, short enough to read at a glance. +MAX_RECENT = 8 + +_cache: dict[str, Any] | None = None + + +def code_dir() -> Path: + """Where Grad itself is installed. Computed here rather than imported from + `paths`, which would be a cycle: `paths.root()` consults this module.""" + return Path(__file__).resolve().parent.parent + + +def pointer_path() -> Path: + return code_dir() / ".grad-workspace.json" + + +def read_pointer(*, reload: bool = False) -> dict[str, Any]: + """The pointer file, cached. + + `paths.root()` is called for every path in the system, so an uncached read + here would be a JSON parse per path lookup. The cache is invalidated by + `select`, which is the only thing that writes. + """ + global _cache + if _cache is not None and not reload: + return _cache + data: dict[str, Any] = {} + path = pointer_path() + try: + if path.exists(): + loaded = json.loads(path.read_text(encoding="utf-8")) + if isinstance(loaded, dict): + data = loaded + except (OSError, json.JSONDecodeError): + # A hand-edited or unreadable pointer must not stop the app starting; + # falling back to the installed directory is always a valid answer. + log.debug("could not read %s", path) + _cache = data + return data + + +def remembered() -> Path | None: + """The remembered root, if it is still a usable directory. + + Checked rather than trusted: the folder may have been deleted, renamed or + live on a drive that is not mounted today. Returning a path that no longer + exists would send every ledger read to a directory that cannot be created. + """ + value = read_pointer().get("root") + if not isinstance(value, str) or not value: + return None + try: + path = Path(value).expanduser().resolve() + except (OSError, ValueError): + return None + return path if path.is_dir() else None + + +def recent() -> list[Path]: + """Previously chosen roots, most recent first, filtered to those that exist.""" + values = read_pointer().get("recent") + out: list[Path] = [] + if not isinstance(values, list): + return out + for value in values: + if not isinstance(value, str): + continue + try: + path = Path(value).expanduser().resolve() + except (OSError, ValueError): + continue + if path.is_dir() and path not in out: + out.append(path) + return out[:MAX_RECENT] + + +def source() -> str: + """Which rule decided the current root. Shown in the chooser, because + "why is it still pointing there?" is otherwise unanswerable from the UI.""" + if os.environ.get("GRAD_ROOT"): + return "environment" + return "remembered" if remembered() is not None else "default" + + +def validate(candidate: str | Path, *, create: bool = False) -> Path: + """Resolve a candidate root, or refuse it with a fix. + + The value comes from a text field, so every failure it can have is a message + someone has to act on: blank, a file rather than a directory, a path that + does not exist, a directory that cannot be written to. + """ + text = str(candidate or "").strip().strip('"') + if not text: + raise UsageError("choose a folder for the workspace", fix="pick one, or type a path") + try: + path = Path(text).expanduser().resolve() + except (OSError, ValueError) as exc: + raise UsageError(f"{text!r} is not a usable path: {exc}") from exc + + if path.exists() and not path.is_dir(): + raise UsageError( + f"{path} is a file, not a folder", + fix="choose the directory that contains it", + ) + if not path.exists(): + if not create: + raise UsageError(f"{path} does not exist", fix="create it, or choose another folder") + try: + path.mkdir(parents=True, exist_ok=True) + except OSError as exc: + raise UsageError(f"could not create {path}: {exc}") from exc + + # Checked by asking the filesystem, not by inspecting the mode bits: a + # network share, a read-only mount and a permission denial all present + # differently, and all of them mean the same thing here. + if not os.access(path, os.W_OK): + raise UsageError( + f"{path} cannot be written to", + fix="choose a folder you own, or fix its permissions", + ) + return path + + +def select(candidate: str | Path, *, create: bool = False) -> Path: + """Point this process, and everything it starts, at another workspace. + + Setting `GRAD_ROOT` rather than caching a value is what makes the switch + total: `paths.root()` reads the environment first, and every CLI the UI or + the agent shells out to inherits it. The alternative -- a module-level + override consulted only by `paths` -- would leave subprocesses reading the + old workspace while the UI showed the new one. + """ + path = validate(candidate, create=create) + # Imported lazily in both directions: `paths.root()` consults this module. + from core import paths # noqa: PLC0415 + + leaving = paths.root() + os.environ["GRAD_ROOT"] = str(path) + _write_pointer(path, leaving=leaving) + return path + + +def _write_pointer(path: Path, *, leaving: Path | None = None) -> None: + """Write the pointer, and remember the folder being left. + + Recording only where you are *going* looks right and leaves the history + empty exactly when it matters: after the first switch the only entry is the + folder you are now in, which the menu filters out as somewhere you already + are. Switching back -- the whole reason to keep a list -- would be the one + thing it could not offer. So the folder being left goes in first. + """ + history = [p for p in recent() if p != path] + if leaving is not None and leaving != path and leaving.is_dir(): + history = [leaving, *[p for p in history if p != leaving]] + previous = [str(p) for p in history] + payload = {"root": str(path), "recent": [str(path), *previous][:MAX_RECENT]} + global _cache + _cache = payload + try: + pointer_path().write_text( + json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8" + ) + except OSError: + # An installation directory that cannot be written to is a real + # situation -- a system-wide install, a read-only image. The switch + # still applies to this process; it just will not be remembered. + log.debug("could not persist the workspace pointer to %s", pointer_path()) diff --git a/design_handoff_grad_ui/README.md b/design_handoff_grad_ui/README.md index 354633a..555f16d 100644 --- a/design_handoff_grad_ui/README.md +++ b/design_handoff_grad_ui/README.md @@ -175,9 +175,12 @@ Standalone reference: `reference/Notebook Paper.dc.html`. - **Gate card**: 2px `#A3122F` border, solid `#A3122F` header (`GATE — YOUR CALL`), a sentence naming the exact cost and resource, then `✓ APPROVE` (teal), `✎ EDIT PLAN`, `✕ DENY` — all 2px ink borders. -- **Composer**: mode chips (`ASK` active ink / `PLAN` / `RUN`), `@notebook @paper @wiki` +- **Composer**: ~~mode chips (`ASK` active ink / `PLAN` / `RUN`)~~, `@notebook @paper @wiki` mention hint, a 2px-bordered field on `#FFFDF8` with a blinking caret, and a `SEND ⏎` button on `#FFD400`. + *Not built: the mode chips were dropped. They only prefixed the prompt with + `[plan]` / `[run]`, and nothing downstream — system prompt, gates, CLIs — ever + gave those tokens a meaning. There is one agent mode.* ### 3. Wiki + references @@ -257,8 +260,12 @@ border+fill failed, dashed empty when queued. State chips: `RUNNING` teal, ## Interactions & behaviour -- **Tiling**: panes resize by dragging the 8px handles; `⌥`+drag a title bar to retile; +- **Tiling**: panes resize by dragging the 8px handles; ~~`⌥`+drag a title bar to retile~~; `⌥1/⌥2/⌥3` switch tile/stack/full. Persist layout per project. Minimum pane width 320px. + *Changed: the retile drag needs no modifier. Dragging a title bar shows a drop + indicator and inserts at the position it marks; dropping onto another window's + title bar swaps the two panes; dropping at a column's edge splits a new column, + up to the three-column cap.* - **Window opener**: clicking a name opens it into the focused pane (or splits if the pane already holds one); clicking an open name closes it. - **Notebook**: Run/Run-all/Stop/Restart map to Jupyter kernel commands. `VERIFY` runs the diff --git a/tests/conftest.py b/tests/conftest.py index 7cfa7b5..ce358b1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,8 +1,8 @@ from __future__ import annotations -import os import sys from pathlib import Path +from typing import Any import pytest @@ -28,6 +28,49 @@ def workspace(tmp_path, monkeypatch): config._cache.clear() +@pytest.fixture(autouse=True) +def clean_process_state(): + """Module-level registries outlive a fixture, so they are emptied around + every test. Both exist for the same reason -- a task and a session are + process-wide facts rather than per-client ones -- and both would otherwise + let one test decide another's outcome.""" + from ui import sessions, tasks + + tasks.reset() + sessions.reset_claims() + yield + tasks.reset() + sessions.reset_claims() + + +@pytest.fixture(autouse=True) +def no_network(monkeypatch): + """Every HTTP client in this project goes through `core.http._httpx`, so + replacing it is enough to make the suite's "no network" claim mechanical. + + It was not, and the failure mode is why this exists rather than being left + to discipline. Changing the funnel's default tier-1 client from Semantic + Scholar to Asta left two tests monkeypatching the class that was no longer + constructed -- so instead of failing they began POSTing to a real endpoint, + with a 60-second timeout and a rate limiter between calls. A suite that + reaches the network does not fail; it *hangs*, which is the one outcome that + does not point at its own cause. + + A test that wants a fake sets `_httpx` itself; monkeypatch applies in order, + so its patch replaces this one for the duration. + """ + from core import http + + def refuse() -> Any: + raise AssertionError( + "a test reached for the network. Fake the client it uses " + "(monkeypatch core.http._httpx, or the class on core.http) -- the suite " + "runs with no network by design, and a real call hangs rather than fails." + ) + + monkeypatch.setattr(http, "_httpx", refuse) + + @pytest.fixture def cfg(): from core import config diff --git a/tests/test_asta.py b/tests/test_asta.py new file mode 100644 index 0000000..8c9ebe8 --- /dev/null +++ b/tests/test_asta.py @@ -0,0 +1,455 @@ +"""Asta over streamable HTTP: the transport, and the shapes it may answer in. + +The endpoint, the transport and the tool names come from Ai2's published +documentation. **The shape of each tool's result does not** -- that needs a live +call with a real corpus behind it, and this suite has neither. So the tests here +are about what `core/http.py` promises regardless of the shape: + + * one JSON-RPC message is recovered whether the server answers with JSON or + with an event stream, because streamable HTTP lets it pick either; + * the handshake happens once and its session id is carried afterwards; + * every plausible envelope around the hits is read; + * an envelope that is *not* recognised raises, rather than returning an empty + list -- a search that quietly finds nothing reads as "the literature has + nothing on this", and that is the one conclusion a schema change must not be + able to manufacture; + * both tier-1 clients answer in one vocabulary, so the funnel cannot tell them + apart and a paper found by both fuses to one candidate. +""" + +from __future__ import annotations + +import json + +import pytest + +from core import config as config_mod, http +from core.errors import UpstreamError + + +class FakeResponse: + def __init__(self, status_code=200, payload=None, text="", content_type="application/json", + headers=None): + self.status_code = status_code + self._payload = payload if payload is not None else {} + self.text = text or (json.dumps(self._payload) if payload is not None else "") + self.headers = {"content-type": content_type, **(headers or {})} + self.content = self.text.encode() + + def json(self): + return self._payload + + +def rpc(result) -> dict: + return {"jsonrpc": "2.0", "id": 1, "result": result} + + +def tool_result(payload) -> dict: + """A `tools/call` result the way MCP wraps one: text content blocks.""" + return {"content": [{"type": "text", "text": json.dumps(payload)}], "isError": False} + + +@pytest.fixture +def transport(monkeypatch): + """A fake streamable-HTTP endpoint. `queue` is the reply to each POST in + order; anything left unqueued replies with an empty handshake result.""" + posts: list[dict] = [] + queue: list[FakeResponse] = [] + + class FakeHttpx: + @staticmethod + def post(url, json=None, headers=None, timeout=None): + posts.append({"url": url, "body": json, "headers": headers}) + if queue: + return queue.pop(0) + return FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"})) + + monkeypatch.setattr(http, "_httpx", lambda: FakeHttpx) + monkeypatch.setattr(http.credentials, "get", lambda name, required=True: None) + return posts, queue + + +def client(**_): + return http.Asta(config_mod.load(reload=True)) + + +def methods(posts) -> list[str]: + return [p["body"]["method"] for p in posts] + + +# --------------------------------------------------------------------------- +# the handshake +# --------------------------------------------------------------------------- +def test_the_first_call_initialises_then_notifies_then_calls_the_tool(workspace, transport): + posts, queue = transport + queue.append(FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"}))) + queue.append(FakeResponse(status_code=202, text="")) + queue.append(FakeResponse(payload=rpc(tool_result({"data": []})))) + + client().snippet_search("attention") + assert methods(posts) == ["initialize", "notifications/initialized", "tools/call"] + assert posts[-1]["body"]["params"]["name"] == "snippet_search" + + +def test_the_session_id_is_carried_after_the_handshake_assigns_it(workspace, transport): + posts, queue = transport + queue.append( + FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"}), + headers={"mcp-session-id": "sess-42"}) + ) + queue.append(FakeResponse(status_code=202, text="")) + queue.append(FakeResponse(payload=rpc(tool_result({"data": []})))) + + client().snippet_search("attention") + assert posts[0]["headers"].get("Mcp-Session-Id") is None, "nothing to send yet" + assert posts[-1]["headers"]["Mcp-Session-Id"] == "sess-42" + + +def test_a_negotiated_protocol_version_is_the_one_sent_afterwards(workspace, transport): + """The server picks the version; a client that keeps announcing its own + preference after being told otherwise is not speaking the protocol.""" + posts, queue = transport + queue.append(FakeResponse(payload=rpc({"protocolVersion": "2099-01-01"}))) + queue.append(FakeResponse(status_code=202, text="")) + queue.append(FakeResponse(payload=rpc(tool_result({"data": []})))) + + client().snippet_search("attention") + assert posts[-1]["headers"]["MCP-Protocol-Version"] == "2099-01-01" + + +def test_a_stored_key_is_sent_and_its_absence_is_not_an_error(workspace, transport, monkeypatch): + posts, queue = transport + assert client().authenticated is False + + monkeypatch.setattr( + http.credentials, "get", + lambda name, required=True: "k-1" if name == http.credentials.ASTA_KEY else None, + ) + keyed = client() + assert keyed.authenticated is True + + queue.append(FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"}))) + queue.append(FakeResponse(status_code=202, text="")) + queue.append(FakeResponse(payload=rpc(tool_result({"data": []})))) + keyed.snippet_search("attention") + assert posts[-1]["headers"]["x-api-key"] == "k-1" + + +def test_an_unreachable_credential_store_still_allows_an_anonymous_call(workspace, transport): + """Same reasoning as Context7: an optional credential whose *store* is + missing must not make an anonymous call impossible.""" + from core.errors import ConfigError + + def explode(name, required=True): + raise ConfigError("keyring is not installed") + + _, queue = transport + http.credentials.get = explode # restored by the fixture's monkeypatch teardown + assert client().authenticated is False + + +# --------------------------------------------------------------------------- +# the transport +# --------------------------------------------------------------------------- +def test_an_event_stream_answer_is_read_as_a_json_rpc_message(workspace, transport): + """The same request may be answered with JSON or with SSE, at the server's + discretion and without saying which in advance.""" + posts, queue = transport + body = tool_result({"data": [{"paper": {"paperId": "p1", "title": "Attention"}}]}) + stream = ( + "event: message\n" + 'data: {"jsonrpc":"2.0","method":"notifications/progress","params":{}}\n' + "\n" + f"event: message\ndata: {json.dumps(rpc(body))}\n\n" + ) + queue.append(FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"}))) + queue.append(FakeResponse(status_code=202, text="")) + queue.append(FakeResponse(text=stream, content_type="text/event-stream")) + + rows = client().snippet_search("attention") + assert [r["title"] for r in rows] == ["Attention"] + + +def test_a_stream_carrying_only_notifications_is_an_error(workspace, transport): + _, queue = transport + queue.append(FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"}))) + queue.append(FakeResponse(status_code=202, text="")) + queue.append( + FakeResponse(text='data: {"jsonrpc":"2.0","method":"x","params":{}}\n\n', + content_type="text/event-stream") + ) + with pytest.raises(UpstreamError, match="no result"): + client().snippet_search("attention") + + +def test_a_json_rpc_error_names_the_method_that_was_refused(workspace, transport): + _, queue = transport + queue.append( + FakeResponse(payload={"jsonrpc": "2.0", "id": 1, + "error": {"code": -32601, "message": "no such tool"}}) + ) + with pytest.raises(UpstreamError) as exc: + client().snippet_search("attention") + assert "initialize" in str(exc.value) + assert "no such tool" in str(exc.value) + + +def test_a_tool_that_reports_its_own_failure_is_not_an_empty_result(workspace, transport): + _, queue = transport + queue.append(FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"}))) + queue.append(FakeResponse(status_code=202, text="")) + queue.append( + FakeResponse(payload=rpc({"content": [{"type": "text", "text": "limit must be > 0"}], + "isError": True})) + ) + with pytest.raises(UpstreamError, match="limit must be"): + client().snippet_search("attention", limit=0) + + +def test_rate_limiting_points_at_the_key_that_can_actually_be_obtained(workspace, transport): + """The S2 advice this replaced -- "store an API key, it is free" -- has no + ending for a personal account. Asta's key comes from a form.""" + _, queue = transport + queue.append(FakeResponse(status_code=429, text="slow down")) + with pytest.raises(UpstreamError) as exc: + client().snippet_search("attention") + assert http.credentials.ASTA_KEY in (exc.value.fix or "") + + +def test_a_moved_endpoint_names_the_config_key_that_moves_with_it(workspace, transport): + _, queue = transport + queue.append(FakeResponse(status_code=404, text="gone")) + with pytest.raises(UpstreamError) as exc: + client().snippet_search("attention") + assert "asta_base" in (exc.value.fix or "") + + +# --------------------------------------------------------------------------- +# the shapes -- the part that is not verified against the live service +# --------------------------------------------------------------------------- +def call_with(workspace, transport, payload, *, structured=False): + _, queue = transport + queue.append(FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"}))) + queue.append(FakeResponse(status_code=202, text="")) + result = ( + {"structuredContent": payload, "content": []} if structured else tool_result(payload) + ) + queue.append(FakeResponse(payload=rpc(result))) + return client().snippet_search("attention") + + +NESTED = {"data": [{"snippet": {"text": "a 500-word excerpt", "snippetKind": "body"}, + "paper": {"corpusId": 991, "paperId": "p1", "title": "Attention", + "publicationDate": "2017-06-12", + "externalIds": {"ArXiv": "1706.03762"}}}]} +FLAT = {"results": [{"corpusId": 991, "paperId": "p1", "title": "Attention", "year": 2017, + "text": "a 500-word excerpt", "externalIds": {"ArXiv": "1706.03762"}}]} + + +@pytest.mark.parametrize("payload", [NESTED, FLAT], ids=["nested", "flat"]) +def test_both_plausible_result_shapes_produce_the_same_candidate(workspace, transport, payload): + row = call_with(workspace, transport, payload)[0] + # The SHA, not the corpus id: `IDENTITY_KEYS` leads with `paperId` because + # it is the field every endpoint returns. Both fixtures carry both. + assert row["id"] == "s2:p1" + assert row["title"] == "Attention" + assert row["year"] in (2017, "2017") + assert row["snippet"] == "a 500-word excerpt" + assert row["external"]["ArXiv"] == "1706.03762" + + +def test_the_typed_channel_is_preferred_over_the_text_one(workspace, transport): + rows = call_with(workspace, transport, NESTED, structured=True) + assert rows[0]["title"] == "Attention" + + +def test_a_bare_list_is_a_result_set_too(workspace, transport): + rows = call_with(workspace, transport, [{"paperId": "p1", "title": "Attention"}]) + assert rows[0]["title"] == "Attention" + + +def test_an_unrecognised_envelope_raises_rather_than_returning_nothing(workspace, transport): + """The failure this exists to prevent: `ok: true` with no results reads as + "the literature has nothing on this".""" + with pytest.raises(UpstreamError) as exc: + call_with(workspace, transport, {"unexpected": {"shape": 1}}) + assert "snippet_search" in str(exc.value) + assert "_rows" in (exc.value.fix or "") + + +def test_prose_where_a_result_set_was_expected_says_what_arrived(workspace, transport): + with pytest.raises(UpstreamError, match="not a result set"): + call_with(workspace, transport, "I could not find anything about that.") + + +def test_one_paper_gets_one_id_down_every_path_that_feeds_the_pool(workspace, transport, monkeypatch): + """`cmd_search` calls snippet search *and* paper search for every expanded + query, and fuses the results by id. `/snippet/search` returns `corpusId` and + `paperId` while `/paper/search` is only asked for the latter -- so reading + them in different orders gave the same paper two ids, and it ranked twice + and took a slot from something else. This is the ordinary path, not a corner + of it, which is why the order lives in one constant.""" + paper = {"paperId": "sha-1", "corpusId": 991, "title": "Attention", + "abstract": "an abstract", "externalIds": {"ArXiv": "1706.03762"}} + + asta_id = call_with(workspace, transport, {"data": [dict(paper)]})[0]["id"] + + class FakeHttpx: + @staticmethod + def get(url, params=None, headers=None, timeout=None): + if "snippet" in url: + return FakeResponse(payload={"data": [ + {"snippet": {"text": "an excerpt"}, "paper": dict(paper)}, + ]}) + # What `/paper/search` actually returns: the SHA, no corpus id. + flat = {k: v for k, v in paper.items() if k != "corpusId"} + return FakeResponse(payload={"data": [flat]}) + + monkeypatch.setattr(http, "_httpx", lambda: FakeHttpx) + s2 = http.SemanticScholar(config_mod.load(reload=True)) + snippet_id = s2.snippet_search("attention")[0]["id"] + paper_id = s2.paper_search("attention")[0]["id"] + + assert snippet_id == paper_id == asta_id == "s2:sha-1" + + +def test_the_id_and_the_citation_seed_name_the_same_paper(workspace, transport): + """`paper_id` is what `neighbours` expands from. If it were read by a + different rule than the id, a candidate and its citation expansion could + disagree about which paper they were.""" + row = call_with(workspace, transport, {"data": [ + {"paperId": "sha-1", "corpusId": 991, "title": "Attention"}, + ]})[0] + assert row["id"] == f"s2:{row['paper_id']}" + + +def test_a_paper_with_only_a_corpus_id_still_gets_one(workspace, transport): + row = call_with(workspace, transport, {"data": [ + {"corpusId": 991, "title": "Attention"}, + ]})[0] + assert row["id"] == "s2:991" + + +def test_hits_with_no_id_stay_distinct_instead_of_fusing_into_one(workspace, transport): + """`corpus.rrf` fuses by id, so a shared placeholder is not a cosmetic + problem: every id-less hit -- from either client, across every query in the + run -- collapsed into one phantom candidate and the rest vanished.""" + rows = call_with(workspace, transport, {"data": [ + {"title": "One paper", "text": "the first excerpt"}, + {"title": "Another paper", "text": "the second excerpt"}, + ]}) + assert len({row["id"] for row in rows}) == 2 + assert not any(row["id"].endswith("None") for row in rows) + + +def test_the_same_id_less_paper_from_both_clients_still_fuses(workspace, transport, monkeypatch): + """The other half: the fallback has to be *stable*, or the deduplication the + shared namespace exists for stops working for exactly these hits.""" + asta_row = call_with(workspace, transport, {"data": [ + {"title": "Attention Is All You Need", "text": "an excerpt"}, + ]})[0] + + class FakeHttpx: + @staticmethod + def get(url, params=None, headers=None, timeout=None): + return FakeResponse(payload={"data": [ + {"snippet": {"text": "an excerpt"}, + "paper": {"title": "Attention Is All You Need"}}, + ]}) + + monkeypatch.setattr(http, "_httpx", lambda: FakeHttpx) + s2_row = http.SemanticScholar(config_mod.load(reload=True)).snippet_search("attention")[0] + assert asta_row["id"] == s2_row["id"] + + +def test_a_hit_with_nothing_to_rank_is_dropped(workspace, transport): + """No id, no title, no text: nothing for the reranker to read and nothing to + cite. Dropping it costs no recall; keeping it under a placeholder did.""" + rows = call_with(workspace, transport, {"data": [ + {"year": 2017}, + {"paperId": "p1", "title": "A real one"}, + ]}) + assert [row["title"] for row in rows] == ["A real one"] + + +def test_a_failed_handshake_can_be_retried(workspace, transport): + """The guard used to be `if self._id:`, and `_call` increments that counter + *before* it sends -- so an `initialize` that failed left the counter + non-zero and every later request skipped straight to `tools/call` on a + connection that was never initialised. One 429 poisoned the client.""" + posts, queue = transport + client_ = client() + + queue.append(FakeResponse(status_code=429, text="slow down")) + with pytest.raises(UpstreamError): + client_.snippet_search("attention") + assert client_._ready is False # noqa: SLF001 + + queue.append(FakeResponse(payload=rpc({"protocolVersion": "2025-06-18"}))) + queue.append(FakeResponse(status_code=202, text="")) + queue.append(FakeResponse(payload=rpc(tool_result({"data": [ + {"paperId": "p1", "title": "Attention"}, + ]})))) + rows = client_.snippet_search("attention") + assert [row["title"] for row in rows] == ["Attention"] + assert methods(posts)[-3:] == ["initialize", "notifications/initialized", "tools/call"] + + +def test_the_backward_citation_direction_is_refused_rather_than_faked(workspace, transport): + """Asta publishes `get_citations` and no references counterpart. Answering + the backward direction with the forward one would double-count it into RRF + under a second name.""" + posts, queue = transport + assert client().neighbours("p1", direction="references") == [] + assert posts == [], "no request should have been made at all" + + +# --------------------------------------------------------------------------- +# one vocabulary +# --------------------------------------------------------------------------- +def test_both_tier_one_clients_answer_in_the_same_shape(workspace, transport, monkeypatch): + """`paper_search.py` does not know which tier a candidate came from and must + not have to: RRF fuses by id, and the reranker reads title and snippet. So + the two clients are asked for the same paper and their answers compared.""" + asta_row = call_with(workspace, transport, NESTED)[0] + + # The same hit as S2's REST API returns it. + class FakeHttpx: + @staticmethod + def get(url, params=None, headers=None, timeout=None): + return FakeResponse(payload=NESTED) + + monkeypatch.setattr(http, "_httpx", lambda: FakeHttpx) + s2_row = http.SemanticScholar(config_mod.load(reload=True)).snippet_search("attention")[0] + + shared = {"id", "paper_id", "title", "year", "snippet", "section", "source", "external"} + assert shared <= set(asta_row) + assert shared <= set(s2_row) + # The same corpus and the same corpus ids, so a paper found through both + # tiers fuses to one candidate instead of ranking twice under two names. + assert asta_row["id"] == s2_row["id"] == "s2:p1" + assert asta_row["title"] == s2_row["title"] + assert asta_row["snippet"] == s2_row["snippet"] + # `source` is the one field that must differ: a trace has to say who spoke. + assert asta_row["source"] != s2_row["source"] + + +def test_the_tier_one_selector_builds_the_clients_the_config_names(workspace): + from tools import paper_search + + cfg = config_mod.load(reload=True) + assert [n for n, _ in paper_search.tier1_clients(cfg, "asta")] == ["asta"] + assert [n for n, _ in paper_search.tier1_clients(cfg, "s2")] == ["s2"] + assert [n for n, _ in paper_search.tier1_clients(cfg, "both")] == ["asta", "s2"] + assert paper_search.tier1_clients(cfg, "none") == [] + # The default, when nothing overrides it. + assert [n for n, _ in paper_search.tier1_clients(cfg)] == ["asta"] + + +def test_an_unknown_tier_one_source_lists_the_real_ones(workspace): + from core.errors import UsageError + from tools import paper_search + + with pytest.raises(UsageError) as exc: + paper_search.tier1_clients(config_mod.load(reload=True), "scholar") + assert "asta" in (exc.value.fix or "") diff --git a/tests/test_funnel.py b/tests/test_funnel.py new file mode 100644 index 0000000..52858cf --- /dev/null +++ b/tests/test_funnel.py @@ -0,0 +1,313 @@ +"""The two ways the retrieval funnel came back empty, and how each says so. + +Both failures below were silent in the same expensive way: the CLI answered +`ok: true` with no results, or died with a message that named nothing. From the +agent's side either one reads as "there is no literature on this", which is the +one conclusion a research tool must never invite by accident. + + * **Stage 0 could not authenticate.** The Haiku stages are Agent SDK clients, + and the agent reaches them the way it reaches every capability here: by + running the CLI over Bash. That hop strips `CLAUDE_CODE_OAUTH_TOKEN` and + nothing else, so a token in the environment works from a terminal and is + gone under the agent. The CLI then answers "Not logged in" and exits + non-zero, which the SDK reports as `Claude Code returned an error result: + success` -- the CLI sends no `errors` array, so the SDK falls back to + printing the result *subtype*. + * **Stage 1 was rate limited on every call.** Semantic Scholar's anonymous + pool is shared and near-permanently exhausted. + +No SDK and no network: the SDK is faked, because what is under test is which +error comes out, not what Haiku says. +""" + +from __future__ import annotations + +import json +from typing import Any + +import pytest + +from core import credentials, haiku, paths +from core.errors import ConfigError, UpstreamError + +TOKEN = "sk-ant-oat-not-a-real-token" + + +# --------------------------------------------------------------------------- +# where stage 0's credentials come from +# --------------------------------------------------------------------------- +def test_the_environment_is_used_when_it_has_a_token(monkeypatch): + """Running a stage by hand in a terminal has to keep working with no setup: + there, the token *is* in the environment.""" + monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) + monkeypatch.setattr( + credentials, "get", lambda *a, **k: pytest.fail("the store was read before the environment") + ) + assert haiku._credentials_env() == {"CLAUDE_CODE_OAUTH_TOKEN": TOKEN} + + +def test_the_credential_store_covers_the_hop_that_strips_the_environment(monkeypatch): + """The case that was broken: under the agent there is no token to inherit.""" + monkeypatch.delenv("CLAUDE_CODE_OAUTH_TOKEN", raising=False) + monkeypatch.setattr( + credentials, "get", lambda name, **k: TOKEN if name == credentials.CLAUDE_TOKEN else None + ) + assert haiku._credentials_env() == {"CLAUDE_CODE_OAUTH_TOKEN": TOKEN} + + +def test_no_credentials_anywhere_names_both_halves_of_the_fix(monkeypatch): + monkeypatch.delenv("CLAUDE_CODE_OAUTH_TOKEN", raising=False) + monkeypatch.setattr(credentials, "get", lambda *a, **k: None) + with pytest.raises(ConfigError) as caught: + haiku._credentials_env() + assert "claude setup-token" in caught.value.fix + assert "credential set claude_oauth_token" in caught.value.fix + + +def test_the_token_is_handed_to_the_subprocess_not_left_to_inheritance(monkeypatch): + """`ClaudeAgentOptions.env` merges over the inherited environment, so this + adds one variable and takes nothing away.""" + sdk = _FakeSdk([_expansion_message()]) + monkeypatch.setattr(haiku, "_sdk", lambda: sdk) + monkeypatch.delenv("CLAUDE_CODE_OAUTH_TOKEN", raising=False) + monkeypatch.setattr(credentials, "get", lambda *a, **k: TOKEN) + + haiku.expand("a question", model="claude-haiku-4-5", log_name="probe") + assert sdk.options.env == {"CLAUDE_CODE_OAUTH_TOKEN": TOKEN} + + +# --------------------------------------------------------------------------- +# what an unauthenticated stage 0 says +# --------------------------------------------------------------------------- +def test_a_not_logged_in_turn_becomes_the_error_that_names_the_fix(monkeypatch): + """Rather than `Claude Code returned an error result: success`, which is the + result *subtype* and describes nothing.""" + sdk = _FakeSdk( + [_message(text="Not logged in", error="authentication_failed")], + raises=Exception("Claude Code returned an error result: success"), + ) + monkeypatch.setattr(haiku, "_sdk", lambda: sdk) + monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) + + with pytest.raises(ConfigError) as caught: + haiku.expand("a question", model="claude-haiku-4-5", log_name="probe") + assert "credential set claude_oauth_token" in caught.value.fix + assert "success" not in caught.value.message + + +def test_an_authentication_failure_that_does_not_raise_is_still_a_failure(monkeypatch): + """The CLI need not exit non-zero for the turn to have been refused; an + empty transcript would otherwise be reported as "no structured output".""" + sdk = _FakeSdk([_message(text="Not logged in", error="authentication_failed")]) + monkeypatch.setattr(haiku, "_sdk", lambda: sdk) + monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) + + with pytest.raises(ConfigError): + haiku.expand("a question", model="claude-haiku-4-5", log_name="probe") + + +def test_any_other_sdk_failure_is_left_alone(monkeypatch): + """Only the authentication case is translated. Swallowing the rest would + hide real upstream faults behind a credentials message.""" + sdk = _FakeSdk([], raises=RuntimeError("connection reset")) + monkeypatch.setattr(haiku, "_sdk", lambda: sdk) + monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) + + with pytest.raises(RuntimeError, match="connection reset"): + haiku.expand("a question", model="claude-haiku-4-5", log_name="probe") + + +def test_a_healthy_stage_still_returns_its_payload(monkeypatch): + sdk = _FakeSdk([_expansion_message()]) + monkeypatch.setattr(haiku, "_sdk", lambda: sdk) + monkeypatch.setenv("CLAUDE_CODE_OAUTH_TOKEN", TOKEN) + + out = haiku.expand("a question", model="claude-haiku-4-5", log_name="probe") + assert out["queries"] == ["adaptive optimizers", "second order methods"] + # The per-query log is the mitigation for these stages being subagents. + assert (paths.notes_dir() / "funnel" / "probe.md").exists() + + +# --------------------------------------------------------------------------- +# what a rate-limited stage 1 says +# --------------------------------------------------------------------------- +@pytest.mark.parametrize( + "source, client, expected_fix", + [ + # Each door's dead end points at the one that is not a dead end. + ("s2", "SemanticScholar", "--tier1 asta"), + ("asta", "Asta", "credential set asta_api_key"), + ], +) +def test_a_search_whose_every_call_failed_is_a_failure_not_an_empty_result( + workspace, capsys, monkeypatch, source, client, expected_fix +): + """`ok: true` with no results reads as "the literature has nothing", which + is not a conclusion anyone should draw from a rate limit.""" + from core import http + from tools import paper_search + + monkeypatch.setattr(http, client, lambda cfg: _RateLimited()) + + code = paper_search.cli.run( + ["search", "efficient optimizers", "--tier1", source, + "--no-expand", "--no-triage", "--no-local", "--json"] + ) + payload = json.loads(capsys.readouterr().out) + assert code != 0 + assert payload["ok"] is False + assert "rate-limited" in payload["error"]["message"] + assert expected_fix in payload["error"]["fix"] + # The old advice, which sent anyone following it to a second empty run. + assert "--no-expand" not in json.dumps(payload) + + +def test_the_s2_dead_end_does_not_recommend_a_key_that_cannot_be_obtained( + workspace, capsys, monkeypatch +): + """Ai2 stopped issuing Semantic Scholar keys to free-domain addresses, so + "store an S2 API key -- it is free" has no ending for a personal account. + Advice that cannot be followed is worse than none: it is followed first.""" + from core import http + from tools import paper_search + + monkeypatch.setattr(http, "SemanticScholar", lambda cfg: _RateLimited()) + paper_search.cli.run( + ["search", "efficient optimizers", "--tier1", "s2", + "--no-expand", "--no-triage", "--no-local", "--json"] + ) + payload = json.loads(capsys.readouterr().out) + assert "credential set s2_api_key" not in payload["error"]["fix"] + assert "institutional" in payload["error"]["fix"] + + +def test_an_empty_run_still_writes_the_trace_the_funnel_view_reads( + workspace, capsys, monkeypatch +): + """"why is the obviously relevant paper not in here" is the question the + funnel view exists for, and it was exactly the runs that answered it that + returned before writing a trace.""" + from core import http + from tools import paper_search + + # The default source, so this covers the path a real run takes. + monkeypatch.setattr(http, "Asta", lambda cfg: _RateLimited()) + paper_search.cli.run( + ["search", "efficient optimizers", "--no-expand", "--no-triage", "--no-local", "--json"] + ) + capsys.readouterr() + + traces = list((paths.notes_dir() / "funnel").glob("*.json")) + assert len(traces) == 1 + written = json.loads(traces[0].read_text(encoding="utf-8")) + assert written["stages"]["1_retrieve"] == {"rankings": 0, "candidates": 0} + # Which client was asked, so "no results" can be read against who was down. + assert written["stages"]["1_sources"] == ["asta"] + assert any("rate-limited" in w for w in written["warnings"]) + + +def test_a_genuinely_empty_index_is_an_empty_result_not_a_failure(workspace, capsys): + """The other side of it: nothing found and nothing broken is `ok: true`.""" + from tools import paper_search + + assert paper_search.cli.run( + ["search", "efficient optimizers", "--local-only", "--no-expand", "--no-triage", "--json"] + ) == 0 + payload = json.loads(capsys.readouterr().out) + assert payload["ok"] is True + assert payload["data"]["results"] == [] + assert "widen --candidates" in payload["data"]["note"] + + +# --------------------------------------------------------------------------- +# fakes +# --------------------------------------------------------------------------- +class _RateLimited: + """Either tier-1 client, rate limited. They share a vocabulary, so one fake + stands in for both -- which is the property `tests/test_asta.py` pins.""" + + def _fail(self, *_a: Any, **_k: Any) -> list[dict[str, Any]]: + raise UpstreamError( + "Semantic Scholar rate-limited the request", + fix="wait a few seconds and retry, or store an S2 API key", + ) + + snippet_search = _fail + paper_search = _fail + neighbours = _fail + + +class _Block: + def __init__(self, text: str) -> None: + self.text = text + + +class _Message: + def __init__(self, text: str = "", error: str | None = None) -> None: + self.content = [_Block(text)] if text else [] + self.error = error + self.usage = None + + +def _message(*, text: str = "", error: str | None = None) -> _Message: + return _Message(text, error) + + +def _expansion_message() -> _Message: + """A turn that calls the tool, which is what the real one does.""" + message = _Message("calling submit_expansion") + message.call = { + "queries": ["adaptive optimizers", "second order methods"], + "hyde": " ".join(["word"] * 40), + } + return message + + +class _FakeSdk: + """Just enough of `claude_agent_sdk` for `haiku._call`. + + A message carrying `.call` invokes the registered tool with that payload, + which is how the real SDK delivers validated tool input. + """ + + def __init__(self, messages: list[_Message], raises: Exception | None = None) -> None: + self.messages = messages + self.raises = raises + self.options: Any = None + self._handler: Any = None + + def tool(self, _name: str, _description: str, _schema: dict[str, Any]) -> Any: + def decorate(fn: Any) -> Any: + self._handler = fn + return fn + + return decorate + + def create_sdk_mcp_server(self, name: str, tools: list[Any]) -> dict[str, Any]: + return {"name": name, "tools": tools} + + def ClaudeAgentOptions(self, **kwargs: Any) -> Any: # noqa: N802 - the SDK's name + self.options = type("Options", (), kwargs) + return self.options + + def query(self, *, prompt: str, options: Any) -> Any: # noqa: ARG002 + async def stream() -> Any: + for message in self.messages: + payload = getattr(message, "call", None) + if payload is not None and self._handler is not None: + await self._handler(payload) + yield message + if self.raises is not None: + raise self.raises + + return stream() + + +def test_the_fix_the_error_prints_is_a_command_that_exists(): + """The error above tells the reader to run `credential set + claude_oauth_token`. That is only useful if the CLI accepts the name.""" + from tools.jobs import CREDENTIAL_NAMES + + assert credentials.CLAUDE_TOKEN in CREDENTIAL_NAMES + assert credentials.CLAUDE_TOKEN in credentials.status() + assert credentials.CLAUDE_TOKEN in haiku.AUTH_FIX diff --git a/tests/test_review_fixes.py b/tests/test_review_fixes.py index 3dbb57f..43632a4 100644 --- a/tests/test_review_fixes.py +++ b/tests/test_review_fixes.py @@ -348,6 +348,27 @@ def test_scrub_removes_the_env_credential_fallbacks(workspace, monkeypatch): assert "GRAD_HF_TOKEN" not in os.environ +def test_every_credential_the_store_knows_is_scrubbed_from_the_environment(monkeypatch): + """The list used to be written out by hand, and it drifted: `credentials.get` + reads `GRAD_` for every credential, but two were added to that lookup + and not to the scrub -- so the agent inherited them. `GRAD_CLAUDE_OAUTH_TOKEN` + is the one that matters most, because the scrub is what bounds who can read + the token `core/haiku.py` hands to its subprocesses on purpose.""" + import os + + from core import credentials + + for name in credentials.ALL: + monkeypatch.setenv(f"GRAD_{name.upper()}", "secret") + + removed = set(credentials.scrub_environment()) + + for name in credentials.ALL: + var = f"GRAD_{name.upper()}" + assert var in removed, var + assert var not in os.environ, var + + # --------------------------------------------------------------------------- # the deny probe does not read tea leaves # --------------------------------------------------------------------------- diff --git a/tests/test_streaming.py b/tests/test_streaming.py new file mode 100644 index 0000000..ce51260 --- /dev/null +++ b/tests/test_streaming.py @@ -0,0 +1,453 @@ +"""Assembling a turn from partial and finished messages (`agent.TextStream`, +`agent.TurnStream`). + +`include_partial_messages` makes the SDK emit the same text twice over: once as +a run of `text_delta` events, and again as the finished `AssistantMessage`. The +whole job of `TextStream` is to show it once, and the failure it exists to +prevent -- every answer appearing twice -- is invisible until someone actually +runs the agent. So it is tested here, against fakes shaped like the SDK's own +messages, with no SDK and no network. + +`TurnStream` is the other half: the tool calls, which `TextStream` filters out +by construction. Its own rule is that **the order is the information** -- which +command ran before which claim -- so most of what is tested below is where a +block lands, not just that it exists. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +import pytest + +import agent + + +# --- fakes shaped like the SDK's messages ----------------------------------- +@dataclass +class FakeBlock: + text: str + + +@dataclass +class ThinkingBlock: + """No `.text`, exactly like the real one -- which is why thinking never + reaches the transcript.""" + + thinking: str + + +@dataclass +class FakeMessage: + """An `AssistantMessage`: a finished list of content blocks.""" + + content: list[Any] = field(default_factory=list) + + +@dataclass +class FakeStreamEvent: + """A `StreamEvent`: the raw Anthropic streaming event, and no `.content`.""" + + event: dict[str, Any] + + +def delta(text: str) -> FakeStreamEvent: + return FakeStreamEvent({"type": "content_block_delta", "delta": {"type": "text_delta", "text": text}}) + + +def message(*texts: str) -> FakeMessage: + return FakeMessage([FakeBlock(t) for t in texts]) + + +def drain(stream: agent.TextStream, messages: list[Any]) -> str: + """Feed a whole turn, returning what a CLI would have printed.""" + return "".join(stream.feed(m) for m in messages) + + +# --------------------------------------------------------------------------- +# the rule the class exists for +# --------------------------------------------------------------------------- +def test_a_streamed_answer_is_not_also_appended_when_it_finishes(): + """The bug this is all for: deltas *and* the finished message both carry the + text, so the obvious loop shows every answer twice.""" + stream = agent.TextStream() + printed = drain(stream, [delta("Hello"), delta(" world"), message("Hello world")]) + assert stream.text == "Hello world" + assert printed == "Hello world" # printed once, as it arrived + + +def test_the_tokens_are_visible_before_the_message_finishes(): + """Streaming is the point: the text has to be readable mid-turn, not only + once the message lands.""" + stream = agent.TextStream() + stream.feed(delta("The loss ")) + assert stream.text == "The loss " + stream.feed(delta("is 3.1")) + assert stream.text == "The loss is 3.1" + + +def test_a_message_that_never_streamed_still_arrives(): + """Not every message comes with deltas -- a resumed turn, a cached reply, an + SDK that stopped emitting them. Losing the text would be far worse than + showing it late.""" + stream = agent.TextStream() + printed = drain(stream, [message("no deltas for this one")]) + assert stream.text == "no deltas for this one" + assert printed == "no deltas for this one" + + +def test_a_partially_streamed_message_is_completed_not_duplicated(): + stream = agent.TextStream() + printed = drain(stream, [delta("half "), message("half the answer")]) + assert stream.text == "half the answer" + assert printed == "half the answer" # "half " streamed, "the answer" caught up + + +def test_the_finished_message_wins_when_the_two_disagree(): + """A dropped or reordered event must not leave a mangled transcript. The + reconstruction is discarded; the message is authoritative.""" + stream = agent.TextStream() + stream.feed(delta("garbled")) + stream.feed(message("the real answer")) + assert stream.text == "the real answer" + + +# --------------------------------------------------------------------------- +# a whole turn +# --------------------------------------------------------------------------- +def test_several_messages_in_one_turn_each_stream_and_settle(): + """A turn is prose, then a tool call, then more prose. Each finished message + resets the run of deltas -- reset once at the end and the second message + would overwrite the first.""" + stream = agent.TextStream() + printed = drain(stream, [ + delta("Checking"), delta(" the ledger."), message("Checking the ledger."), + FakeMessage([]), # the tool call itself: no text + delta(" Found"), delta(" it."), message(" Found it."), + ]) + assert stream.text == "Checking the ledger. Found it." + assert printed == "Checking the ledger. Found it." + + +def test_a_message_carrying_no_text_does_not_disturb_a_stream_in_flight(): + """Tool results and system messages arrive mid-turn. Treating one as a + finished assistant message would drop the deltas it interrupted.""" + stream = agent.TextStream() + stream.feed(delta("half an answer")) + stream.feed(FakeMessage([])) # a tool result: content, but no text + stream.feed(FakeMessage()) # empty content + assert stream.text == "half an answer" + stream.feed(message("half an answer, finished")) + assert stream.text == "half an answer, finished" + + +# --------------------------------------------------------------------------- +# what counts as visible text +# --------------------------------------------------------------------------- +def test_thinking_deltas_never_reach_the_transcript(): + """`_text_of` skips a finished `ThinkingBlock` because it has no `.text`. If + the deltas did not skip thinking too, the stream would show reasoning that + vanished the moment the message settled.""" + stream = agent.TextStream() + stream.feed(FakeStreamEvent( + {"type": "content_block_delta", "delta": {"type": "thinking_delta", "thinking": "hmm"}} + )) + assert stream.text == "" + stream.feed(FakeMessage([ThinkingBlock("hmm"), FakeBlock("the answer")])) + assert stream.text == "the answer" + + +def test_tool_input_deltas_are_not_answer_text(): + stream = agent.TextStream() + stream.feed(FakeStreamEvent( + {"type": "content_block_delta", "delta": {"type": "input_json_delta", "partial_json": '{"a":'}} + )) + assert stream.text == "" + + +def test_non_delta_stream_events_are_ignored(): + stream = agent.TextStream() + for event in ( + {"type": "message_start", "message": {}}, + {"type": "content_block_start", "content_block": {"type": "text", "text": ""}}, + {"type": "content_block_stop", "index": 0}, + {"type": "message_stop"}, + ): + assert stream.feed(FakeStreamEvent(event)) == "" + assert stream.text == "" + + +def test_a_malformed_stream_event_is_not_a_crash(): + """`event` is a raw dict off the wire, so every field is untrusted. A turn + must not die because one event was shaped unexpectedly.""" + stream = agent.TextStream() + for event in (None, "nope", {}, {"type": "content_block_delta"}, + {"type": "content_block_delta", "delta": None}, + {"type": "content_block_delta", "delta": {"type": "text_delta"}}, + {"type": "content_block_delta", "delta": {"type": "text_delta", "text": 7}}): + assert stream.feed(FakeStreamEvent(event)) == "" + assert stream.text == "" + + +# --------------------------------------------------------------------------- +# the tool calls (`TurnStream`) +# --------------------------------------------------------------------------- +@dataclass +class ToolUse: + """A `ToolUseBlock`.""" + + id: str + name: str + input: dict[str, Any] + + +@dataclass +class ToolResult: + """A `ToolResultBlock`. These arrive on a `UserMessage`, not the assistant's.""" + + tool_use_id: str + content: Any = None + is_error: bool | None = None + + +def kinds(stream: agent.TurnStream) -> list[str]: + return [b["kind"] for b in stream.blocks] + + +def test_a_call_becomes_a_block_of_its_own(): + """The gap this closes: every capability in this project is reached by a + Bash into `tools/`, and none of it used to reach the transcript.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([ + FakeBlock("Checking the ledger."), + ToolUse("tu_1", "Bash", {"command": "python -m tools.ledger show"}), + ])) + assert kinds(stream) == ["text", "tool"] + call = stream.blocks[1] + assert call["name"] == "Bash" + assert call["title"] == "python -m tools.ledger show" + assert call["status"] == "running" + + +def test_prose_after_a_call_lands_below_it(): + """The order is the whole point: a claim made *after* a command ran reads + differently from one made before it.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([ + FakeBlock("Checking."), + ToolUse("tu_1", "Bash", {"command": "ls"}), + ])) + stream.feed(FakeMessage([ToolResult("tu_1", "one\ntwo")])) + stream.feed(delta(" Two entries.")) + stream.feed(FakeMessage([FakeBlock(" Two entries.")])) + assert kinds(stream) == ["text", "tool", "text"] + assert [b["text"] for b in stream.blocks if b["kind"] == "text"] == ["Checking.", " Two entries."] + + +def test_a_result_attaches_to_the_call_it_answers(): + """Two calls in flight and the results back in the other order is the case + that matching by position would get exactly backwards.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([ + ToolUse("tu_1", "Read", {"file_path": "core/budget.py"}), + ToolUse("tu_2", "Grep", {"pattern": "def status"}), + ])) + stream.feed(FakeMessage([ToolResult("tu_2", "core/budget.py:41"), ToolResult("tu_1", "…source…")])) + first, second = stream.blocks + assert (first["title"], first["result"]) == ("core/budget.py", "…source…") + assert (second["title"], second["result"]) == ("def status", "core/budget.py:41") + assert [b["status"] for b in stream.blocks] == ["ok", "ok"] + + +def test_a_refused_call_is_drawn_as_failed_not_as_output(): + """A `PreToolUse` denial comes back as an error result. Showing it as + ordinary output would make a blocked spend look like a successful one.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([ToolUse("tu_1", "Bash", {"command": "ssh probe-host echo hello"})])) + stream.feed(FakeMessage([ToolResult("tu_1", "denied by the gate", is_error=True)])) + assert stream.blocks[0]["status"] == "error" + assert stream.blocks[0]["result"] == "denied by the gate" + + +def test_a_result_for_a_call_this_stream_never_saw_is_dropped(): + """A turn resumed from cache can carry a result whose call was never + streamed. Inventing a card for it would claim an order we do not know.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([ToolResult("tu_missing", "output")])) + assert stream.blocks == [] + + +def test_result_content_that_arrives_as_blocks_is_flattened(): + stream = agent.TurnStream() + stream.feed(FakeMessage([ToolUse("tu_1", "Bash", {"command": "ls"})])) + stream.feed(FakeMessage([ToolResult("tu_1", [{"type": "text", "text": "one"}, {"type": "text", "text": "two"}])])) + assert stream.blocks[0]["result"] == "one\ntwo" + + +def test_a_long_result_is_clipped_and_says_so(): + """A `Read` of a long file is held for the session, written to the + transcript file and drawn again on restore. The card is a record that the + call happened, not a second copy of its output.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([ToolUse("tu_1", "Read", {"file_path": "notes/long.md"})])) + stream.feed(FakeMessage([ToolResult("tu_1", "\n".join(f"line {i}" for i in range(500)))])) + result = stream.blocks[0]["result"] + assert len(result.splitlines()) <= agent.RESULT_LINES + 2 + assert "more lines" in result + + +def test_the_turns_text_is_the_prose_alone(): + """`text` is what a transcript with no cards should say. Tool output is not + something the agent said, and folding it in would put a command's stdout in + the agent's own voice.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([FakeBlock("Running it."), ToolUse("tu_1", "Bash", {"command": "ls"})])) + stream.feed(FakeMessage([ToolResult("tu_1", "budget.py")])) + assert stream.text == "Running it." + + +def test_a_card_is_titled_by_what_the_call_was_on(): + """Not by the whole input: an `Edit` carries its entire replacement text, + and a head that is a wall of source is worse than one that is empty.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([ToolUse("tu_1", "Edit", { + "file_path": "core/budget.py", + "old_string": "x" * 5000, + "new_string": "y" * 5000, + })])) + call = stream.blocks[0] + assert call["title"] == "core/budget.py" + assert dict(call["rows"]).keys() == {"old_string", "new_string"} + assert all(len(value) < 400 for value in dict(call["rows"]).values()) + + +def test_a_multiline_command_is_one_line_in_the_head_and_whole_in_the_body(): + stream = agent.TurnStream() + stream.feed(FakeMessage([ToolUse("tu_1", "Bash", {"command": "cd tools \\\n&& python -m evolve"})])) + call = stream.blocks[0] + assert "\n" not in call["title"] + assert "python -m evolve" in call["text"] + + +def test_the_call_in_flight_is_the_one_still_running(): + stream = agent.TurnStream() + stream.feed(FakeMessage([ToolUse("tu_1", "Bash", {"command": "first"})])) + stream.feed(FakeMessage([ToolResult("tu_1", "done")])) + stream.feed(FakeMessage([ToolUse("tu_2", "Bash", {"command": "second"})])) + assert stream.active()["title"] == "second" + stream.feed(FakeMessage([ToolResult("tu_2", "done")])) + assert stream.active() is None + + +def test_a_cli_gets_a_line_naming_each_call_and_its_outcome(): + """`feed` returns what a terminal should print next -- the same stream the + UI draws as cards, in one line each.""" + stream = agent.TurnStream() + printed = drain(stream, [ + delta("Checking."), FakeMessage([FakeBlock("Checking."), ToolUse("tu_1", "Bash", {"command": "ls"})]), + FakeMessage([ToolResult("tu_1", "one\ntwo")]), + ]) + assert "Checking." in printed + assert "[tool] Bash ls" in printed + assert "[tool] Bash ok (2 lines)" in printed + + +def test_a_failed_call_says_why_on_the_command_line(): + stream = agent.TurnStream() + stream.feed(FakeMessage([ToolUse("tu_1", "Bash", {"command": "ssh probe-host"})])) + printed = stream.feed(FakeMessage([ToolResult("tu_1", "denied: ssh is gated", is_error=True)])) + assert "[tool] Bash failed: denied: ssh is gated" in printed + + +def test_a_printed_line_is_ascii_so_a_windows_console_survives_it(): + """`print` to a cp1252 console raises on a stray glyph, and that would take + the whole turn down at the moment it was most worth watching.""" + stream = agent.TurnStream() + printed = stream.feed(FakeMessage([ToolUse("tu_1", "Bash", {"command": "ls"})])) + printed += stream.feed(FakeMessage([ToolResult("tu_1", "x" * 6000, is_error=True)])) + printed.encode("ascii") # raises if anything above is not + + +def test_the_no_duplication_rule_still_holds_around_a_call(): + """`TurnStream` delegates text to `TextStream`, so the bug that class exists + to prevent must not come back through the wrapper.""" + stream = agent.TurnStream() + printed = drain(stream, [ + delta("Checking"), delta(" the ledger."), + FakeMessage([FakeBlock("Checking the ledger."), ToolUse("tu_1", "Bash", {"command": "ls"})]), + FakeMessage([ToolResult("tu_1", "one")]), + delta(" Done."), FakeMessage([FakeBlock(" Done.")]), + ]) + assert stream.text == "Checking the ledger. Done." + assert printed.count("Checking the ledger.") == 1 + + +def test_a_note_is_the_sessions_own_voice_at_the_end_of_the_turn(): + """How a turn that died says so: the prose that streamed before it is kept, + and the card of the call it died on is left mid-flight.""" + stream = agent.TurnStream() + stream.feed(FakeMessage([FakeBlock("Starting."), ToolUse("tu_1", "Bash", {"command": "ls"})])) + stream.note("\n\n**the session failed:** `ConnectionError`") + assert kinds(stream) == ["text", "tool", "text"] + assert stream.blocks[1]["status"] == "running" + assert stream.text.endswith("`ConnectionError`") + + +# --------------------------------------------------------------------------- +# what survives to the transcript file +# --------------------------------------------------------------------------- +def test_the_calls_survive_a_restart_and_a_transcript_without_them_still_opens(): + """`blocks` is optional on the way back in: transcripts written before tool + calls were captured have none, and those still have to open.""" + app = pytest.importorskip("ui.app", reason="the ui extra is not installed") + + session = app.Session("test") + session.settled = [ + {"role": "user", "text": "check the ledger"}, + {"role": "assistant", "text": "Checking.", "blocks": [ + {"kind": "text", "text": "Checking."}, + {"kind": "tool", "name": "Bash", "title": "ls", "status": "ok", "result": "one"}, + ]}, + ] + session._persist() # noqa: SLF001 - the round trip is the test + + reopened = app.Session("test") + reopened.restore() + assert reopened.settled[0] == {"role": "user", "text": "check the ledger"} + assert reopened.settled[1]["blocks"][1]["name"] == "Bash" + + +def test_a_transcript_line_whose_blocks_are_junk_still_opens_the_window(): + """The file outlives the version that wrote it, so `blocks` is untrusted + input. A block with no `kind` would reach the renderer's dispatch and take + the page down at build time.""" + app = pytest.importorskip("ui.app", reason="the ui extra is not installed") + + session = app.Session("junk") + session.path().parent.mkdir(parents=True, exist_ok=True) + session.path().write_text( + "\n".join([ + '{"role": "assistant", "text": "fine", "blocks": "not a list"}', + '{"role": "assistant", "text": "fine", "blocks": [{"no": "kind"}, 7, {"kind": "tool"}]}', + ]), + encoding="utf-8", + ) + session.restore() + assert "blocks" not in session.settled[0] + assert session.settled[1]["blocks"] == [{"kind": "tool"}] + + +def test_the_options_actually_ask_for_partial_messages(monkeypatch): + """The whole feature is one flag on the options, and it defaults to off in + the SDK. Every other test here would still pass with it dropped -- they feed + deltas by hand -- so this is the one that notices.""" + sdk = pytest.importorskip("claude_agent_sdk", reason="the SDK is not installed") + from core import config as config_mod + + # The prompt file lives in the real workspace, not the fixture's temp one; + # this test is about the flag, not about reading it. + monkeypatch.setattr(agent, "system_prompt", lambda: "prompt") + options = agent.build_options(config_mod.load()) + assert isinstance(options, sdk.ClaudeAgentOptions) + assert options.include_partial_messages is True diff --git a/tests/test_ui_layout.py b/tests/test_ui_layout.py index e2eebf9..3f9e071 100644 --- a/tests/test_ui_layout.py +++ b/tests/test_ui_layout.py @@ -224,6 +224,110 @@ def test_move_of_an_unopened_window_is_a_no_op(): assert layout.windows == ["chat"] +def test_a_drag_cannot_create_a_fourth_column(): + """`open` has always stopped at the cap; dropping a title bar past the right + edge used to walk straight past it, which is how you ended up with 240px + panes the opener strip could never have produced.""" + layout = L.Layout() + for window in ("chat", "notebook", "ledger", "quota"): + layout.open(window) + layout.move("quota", 9) + layout.move("ledger", 9) + assert len(layout.columns) <= L.MAX_COLUMNS + assert sorted(layout.windows) == ["chat", "ledger", "notebook", "quota"] + + +def test_a_window_alone_in_its_column_can_still_be_dragged_out_at_the_cap(): + """It takes its column with it, so the count never actually rises. Counting + the cap before the pull would refuse a move that is plainly legal.""" + layout = L.Layout().open("chat").open("ledger").open("quota") + assert len(layout.columns) == L.MAX_COLUMNS + layout.move("quota", 0, new_column=True) + assert [c.windows for c in layout.columns] == [["quota"], ["chat"], ["ledger"]] + + +def test_a_drop_names_a_position_in_the_column_not_just_the_column(): + layout = L.Layout().open("chat").open("ledger").open("quota").apply_preset("stack") + assert layout.columns[0].windows == ["chat", "ledger", "quota"] + layout.move("quota", 0, 0) + assert layout.columns[0].windows == ["quota", "chat", "ledger"] + + +def test_moving_down_inside_one_column_lands_where_the_indicator_was(): + """`slot_index` is a boundary in the column as the browser sees it -- with + the dragged window still in it. Insert without correcting for the pull and + a pane dragged one place down moves two.""" + layout = L.Layout().open("chat").open("ledger").open("quota").apply_preset("stack") + layout.move("chat", 0, 2) # the boundary between ledger and quota + assert layout.columns[0].windows == ["ledger", "chat", "quota"] + + +def test_a_slot_index_past_the_end_appends(): + layout = L.Layout().open("chat").open("ledger").apply_preset("stack") + layout.move("chat", 0, 99) + assert layout.columns[0].windows == ["ledger", "chat"] + + +def test_a_new_column_can_be_split_in_before_an_existing_one(): + layout = L.Layout().open("chat").open("ledger") + layout.move("ledger", 0, new_column=True) + assert [c.windows for c in layout.columns] == [["ledger"], ["chat"]] + + +# --------------------------------------------------------------------------- +# swapping +# --------------------------------------------------------------------------- +def test_swap_exchanges_two_windows(): + layout = L.Layout().open("chat").open("ledger").open("quota") + layout.swap("chat", "quota") + assert [c.windows for c in layout.columns] == [["quota"], ["ledger"], ["chat"]] + + +def test_swap_leaves_the_panes_their_own_sizes(): + """The windows trade places; the geometry does not travel with them. Dropping + the ledger onto the chat puts it where the chat was, at the chat's size.""" + layout = L.Layout().open("chat").open("ledger") + layout.resize_columns([0.7, 0.3]) + before = fractions(layout) + layout.swap("chat", "ledger") + assert layout.columns[0].windows == ["ledger"] + for a, b in zip(before, fractions(layout)): + assert math.isclose(a, b, abs_tol=1e-9) + + +def test_swap_works_across_a_stack(): + layout = L.Layout() + for window in ("chat", "notebook", "ledger", "quota"): + layout.open(window) + layout.swap("chat", "quota") + assert [c.windows for c in layout.columns] == [["quota"], ["notebook"], ["ledger", "chat"]] + + +def test_swapping_a_window_with_itself_changes_nothing(): + layout = L.Layout().open("chat").open("ledger") + layout.swap("chat", "chat") + assert [c.windows for c in layout.columns] == [["chat"], ["ledger"]] + + +def test_swapping_with_a_window_that_is_not_open_is_a_no_op(): + """The ids come from the browser. A closed one must not blank a live pane.""" + layout = L.Layout().open("chat").open("ledger") + layout.swap("chat", "evolve") + assert layout.windows == ["chat", "ledger"] + layout.swap("evolve", "chat") + assert layout.windows == ["chat", "ledger"] + + +def test_swap_never_duplicates_or_drops_a_window(): + layout = L.Layout() + for window in ("chat", "notebook", "ledger", "quota", "wiki"): + layout.open(window) + before = sorted(layout.windows) + for a, b in (("chat", "wiki"), ("ledger", "chat"), ("quota", "notebook")): + layout.swap(a, b) + assert sorted(layout.windows) == before + + # --------------------------------------------------------------------------- # presets # --------------------------------------------------------------------------- diff --git a/tests/test_ui_models.py b/tests/test_ui_models.py index 11e92f6..93337ca 100644 --- a/tests/test_ui_models.py +++ b/tests/test_ui_models.py @@ -11,6 +11,7 @@ import datetime as dt import json +import re import pytest @@ -24,13 +25,27 @@ def test_the_model_layer_does_not_drag_in_nicegui(): """`ui/models.py` and `ui/layout.py` are the tested half of the UI. If either grows a NiceGUI import the tests stop being runnable without the extra, and - the layering claim in `ui/__init__.py` stops being true.""" + the layering claim in `ui/__init__.py` stops being true. + + The rule is about *importing* NiceGUI, not about naming it. `ui/tokens.py` + has to write `.nicegui-content` into the stylesheet -- that wrapper is what + the shell is nested inside, and its padding has to be zeroed out from CSS -- + and a selector for someone else's class name costs nothing at import time. + So this matches import statements rather than the bare substring. + """ + imports = re.compile( + r"""^\s*(?:from|import)\s+nicegui\b""" # from nicegui import ... / import nicegui + r"""|__import__\(\s*['"]nicegui""" # the dynamic spellings + r"""|import_module\(\s*['"]nicegui""", + re.MULTILINE, + ) for module in ("ui.models", "ui.layout", "ui.registry", "ui.tokens", "ui.fonts"): source = __import__(module, fromlist=["__file__"]).__file__ assert source with open(source, encoding="utf-8") as fh: text = fh.read() - assert "nicegui" not in text, f"{module} must not reference nicegui" + found = imports.search(text) + assert not found, f"{module} must not import nicegui: {found.group(0).strip()!r}" # --------------------------------------------------------------------------- diff --git a/tests/test_ui_registry.py b/tests/test_ui_registry.py index e06543e..06c47b3 100644 --- a/tests/test_ui_registry.py +++ b/tests/test_ui_registry.py @@ -1,8 +1,8 @@ """The window registry, and the contract every window module signs. -The registry is the one list the opener strip, the layout presets, the command -palette, the persisted layout's validation and the status bar's count are all -derived from. If it is ever more than one list, two of those will drift. +The registry is the one list the `⋯` menu, the persisted layout's validation and +the status bar's count are all derived from. If it is ever more than one list, +two of those will drift. These tests import the window modules, which imports `ui/kit.py` -- but not NiceGUI, because `kit` imports it inside its functions. That is deliberate: it @@ -24,10 +24,13 @@ def test_ids_are_unique(): assert len(registry.ids()) == len(set(registry.ids())) -def test_the_eleven_windows_the_handoff_lists_are_all_here(): +def test_the_windows_the_handoff_lists_are_all_here(): + """The handoff's eleven, and `tasks` -- which is not one of them because the + handoff had nowhere to put a local command that runs for twenty minutes. + Everything long was awaited under a wall clock until it was.""" assert set(registry.ids()) == { "chat", "notebook", "wiki", "papers", "evolve", "editor", - "ledger", "preflight", "quota", "funnel", "queue", + "ledger", "preflight", "quota", "funnel", "queue", "tasks", } diff --git a/tests/test_ui_sessions.py b/tests/test_ui_sessions.py new file mode 100644 index 0000000..5715d51 --- /dev/null +++ b/tests/test_ui_sessions.py @@ -0,0 +1,360 @@ +"""Named chat sessions: the file format, the listing, and the two ids. + +No NiceGUI and no SDK. What is under test is the part that has rules -- which +file an id names, what a session is called when nobody named it, what survives +an upgrade, and the difference between resuming a conversation and redisplaying +a transcript. + +That last one is the property worth stating plainly, because getting it wrong is +invisible: a session reopened without its *SDK* id shows every turn above a +composer whose next turn the agent has no memory of. The transcript and the +conversation are two different things and only one of them is in the file. +""" + +from __future__ import annotations + +import json + +import pytest + +from ui import sessions + + +def write_raw(workspace, session_id: str, lines: list[dict]) -> None: + path = sessions.path_for(session_id) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("\n".join(json.dumps(line) for line in lines), encoding="utf-8") + + +def user(text: str) -> dict: + return {"role": "user", "text": text} + + +def assistant(text: str) -> dict: + return {"role": "assistant", "text": text} + + +# --------------------------------------------------------------------------- +# ids and paths +# --------------------------------------------------------------------------- +def test_a_new_id_is_sortable_and_unique(workspace): + first, second = sessions.new_id(), sessions.new_id() + assert first != second + assert sessions.is_id(first) + # Timestamp first, so a plain glob comes back in a sensible order. + assert first[:8].isdigit() + + +@pytest.mark.parametrize( + "bad", + ["../../etc/passwd", "with/slash", "with\\slash", "", ".hidden", "a" * 200, None, 12], +) +def test_an_id_that_is_not_one_never_reaches_a_filename(workspace, bad): + """The id comes off disk or out of a click handler, so it is validated + rather than trusted -- the same reason `state.layout_path` sanitises.""" + assert sessions.is_id(bad) is False + with pytest.raises(ValueError): + sessions.path_for(bad) + + +def test_the_id_round_trips_through_the_filename(workspace): + session_id = sessions.new_id() + assert sessions.id_of(sessions.path_for(session_id)) == session_id + + +# --------------------------------------------------------------------------- +# what a session is called +# --------------------------------------------------------------------------- +def test_a_session_is_named_after_the_first_thing_asked_in_it(workspace): + write_raw(workspace, "abc", [user("derive the update rule for Adam"), assistant("Sure.")]) + assert sessions.read_meta(sessions.path_for("abc"))["title"] == ( + "derive the update rule for Adam" + ) + + +def test_an_explicit_title_outranks_the_derived_one(workspace): + write_raw( + workspace, "abc", + [{"type": "meta", "id": "abc", "title": "width vs depth"}, user("something else")], + ) + assert sessions.read_meta(sessions.path_for("abc"))["title"] == "width vs depth" + + +def test_a_long_first_message_is_cut_rather_than_filling_the_picker(workspace): + write_raw(workspace, "abc", [user("x" * 400)]) + title = sessions.read_meta(sessions.path_for("abc"))["title"] + assert len(title) <= sessions.TITLE_CHARS + assert title.endswith("…") + + +def test_a_session_with_nothing_in_it_still_has_a_name(workspace): + write_raw(workspace, "abc", [{"type": "meta", "id": "abc", "title": ""}]) + assert sessions.read_meta(sessions.path_for("abc"))["title"] == "empty session" + + +# --------------------------------------------------------------------------- +# the listing +# --------------------------------------------------------------------------- +def test_the_listing_is_most_recently_written_first(workspace): + import os + import time + + for index, session_id in enumerate(("aaa", "bbb", "ccc")): + write_raw(workspace, session_id, [user(f"question {index}")]) + # mtimes, set explicitly: three writes inside one filesystem tick would + # otherwise be indistinguishable and the order arbitrary. + stamp = time.time() + index + os.utime(sessions.path_for(session_id), (stamp, stamp)) + + assert [row["id"] for row in sessions.listing()] == ["ccc", "bbb", "aaa"] + + +def test_the_listing_counts_messages_and_ignores_the_meta_line(workspace): + write_raw( + workspace, "abc", + [{"type": "meta", "id": "abc", "title": "t"}, user("one"), assistant("two")], + ) + assert sessions.listing()[0]["messages"] == 2 + + +def test_a_damaged_transcript_does_not_make_the_picker_unopenable(workspace): + path = sessions.path_for("broken") + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("{not json at all\n" + json.dumps(user("hello")), encoding="utf-8") + write_raw(workspace, "fine", [user("a good one")]) + + listed = {row["id"]: row for row in sessions.listing()} + assert "fine" in listed + assert listed["broken"]["title"] == "hello", "the readable lines still count" + + +def test_files_that_are_not_sessions_are_left_alone(workspace): + (sessions.sessions_dir() / "notes.jsonl").write_text("{}", encoding="utf-8") + write_raw(workspace, "abc", [user("hello")]) + assert [row["id"] for row in sessions.listing()] == ["abc"] + + +# --------------------------------------------------------------------------- +# the upgrade path +# --------------------------------------------------------------------------- +def test_the_transcript_that_existed_before_sessions_is_already_one(workspace): + """It was `ui_session-default.jsonl` and the scheme here is + `ui_session-.jsonl`, so there is no migration step -- which is the whole + reason the naming was left alone.""" + path = sessions.sessions_dir() / "ui_session-default.jsonl" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(user("the conversation from before")), encoding="utf-8") + + listed = sessions.listing() + assert [row["id"] for row in listed] == [sessions.LEGACY_ID] + assert listed[0]["title"] == "the conversation from before" + assert listed[0]["resumable"] is False, "no SDK id was ever recorded for it" + + +def test_most_recent_is_none_on_a_workspace_with_no_sessions(workspace): + assert sessions.most_recent() is None + + +# --------------------------------------------------------------------------- +# the two ids +# --------------------------------------------------------------------------- +def test_a_written_session_round_trips_with_both_ids(workspace): + sessions.write( + "abc", [user("hello"), assistant("hi")], + title="a title", created_at="2026-08-15T00:00:00Z", sdk_session_id="sdk-1", + ) + meta = sessions.read_meta(sessions.path_for("abc")) + assert meta["title"] == "a title" + assert meta["created_at"] == "2026-08-15T00:00:00Z" + assert meta["sdk_session_id"] == "sdk-1" + assert meta["messages"] == 2 + + +def test_resumable_is_exactly_whether_the_sdk_id_is_known(workspace): + """The difference between reopening a conversation and reopening a + transcript. A picker that showed both the same way would be lying about the + more important half.""" + sessions.write("with", [user("hi")], sdk_session_id="sdk-1") + sessions.write("without", [user("hi")]) + + listed = {row["id"]: row for row in sessions.listing()} + assert listed["with"]["resumable"] is True + assert listed["without"]["resumable"] is False + + +def test_the_meta_line_is_not_mistaken_for_a_transcript_record(workspace): + """`app.Session.restore` keeps only records whose `role` is a real role, so + the header costs nothing there -- which is why it is a line in the same file + rather than a second file that could disagree with it.""" + from ui import app as app_mod + + sessions.write("abc", [user("hello")], title="t", sdk_session_id="sdk-1") + lines = sessions.path_for("abc").read_text(encoding="utf-8").splitlines() + first = json.loads(lines[0]) + assert first["type"] == "meta" + assert first.get("role") not in app_mod.ROLES + + +def test_deleting_a_session_is_idempotent(workspace): + sessions.write("abc", [user("hello")]) + assert sessions.delete("abc") is True + assert sessions.delete("abc") is False + assert sessions.listing() == [] + + +# --------------------------------------------------------------------------- +# listing cost +# --------------------------------------------------------------------------- +def test_reading_a_written_session_stops_at_the_header(workspace, monkeypatch): + """`listing()` calls `read_meta` once per session, so a scan to the end + would make listing cost the total size of every transcript in the + workspace -- and transcripts grow with the conversations most worth coming + back to.""" + sessions.write("abc", [user("hello"), assistant("hi")], title="a title") + + path = sessions.path_for("abc") + real_open = open + read_lines: list[int] = [] + + class CountingFile: + def __init__(self, handle): + self._handle = handle + self._count = 0 + + def __iter__(self): + for line in self._handle: + self._count += 1 + yield line + + def __enter__(self): + return self + + def __exit__(self, *exc): + read_lines.append(self._count) + return self._handle.__exit__(*exc) + + def counting_open(file, *args, **kwargs): + handle = real_open(file, *args, **kwargs) + if str(file) == str(path): + return CountingFile(handle.__enter__().__class__ and handle) + return handle + + monkeypatch.setattr("builtins.open", counting_open) + meta = sessions.read_meta(path) + + assert meta["messages"] == 2 + assert read_lines == [1], "the header alone answered it" + + +def test_a_legacy_file_with_no_header_still_falls_back_to_a_full_scan(workspace): + """One file, once. A transcript written before this format existed has no + header, so its title is its first user message and its count is its lines.""" + write_raw(workspace, sessions.LEGACY_ID, [user("from before"), assistant("hi")]) + meta = sessions.read_meta(sessions.path_for(sessions.LEGACY_ID)) + assert meta["title"] == "from before" + assert meta["messages"] == 2 + + +# --------------------------------------------------------------------------- +# one client per session +# --------------------------------------------------------------------------- +def test_a_session_is_claimed_by_one_client_at_a_time(workspace): + """`_persist` writes the whole file, so two clients on one session is two + writers and the loser's turns disappear with nothing to say they had.""" + sessions.write("abc", [user("hello")]) + assert sessions.claim("abc", "client-1") is True + assert sessions.claim("abc", "client-2") is False + assert sessions.claim("abc", "client-1") is True, "the holder may re-take it" + assert sessions.holder("abc") == "client-1" + + sessions.release("client-1") + assert sessions.holder("abc") is None + assert sessions.claim("abc", "client-2") is True + + +def test_a_second_window_gets_the_next_session_not_the_same_one(workspace): + import os + import time + + for index, session_id in enumerate(("older", "newer")): + sessions.write(session_id, [user(f"q{index}")]) + stamp = time.time() + index + os.utime(sessions.path_for(session_id), (stamp, stamp)) + + assert sessions.most_recent("client-1") == "newer" + assert sessions.most_recent("client-2") == "older" + # Nothing left to hand out. + assert sessions.most_recent("client-3") is None + + +def test_most_recent_without_an_owner_claims_nothing(workspace): + sessions.write("abc", [user("hello")]) + assert sessions.most_recent() == "abc" + assert sessions.holder("abc") is None + + +def test_two_clients_on_a_fresh_workspace_do_not_land_on_the_same_session(workspace): + """The gap a fixed fallback name left. `adopt` writes no file, so both + clients see an empty listing -- and reaching for `default` there put two + writers on one transcript before either had said anything, which is the + exact race the claim exists to prevent.""" + from ui import app as app_mod + + first, second = app_mod.Session("client-1"), app_mod.Session("client-2") + first.adopt() + second.adopt() + + assert first.session_id != second.session_id + assert sessions.holder(first.session_id) == "client-1" + assert sessions.holder(second.session_id) == "client-2" + + +def test_a_second_client_takes_the_next_session_and_then_a_new_one(workspace): + """With sessions on disk: the second client gets the next-most-recent, and + a third -- with nothing left to hand out -- gets one of its own.""" + from ui import app as app_mod + + sessions.write("only-one", [user("hello")]) + clients = [app_mod.Session(f"client-{n}") for n in range(3)] + for client in clients: + client.adopt() + + ids = [client.session_id for client in clients] + assert ids[0] == "only-one" + assert len(set(ids)) == 3, "no two clients on one transcript" + + +def test_a_client_that_disconnects_hands_its_session_back(workspace): + import asyncio + + from ui import app as app_mod + + sessions.write("abc", [user("hello")]) + first = app_mod.Session("client-1") + first.adopt() + assert first.session_id == "abc" + + asyncio.run(first.release()) + + second = app_mod.Session("client-2") + second.adopt() + assert second.session_id == "abc", "the session was never handed back" + + +def test_a_claim_is_scoped_to_the_workspace_it_was_made_in(monkeypatch, tmp_path, workspace): + """Every root has a `default`, so a claim keyed on the bare id would let one + client switching workspaces find another client's claim on a file it has + never seen.""" + sessions.write(sessions.LEGACY_ID, [user("in the first workspace")]) + assert sessions.claim(sessions.LEGACY_ID, "client-1") is True + + other = tmp_path / "another-workspace" + other.mkdir() + monkeypatch.setenv("GRAD_ROOT", str(other)) + from core import paths + + paths.ensure_workspace() + sessions.write(sessions.LEGACY_ID, [user("in the second workspace")]) + + assert sessions.holder(sessions.LEGACY_ID) is None + assert sessions.claim(sessions.LEGACY_ID, "client-2") is True diff --git a/tests/test_ui_shell.py b/tests/test_ui_shell.py index dd1fcd4..936693f 100644 --- a/tests/test_ui_shell.py +++ b/tests/test_ui_shell.py @@ -19,6 +19,8 @@ from __future__ import annotations +from typing import Any + import pytest pytest.importorskip("nicegui", reason="the ui extra is not installed") @@ -34,14 +36,25 @@ class FakeSession: `Session.start` only runs on the first `ask`, so a render touches nothing.""" busy = False - buffer = "" def __init__(self) -> None: - self.settled: list[dict[str, str]] = [] + self.settled: list[dict[str, Any]] = [] + self.blocks: list[dict[str, Any]] = [] + self.session_id = "default" + self.title = "" + self.sdk_session_id: str | None = None def interrupt(self) -> None: pass + async def open_session(self, session_id: str) -> str: + self.session_id = session_id + return f"opened {session_id}" + + async def new_session(self, title: str = "") -> str: + self.session_id = "fresh" + return "new session" + @pytest.fixture def rendered(workspace): @@ -108,13 +121,77 @@ def test_a_window_whose_render_raises_does_not_take_the_shell_down(rendered, mon # --------------------------------------------------------------------------- # the chrome reflects the layout # --------------------------------------------------------------------------- -def test_the_opener_marks_open_windows(rendered): +def _open_window_menu(client: Client, space): + """The `⋯` menu, drawn. Its body is built on open rather than at build time, + for the same reason the project menu's is: a toggle makes the list it was + read from stale.""" + from nicegui import ui as nicegui_ui + + with client: + menu = shell._windows_menu(nicegui_ui, space) # noqa: SLF001 - no public hook + menu.open() + return menu + + +def _menu_rows(client: Client) -> list: + return [e for e in client.elements.values() if "grad-menu-row" in getattr(e, "classes", [])] + + +def _menu_row(client: Client, window_id: str): + """One window's row, found by the hint it carries as a tooltip.""" + from ui import kit + + wanted = kit.attr(registry.spec(window_id).hint) + for element in _menu_rows(client): + if element.props.get("title") == wanted: + return element + raise AssertionError(f"no menu row for {window_id!r}") + + +def test_the_window_menu_marks_what_is_open(rendered): + """The `⋯` menu replaced a permanent strip of eleven names and a `⌘K` + palette that listed the same eleven. It is the only opener now, so it is the + only place the open/closed state appears.""" client, space = rendered(["chat"]) - opener_cells = [ - e for e in client.elements.values() if "grad-opener-cell" in getattr(e, "classes", []) - ] - assert len(opener_cells) == len(registry.ids()) - assert len([c for c in opener_cells if "open" in c.classes]) == 1 + _open_window_menu(client, space) + + rows = _menu_rows(client) + assert len(rows) == len(registry.ids()) + len(shell.PRESET_ROWS) + assert [r.props.get("title") for r in rows].count(None) == 0 + assert len([r for r in rows if "open" in r.classes]) == 1 + assert "open" in _menu_row(client, "chat").classes + + +def test_the_window_menu_toggles_in_place_rather_than_closing(rendered): + """Opening three windows is three clicks. A menu that dismissed itself after + each one would be three trips back to the same button.""" + client, space = rendered(["chat"]) + _open_window_menu(client, space) + + click(_menu_row(client, "ledger")) + assert "ledger" in space.layout.windows + # Redrawn in place, so the mark beside the row is no longer stale. + assert "open" in _menu_row(client, "ledger").classes + + click(_menu_row(client, "ledger")) + assert "ledger" not in space.layout.windows + assert "open" not in _menu_row(client, "ledger").classes + + +def test_a_quote_in_a_tooltip_cannot_truncate_the_props_string(rendered): + """`props('title="…")` is parsed by NiceGUI, so a `"` in the value ends it + early and silently drops whatever came after. Ledger text reaches these -- + a preflight remedy, a candidate id -- so it goes through `kit.attr`.""" + from ui import kit + + client, _ = rendered(["chat"]) + with client: + element = kit.button("FIX", title='run --note "see below"\nand retry') + + # One line, and no double quote left to close the attribute early. + assert '"' not in element.props["title"] + assert "\n" not in element.props["title"] + assert "see below" in element.props["title"] def test_a_handle_sits_between_every_pair_of_columns(rendered): @@ -177,8 +254,14 @@ def test_a_verify_flips_the_notebook_chip_without_a_retile(rendered): def click(element) -> None: - """Invoke an element's click handlers, the way the browser would.""" - for listener in element._event_listeners.values(): # noqa: SLF001 - no public hook + """Invoke an element's click handlers, the way the browser would. + + Snapshotted first: a handler may rebuild the subtree it was clicked in -- + the `⋯` menu redraws itself after a toggle -- and deleting the element + mutates the dict this is walking. The browser has the same freedom because + it dispatches by id rather than by iterating. + """ + for listener in list(element._event_listeners.values()): # noqa: SLF001 - no public hook if listener.type == "click" and listener.handler is not None: listener.handler() @@ -224,6 +307,136 @@ def test_answering_a_gate_sends_the_decision_into_the_session(rendered): assert sent and "denied" in sent[0] +# --------------------------------------------------------------------------- +# the calls the agent made +# --------------------------------------------------------------------------- +def test_a_settled_turn_draws_a_card_for_every_call_it_made(rendered): + """What the transcript is for: a command the agent ran and a command it + only claimed to run must not look alike.""" + client, space = rendered(["chat"]) + with client: + from ui.windows.chat import _message + + _message( + { + "role": "assistant", + "text": "Checking.", + "blocks": [ + {"kind": "text", "text": "Checking."}, + {"kind": "tool", "name": "Bash", "title": "python -m tools.ledger show", + "text": "python -m tools.ledger show", "rows": [], + "status": "ok", "result": "3 expectations"}, + {"kind": "tool", "name": "Bash", "title": "ssh probe-host", "text": "ssh probe-host", + "rows": [], "status": "error", "result": "denied by the gate"}, + ], + }, + space, + ) + + markup = html_of(client) + assert markup.count("grad-card tool") == 2 + assert "python -m tools.ledger show" in markup + assert "3 expectations" in markup + assert "OK" in markup and "ERROR" in markup + + +def test_the_turn_in_flight_scrolls_with_the_transcript(rendered): + """The tail has to be *inside* the scrolling region. As a sibling below it + the tail grows without bound, so a turn with three tool cards scrolls + `.grad-body` instead and paints over the composer.""" + client, _ = rendered(["chat"]) + scroller = _by_id(client, "grad-transcript") + tail = _by_id(client, "grad-tail") + assert scroller in _ancestors(tail) + assert scroller._style.get("overflow-y") == "auto" # noqa: SLF001 - no public accessor + + +def _by_id(client: Client, element_id: str): + for element in client.elements.values(): + if element.props.get("id") == element_id: + return element + raise AssertionError(f"no element with id {element_id!r}") + + +def _ancestors(element) -> list: + """Every element between this one and the page root.""" + out = [] + slot = element.parent_slot + while slot is not None: + out.append(slot.parent) + slot = slot.parent.parent_slot + return out + + +def test_a_turn_with_no_blocks_still_draws_as_prose(rendered): + """Transcripts written before the calls were captured have no `blocks`, and + a user's own message never will.""" + client, space = rendered(["chat"]) + with client: + from ui.windows.chat import _message + + _message({"role": "assistant", "text": "GATE — YOUR CALL\ncost: $18.40\n"}, space) + + assert "GATE" in html_of(client) + + +def test_the_tail_appends_a_card_rather_than_redrawing_the_turn(rendered): + """The split-tail rule, extended: prose already in the tail must not be + rebuilt 15 times a second just because a call landed under it.""" + client, space = rendered(["chat"]) + with client: + from ui.windows import chat as chat_window + + tail = chat_window._Tail(chat_window.kit.column("", gap=0)) + tail.sync([{"kind": "text", "text": "Checking."}]) + prose = tail._drawn[0]["body"].id + tail.sync([ + {"kind": "text", "text": "Checking."}, + {"kind": "tool", "name": "Bash", "title": "ls", "text": "ls", "rows": [], + "status": "running", "result": ""}, + ]) + assert tail._drawn[0]["body"].id == prose # the prose element is the same one + assert "RUNNING" in html_of(client) + + # ... and the same card is repainted in place when the result lands. + card = tail._drawn[1]["state"].id + tail.sync([ + {"kind": "text", "text": "Checking."}, + {"kind": "tool", "name": "Bash", "title": "ls", "text": "ls", "rows": [], + "status": "ok", "result": "budget.py"}, + ]) + assert tail._drawn[1]["state"].id == card + markup = html_of(client) + assert "RUNNING" not in markup + assert "budget.py" in markup + + +def test_a_new_turn_clears_the_tail_rather_than_stacking_onto_it(rendered): + client, space = rendered(["chat"]) + with client: + from ui.windows import chat as chat_window + + tail = chat_window._Tail(chat_window.kit.column("", gap=0)) + tail.sync([{"kind": "text", "text": "the first turn"}]) + tail.sync([]) # settled: the tail was promoted + assert tail._drawn == [] + tail.sync([{"kind": "text", "text": "the second turn"}]) + assert len(tail._drawn) == 1 + assert "the first turn" not in html_of(client) + + +def test_the_status_line_names_the_call_in_flight(rendered): + """A spinner says something is happening; naming the command says a + 40-minute job is running and which one.""" + from ui.windows.chat import _activity + + assert _activity([]) == "running …" + assert _activity([ + {"kind": "tool", "name": "Bash", "title": "python -m tools.jobs run", "status": "running"}, + ]) == "running Bash python -m tools.jobs run" + assert _activity([{"kind": "tool", "name": "Bash", "title": "ls", "status": "ok"}]) == "running …" + + def test_the_focused_window_is_marked(rendered): client, space = rendered(["chat", "ledger"]) space.focus("ledger") @@ -276,6 +489,36 @@ def test_the_chat_transcript_survives_a_retile(rendered): assert [e.id for e in still] == [identity] +def test_a_swap_moves_the_windows_and_keeps_both_roots(rendered): + """A swap retiles, so it goes through the same teardown as a drag -- and the + set of live windows does not change, so every root has to come back out of + the attic. If it did not, dropping the ledger onto the chat would swap the + panes and wipe the transcript in the same gesture.""" + client, space = rendered(["chat", "ledger"]) + bodies_before = { + e.id for e in client.elements.values() if "grad-body" in getattr(e, "classes", []) + } + assert [c.windows for c in space.layout.columns] == [["chat"], ["ledger"]] + + space.swap("chat", "ledger") + + assert [c.windows for c in space.layout.columns] == [["ledger"], ["chat"]] + bodies_after = { + e.id for e in client.elements.values() if "grad-body" in getattr(e, "classes", []) + } + assert bodies_before == bodies_after, "a swap rebuilt a window root" + + +def test_a_drop_at_a_slot_boundary_reorders_within_the_column(rendered): + client, space = rendered(["chat", "ledger", "quota"]) + space.preset("stack") + assert space.layout.columns[0].windows == ["chat", "ledger", "quota"] + space.retile("quota", 0, 0) + assert space.layout.columns[0].windows == ["quota", "chat", "ledger"] + windows = [e for e in client.elements.values() if "grad-window" in getattr(e, "classes", [])] + assert len(windows) == 3 + + def test_closing_a_window_destroys_its_root(rendered): """Otherwise the attic accumulates a detached subtree per window per session, each one still bound to the poll.""" @@ -293,6 +536,166 @@ def test_reopening_a_closed_window_builds_a_fresh_root(rendered): assert "ledger" in _root_ids(client) +# --------------------------------------------------------------------------- +# switching project and folder +# --------------------------------------------------------------------------- +def test_the_project_menu_lists_the_folder_and_its_projects(rendered): + from core import budget as budget_mod + + budget_mod.create("proj-a", title="Scaling laws", budget={}) + budget_mod.set_current("proj-a") + client, space = rendered(["chat"]) + + from ui import shell as shell_mod + + menu = [e for e in client.elements.values() if "grad-card" in getattr(e, "classes", [])] + assert menu, "the menu dialog was not built" + # Drawn on open, not at build time: creating a project makes the list it was + # read from stale, so it is rebuilt each time. + with client: + shell_mod._draw_project_menu( # noqa: SLF001 - no public hook + __import__("nicegui").ui, space, menu[0], _NullMenu() + ) + markup = html_of(client) + assert "proj-a" in markup + assert "Scaling laws" in markup + assert "WORKSPACE" in markup + + +class _NullMenu: + def close(self) -> None: + pass + + def redraw(self) -> None: + pass + + +def test_the_workspace_menu_can_store_a_credential_without_a_terminal(rendered, monkeypatch): + """The one thing the workspace could not do. `credential set` prompts with + `getpass`, which needs a terminal -- so a fresh machine needed a shell open + beside the app before the app was usable.""" + from core import budget as budget_mod + from ui import shell as shell_mod, tasks as tasks_mod + + calls: list[dict] = [] + + async def fake_run_tool(*argv, timeout=120.0, stdin=None): + calls.append({"argv": argv, "stdin": stdin}) + return {"ok": True, "data": {"message": "stored"}} + + monkeypatch.setattr(tasks_mod, "run_tool", fake_run_tool) + monkeypatch.setattr("ui.state.run_tool", fake_run_tool) + + budget_mod.create("proj-a", title="A", budget={}) + budget_mod.set_current("proj-a") + client, space = rendered(["chat"]) + card = [e for e in client.elements.values() if "grad-card" in getattr(e, "classes", [])][0] + with client: + shell_mod._draw_project_menu( # noqa: SLF001 - no public hook + __import__("nicegui").ui, space, card, _NullMenu() + ) + + assert "CREDENTIALS" in html_of(client) + import asyncio + + asyncio.run(space.set_credential("hf_token", "hf_the-actual-token")) + + assert calls, "no command was run" + argv, stdin = calls[0]["argv"], calls[0]["stdin"] + # Down a pipe, never in an argument: an argv is visible to anything that can + # list processes. + assert stdin == "hf_the-actual-token" + assert "--stdin" in argv + assert not any("hf_the-actual-token" in part for part in argv) + + +def test_an_empty_credential_is_refused_before_a_command_runs(rendered, monkeypatch): + from ui import tasks as tasks_mod + + async def explode(*argv, **kwargs): + raise AssertionError("a command ran for an empty value") + + monkeypatch.setattr("ui.state.run_tool", explode) + monkeypatch.setattr(tasks_mod, "run_tool", explode) + + _, space = rendered(["chat"]) + import asyncio + + asyncio.run(space.set_credential("hf_token", " ")) + assert "nothing to store" in (space.notice or "") + + +def test_the_folder_picker_argument_survives_a_process_boundary(): + """Native mode marshals `create_file_dialog` to the pywebview process over a + multiprocessing queue, so its arguments have to pickle. + + `webview.FOLDER_DIALOG` does not: it is a deprecated `proxy_tools.Proxy` + that reprs as `20` while being a proxy around a function, and it fails with + "it's not the same object as webview.FOLDER_DIALOG". Worse, the error is + raised in the queue's feeder thread, so it prints a traceback and hangs the + picker instead of raising anywhere it could be caught -- which is why this + is asserted here rather than left to a try/except at the call site. + """ + import pickle + + from ui import shell as shell_mod + + value = shell_mod.folder_dialog_type() + assert type(value) is int, f"a plain int, not {type(value).__name__}" + assert pickle.loads(pickle.dumps(value)) == value + + webview = pytest.importorskip("webview", reason="pywebview is not installed") + # Still the value pywebview means by "folder", however it spells it now. + assert value == int(webview.FileDialog.FOLDER) + + +def test_switching_project_reloads_the_layout_for_that_project(rendered): + """Layout persists per project, so the panes have to follow the switch -- + otherwise the new project opens with the old one's arrangement and silently + overwrites its layout file on the next drag.""" + from core import budget as budget_mod + from ui import state as state_module + + budget_mod.create("proj-a", title="A", budget={}) + budget_mod.create("proj-b", title="B", budget={}) + budget_mod.set_current("proj-a") + + client, space = rendered(["chat", "ledger"]) + space.project = "proj-a" + space.preset("stack") + stacked = [c.windows for c in space.layout.columns] + + # proj-b has never been opened, so it gets the default arrangement. + budget_mod.set_current("proj-b") + space.reload() + assert space.project == "proj-b" + assert [c.windows for c in space.layout.columns] != stacked + assert state_module.layout_path("proj-b").name == "proj-b.json" + + +def test_a_reload_redraws_the_windows_rather_than_leaving_them_stale(rendered): + """A retile reuses live roots -- that is what stops a drag wiping the + transcript -- so a reload has to redraw the bodies explicitly or the panes + would be rearranged for the new workspace while still showing the old one.""" + from core import ledger_store as ls + + client, space = rendered(["ledger"]) + ls.append_expectation( + {"id": "exp-1", "task": "t", "created_at": ls.now_iso(), "quantity": "val_loss", + "claim": "a claim from the first workspace", + "predicted": {"low": 1.0, "high": 2.0, "direction": None}, + "basis": [], "comparability": "same", "confidence": "low"} + ) + space.tick() + assert "a claim from the first workspace" in html_of(client) + + space.reload() + # Same workspace here, so the claim is still true -- what is being checked + # is that the body was re-rendered at all, not that it changed. + assert "a claim from the first workspace" in html_of(client) + assert space.models == {} or "ledger" in space.models + + # --------------------------------------------------------------------------- # the whole lifecycle, in one pass # --------------------------------------------------------------------------- diff --git a/tests/test_ui_state.py b/tests/test_ui_state.py index 2ce2c27..ae6c9b6 100644 --- a/tests/test_ui_state.py +++ b/tests/test_ui_state.py @@ -254,3 +254,140 @@ def test_envelope_message_prefers_the_fix(): ) assert "gate refused" in message assert "run preflight" in message + + +# --------------------------------------------------------------------------- +# switching the workspace folder +# --------------------------------------------------------------------------- +class RebindableSession(FakeSession): + """A session that records the two things a folder switch must do to it.""" + + def __init__(self) -> None: + self.settled = [{"role": "user", "text": "from the old workspace"}] + self.rebound = 0 + + async def rebind(self) -> None: + self.rebound += 1 + self.settled.clear() + + +@pytest.fixture +def pointer(tmp_path, monkeypatch): + """Redirect the pointer file. The real one lives beside the code, and a test + that wrote it would rewrite the developer's own workspace choice.""" + from core import workspace as workspace_mod + + monkeypatch.setattr( + workspace_mod, "pointer_path", lambda: tmp_path / "pointer" / "p.json" + ) + (tmp_path / "pointer").mkdir(parents=True, exist_ok=True) + monkeypatch.setattr(workspace_mod, "_cache", None, raising=False) + yield + workspace_mod._cache = None + + +@pytest.mark.asyncio +async def test_switching_folder_moves_the_paths_the_config_and_the_session( + workspace, tmp_path, pointer +): + """The three things that have to move together. Missing any one leaves the + app half-switched in a way that is hard to see from the screen.""" + from core import config as config_mod, paths + + elsewhere = tmp_path / "another-workspace" + space = state_mod.Workspace(RebindableSession(), "proj") + config_mod.load() # populate the cache the switch has to clear + + await space.switch_root(str(elsewhere), create=True) + + assert paths.root() == elsewhere.resolve() # the paths + assert not config_mod._cache # the config, which moved with them + assert space.session.rebound == 1 # the session and its agent cwd + assert space.session.settled == [] + # The new folder is a workspace, not just a directory. + assert paths.ledger_dir().is_dir() + assert str(elsewhere.resolve()) in (space.notice or "") + + +@pytest.mark.asyncio +async def test_switching_folder_reloads_the_project_and_its_layout(workspace, tmp_path, pointer): + from core import budget as budget_mod, paths + + space = state_mod.Workspace(RebindableSession(), None) + space.preset("stack") + stacked = [c.windows for c in space.layout.columns] + + elsewhere = tmp_path / "another-workspace" + await space.switch_root(str(elsewhere), create=True) + + # A fresh folder has no project selected and no saved layout, so both come + # back to their defaults rather than carrying over from the old workspace. + assert budget_mod.current_project() is None + assert space.project is None + assert [c.windows for c in space.layout.columns] != stacked + assert state_mod.layout_dir() == paths.data_dir() / "layouts" + + +@pytest.mark.asyncio +async def test_a_folder_that_cannot_be_used_is_a_notice_not_a_crash(workspace, tmp_path, pointer): + """The path comes from a text field, so a bad one is an everyday event. The + app must stay where it is and say why.""" + from core import paths + + before = paths.root() + space = state_mod.Workspace(RebindableSession(), "proj") + + a_file = tmp_path / "notes.txt" + a_file.write_text("hi", encoding="utf-8") + await space.switch_root(str(a_file)) + + assert paths.root() == before + assert space.session.rebound == 0 # nothing was torn down + assert "not a folder" in (space.notice or "") + + +@pytest.mark.asyncio +async def test_a_blank_folder_does_not_move_the_workspace(workspace, pointer): + from core import paths + + before = paths.root() + space = state_mod.Workspace(RebindableSession(), "proj") + await space.switch_root(" ") + assert paths.root() == before + assert space.notice + + +@pytest.mark.asyncio +async def test_creating_a_project_selects_it_and_reloads(workspace, pointer): + """Through the CLI, like every other button that does something -- so it + lands in the same ledger the agent's own `tools.budget new` would write.""" + from core import budget as budget_mod + + space = state_mod.Workspace(RebindableSession(), None) + await space.create_project("proj-new", "A new piece of research") + + assert budget_mod.current_project() == "proj-new" + assert budget_mod.projects()["proj-new"]["title"] == "A new piece of research" + assert space.project == "proj-new" + + +@pytest.mark.asyncio +async def test_a_refused_project_id_reports_the_clis_own_message(workspace, pointer): + space = state_mod.Workspace(RebindableSession(), None) + await space.create_project("not a valid slug!", "whatever") + assert space.notice and "slug" in space.notice.lower() + assert space.project is None + + +@pytest.mark.asyncio +async def test_using_a_project_switches_the_selection(workspace, pointer): + from core import budget as budget_mod + + budget_mod.create("proj-a", title="A", budget={}) + budget_mod.create("proj-b", title="B", budget={}) + budget_mod.set_current("proj-a") + + space = state_mod.Workspace(RebindableSession(), "proj-a") + await space.use_project("proj-b") + assert budget_mod.current_project() == "proj-b" + assert space.project == "proj-b" diff --git a/tests/test_ui_tasks.py b/tests/test_ui_tasks.py new file mode 100644 index 0000000..125b792 --- /dev/null +++ b/tests/test_ui_tasks.py @@ -0,0 +1,333 @@ +"""The background task registry. + +Real subprocesses, not mocks. The whole point of this module is what happens to +a process -- that it is not killed by a wall clock, that stopping it asks the +tool first, that its output arrives while it runs -- and none of those are +properties of a fake. + +The commands under test are throwaway scripts written into the temp workspace. +`tasks.start` runs `python -m ` with `cwd=paths.root()`, and `-m` puts the +working directory on `sys.path`, so a file dropped in the workspace root is +importable by name. +""" + +from __future__ import annotations + +import asyncio +import textwrap +import time + +import pytest + +from ui import tasks as tasks_mod + +SETTLE_TIMEOUT_S = 60.0 + + +def script(workspace, name: str, body: str) -> str: + (workspace / f"{name}.py").write_text(textwrap.dedent(body), encoding="utf-8") + return name + + +async def settled(task, timeout: float = SETTLE_TIMEOUT_S): + deadline = time.monotonic() + timeout + while task.running and time.monotonic() < deadline: + await asyncio.sleep(0.02) + assert not task.running, f"{task.label} did not settle in {timeout}s" + return task + + +def lines(task) -> list[str]: + return [line for _, line in task.tail] + + +# --------------------------------------------------------------------------- +# the tail, without a process +# --------------------------------------------------------------------------- +def test_the_envelope_is_the_last_json_object_on_stdout(): + """The §8 envelope, tracked as the lines arrive. A command that printed a + gigabyte still costs one dict, and the *last* object wins because that is + what `run_tool` means by the envelope.""" + task = tasks_mod.Task("t", "x", ()) + task.append('{"ok": false, "error": {"message": "early"}}') + task.append("some progress") + task.append('{"ok": true, "data": {"message": "done"}}') + assert task.envelope == {"ok": True, "data": {"message": "done"}} + + +def test_stderr_does_not_become_the_envelope(): + task = tasks_mod.Task("t", "x", ()) + task.append('{"ok": true}', tag="err") + assert task.envelope is None + + +def test_a_tail_that_overflows_says_how_much_it_dropped(): + """A tail that silently forgets reads as complete output.""" + task = tasks_mod.Task("t", "x", ()) + for index in range(tasks_mod.TAIL_LINES + 25): + task.append(f"line {index}") + assert len(task.tail) == tasks_mod.TAIL_LINES + assert task.dropped == 25 + assert lines(task)[0] == "line 25" + + +def test_one_enormous_line_is_cut_rather_than_kept(): + task = tasks_mod.Task("t", "x", ()) + task.append("x" * (tasks_mod.MAX_LINE_CHARS * 3)) + only = lines(task)[0] + assert len(only) < tasks_mod.MAX_LINE_CHARS * 2 + assert "characters" in only + + +# --------------------------------------------------------------------------- +# running one +# --------------------------------------------------------------------------- +@pytest.mark.asyncio +async def test_a_task_streams_its_output_and_settles_ok(workspace): + name = script(workspace, "grad_ok", """ + import json, sys + print("working") + sys.stdout.flush() + print(json.dumps({"ok": True, "data": {"message": "finished"}})) + """) + task = tasks_mod.start("a task", name) + assert tasks_mod.get(task.id) is task, "registered before the click is over" + + await settled(task) + assert task.state == tasks_mod.OK + assert task.exit_code == 0 + assert "working" in lines(task) + assert tasks_mod.task_message(task) == "a task: finished" + + +@pytest.mark.asyncio +async def test_a_failing_task_keeps_its_exit_code_and_its_stderr(workspace): + name = script(workspace, "grad_bad", """ + import sys + print("about to fail") + print("the reason", file=sys.stderr) + sys.exit(9) + """) + task = await settled(tasks_mod.start("a failure", name)) + assert task.state == tasks_mod.FAILED + assert task.exit_code == 9 + assert ("err", "the reason") in list(task.tail) + + +@pytest.mark.asyncio +async def test_a_command_that_cannot_start_fails_rather_than_hanging(workspace): + task = await settled(tasks_mod.start("nonsense", "grad_no_such_module_at_all")) + assert task.state == tasks_mod.FAILED + + +@pytest.mark.asyncio +async def test_a_line_longer_than_the_stream_readers_limit_survives(workspace): + """`StreamReader.readline` raises past 64 KiB and a training log's progress + line can pass it, which is why `_pump` reads chunks.""" + name = script(workspace, "grad_long", """ + print("y" * 200_000) + print("after") + """) + task = await settled(tasks_mod.start("a long line", name)) + assert task.state == tasks_mod.OK + assert "after" in lines(task) + + +@pytest.mark.asyncio +async def test_the_completion_callback_runs_once_and_cannot_strand_the_task(workspace): + calls: list[str] = [] + name = script(workspace, "grad_quiet", "print('hi')") + + def boom(task): + calls.append(task.state) + raise RuntimeError("the workspace blew up recording this") + + task = await settled(tasks_mod.start("a task", name, on_done=boom)) + assert calls == [tasks_mod.OK] + assert task.state == tasks_mod.OK, "a callback that raised must not unsettle the task" + assert any("could not record" in line for line in lines(task)) + + +# --------------------------------------------------------------------------- +# stopping one +# --------------------------------------------------------------------------- +@pytest.mark.asyncio +async def test_stopping_a_task_with_no_halt_verb_signals_it(workspace): + name = script(workspace, "grad_forever", """ + import time + print("started", flush=True) + time.sleep(600) + """) + task = tasks_mod.start("a long one", name) + for _ in range(400): # let it get as far as printing + if "started" in lines(task): + break + await asyncio.sleep(0.02) + + message = await tasks_mod.cancel(task.id) + assert "stopped" in message or "killed" in message + await settled(task, timeout=30.0) + assert task.state == tasks_mod.CANCELLED, "a stop is not a failure" + + +@pytest.mark.asyncio +async def test_stopping_asks_the_tool_before_it_signals(workspace): + """The whole reason `halt` exists. `nb verify` starts its kernel *detached* + so it outlives the CLI -- so signalling the CLI would leave the kernel + holding the VRAM the verify was meant to free.""" + flag = workspace / "please-stop" + name = script(workspace, "grad_pollster", f""" + import pathlib, time + stop = pathlib.Path(r"{flag}") + print("started", flush=True) + for _ in range(3000): + if stop.exists(): + print("asked to stop, exiting cleanly", flush=True) + raise SystemExit(0) + time.sleep(0.02) + """) + halt = script(workspace, "grad_halt", f""" + import json, pathlib + pathlib.Path(r"{flag}").write_text("stop") + print(json.dumps({{"ok": True, "data": {{"message": "asked"}}}})) + """) + + task = tasks_mod.start("a pollster", name, halt=(halt,)) + for _ in range(400): + if "started" in lines(task): + break + await asyncio.sleep(0.02) + + await tasks_mod.cancel(task.id) + await settled(task, timeout=30.0) + assert task.state == tasks_mod.CANCELLED + assert flag.exists(), "the tool's own stop verb was never run" + assert any("exiting cleanly" in line for line in lines(task)), ( + "it was signalled rather than asked" + ) + assert not any("killed" in line for line in lines(task)) + + +@pytest.mark.asyncio +async def test_stopping_something_already_finished_says_so(workspace): + task = await settled(tasks_mod.start("quick", script(workspace, "grad_quick", "pass"))) + assert "already finished" in await tasks_mod.cancel(task.id) + assert task.state == tasks_mod.OK, "a late cancel must not rewrite the verdict" + + +@pytest.mark.asyncio +async def test_cancelling_an_unknown_task_is_a_message_not_a_crash(): + assert "no task" in await tasks_mod.cancel("task-404") + + +@pytest.mark.asyncio +async def test_stopping_a_task_before_its_process_exists_still_stops_it(workspace): + """`start` registers the task and spawns on the next tick, so a fast click + finds `_process` still unset. Settling the task there and walking away would + leave the process to run to completion with nothing on screen saying so.""" + marker = workspace / "it-ran-to-completion" + name = script(workspace, "grad_marker", f""" + import pathlib, time + time.sleep(3) + pathlib.Path(r"{marker}").write_text("finished") + """) + task = tasks_mod.start("a fast cancel", name) + assert task._process is None, "the spawn should not have happened yet" # noqa: SLF001 + + await tasks_mod.cancel(task.id) + assert task.state == tasks_mod.CANCELLED + await tasks_mod.drained(task) + assert not marker.exists(), "the process outlived the task that reported it stopped" + + +@pytest.mark.asyncio +async def test_a_running_tasks_driver_is_held_against_collection(workspace): + """asyncio keeps only a weak reference to a running task, so a bare + `create_task` can vanish mid-flight -- leaving a live process with no pump + and nothing left to settle it.""" + task = tasks_mod.start("held", script(workspace, "grad_held", "print('hi')")) + assert task._driver is not None # noqa: SLF001 + await settled(task) + + +# --------------------------------------------------------------------------- +# the registry +# --------------------------------------------------------------------------- +@pytest.mark.asyncio +async def test_finished_tasks_are_clearable_and_running_ones_are_not(workspace): + done = await settled(tasks_mod.start("done", script(workspace, "grad_done", "pass"))) + running = tasks_mod.start( + "running", + script(workspace, "grad_slow", "import time; time.sleep(600)"), + ) + try: + assert tasks_mod.clear_finished() == 1 + assert tasks_mod.get(done.id) is None + assert tasks_mod.get(running.id) is running + finally: + await tasks_mod.cancel(running.id) + await tasks_mod.drained(running) + + +def test_the_newest_task_is_listed_first(): + for index in range(3): + tasks_mod._register(tasks_mod.Task(f"task-{index}", f"t{index}", ())) # noqa: SLF001 + assert [t.id for t in tasks_mod.all_tasks()] == ["task-2", "task-1", "task-0"] + + +def test_the_registry_keeps_a_bounded_history_of_finished_tasks(): + """Enough to look back over a session, not so many that the window becomes + a history of the machine.""" + for index in range(tasks_mod.KEEP_FINISHED + 12): + task = tasks_mod.Task(f"task-{index}", "t", ()) + task.state = tasks_mod.OK + tasks_mod._register(task) # noqa: SLF001 + assert len(tasks_mod.all_tasks()) == tasks_mod.KEEP_FINISHED + + +def test_a_running_task_is_never_evicted_by_the_history_bound(): + survivor = tasks_mod._register(tasks_mod.Task("task-keep", "long", ())) # noqa: SLF001 + for index in range(tasks_mod.KEEP_FINISHED + 12): + task = tasks_mod.Task(f"task-{index}", "t", ()) + task.state = tasks_mod.OK + tasks_mod._register(task) # noqa: SLF001 + assert tasks_mod.get("task-keep") is survivor + + +# --------------------------------------------------------------------------- +# run_tool, the other half of the same decision +# --------------------------------------------------------------------------- +@pytest.mark.asyncio +async def test_run_tool_returns_the_envelope(workspace): + name = script(workspace, "grad_env", """ + import json + print("noise on the way") + print(json.dumps({"ok": True, "data": {"message": "hello"}})) + """) + payload = await tasks_mod.run_tool(name) + assert payload == {"ok": True, "data": {"message": "hello"}} + assert tasks_mod.envelope_message(payload) == "hello" + + +@pytest.mark.asyncio +async def test_run_tool_says_where_a_long_command_belongs_when_it_times_out(workspace): + """The timeout here is enforced by killing the process, which is exactly why + the long commands do not go through this function -- `nb verify` allows 1800 + seconds *per cell* and was being killed at 900.""" + name = script(workspace, "grad_sleepy", "import time; time.sleep(600)") + payload = await tasks_mod.run_tool(name, timeout=1.0) + assert payload["ok"] is False + assert "timed out" in payload["error"]["message"] + assert "background" in payload["error"]["fix"] + + +@pytest.mark.asyncio +async def test_run_tool_reports_a_command_that_printed_no_envelope(workspace): + name = script(workspace, "grad_silent", """ + import sys + print("something went wrong", file=sys.stderr) + sys.exit(2) + """) + payload = await tasks_mod.run_tool(name) + assert payload["ok"] is False + assert "something went wrong" in payload["error"]["message"] diff --git a/tests/test_ui_tokens.py b/tests/test_ui_tokens.py index 2f1433a..0a51a5b 100644 --- a/tests/test_ui_tokens.py +++ b/tests/test_ui_tokens.py @@ -143,7 +143,7 @@ def test_the_stylesheet_covers_every_component_class_the_kit_emits(): the failure mode hardest to notice in review.""" sheet = tokens.stylesheet() for name in ( - "grad-shell", "grad-appbar", "grad-opener", "grad-statusbar", "grad-tiles", + "grad-shell", "grad-appbar", "grad-dots", "grad-menu-row", "grad-statusbar", "grad-tiles", "grad-column", "grad-slot", "grad-handle", "grad-window", "grad-titlebar", "grad-body", "grad-btn", "grad-chip", "grad-kv", "grad-bar", "grad-progress", "grad-status-square", "grad-band", "grad-stage", "grad-lineage", "grad-diff", diff --git a/tests/test_workspace.py b/tests/test_workspace.py new file mode 100644 index 0000000..8a63d8e --- /dev/null +++ b/tests/test_workspace.py @@ -0,0 +1,305 @@ +"""Choosing the workspace folder (`core/workspace.py`, `paths.root`). + +The rule this file exists to hold still is the **precedence**: GRAD_ROOT, then +the remembered choice, then the installed directory. Getting it wrong in either +direction is bad in a way that is hard to see -- a remembered folder that beat +the environment would silently redirect the test suite and every explicit +command line, and one that never won would make the app's own folder chooser +forget on every restart. + +Every test here redirects the pointer file into a temp directory. The real one +lives beside the code, so a test that forgot would rewrite the developer's own +workspace choice as a side effect of running the suite. +""" + +from __future__ import annotations + +import json +import os + +import pytest + +from core import paths, workspace as workspace_mod +from core.errors import UsageError + + +@pytest.fixture +def pointer(tmp_path, monkeypatch): + """A pointer file of our own, and a cache that does not leak between tests.""" + path = tmp_path / "pointer" / ".grad-workspace.json" + path.parent.mkdir(parents=True, exist_ok=True) + monkeypatch.setattr(workspace_mod, "pointer_path", lambda: path) + monkeypatch.setattr(workspace_mod, "_cache", None, raising=False) + yield path + workspace_mod._cache = None + + +@pytest.fixture +def no_env(monkeypatch): + """Without this, GRAD_ROOT wins and the pointer is never consulted.""" + monkeypatch.delenv("GRAD_ROOT", raising=False) + + +@pytest.fixture +def restart(monkeypatch): + """Simulate relaunching the app. + + `select` sets GRAD_ROOT in this process -- that is how the switch reaches + the CLIs -- so after a switch the environment legitimately wins and the + pointer is never consulted. Only a new process asks the pointer anything, + and a new process is exactly what these tests are about. + """ + + def _restart() -> None: + monkeypatch.delenv("GRAD_ROOT", raising=False) + workspace_mod._cache = None + + return _restart + + +# --------------------------------------------------------------------------- +# precedence +# --------------------------------------------------------------------------- +def test_the_environment_beats_a_remembered_folder(tmp_path, pointer, monkeypatch): + """An explicit override stays explicit. The suite itself relies on this: + `conftest` sets GRAD_ROOT, and a remembered folder that won would point every + test at a real workspace.""" + remembered = tmp_path / "remembered" + remembered.mkdir() + explicit = tmp_path / "explicit" + explicit.mkdir() + workspace_mod.select(remembered) + + monkeypatch.setenv("GRAD_ROOT", str(explicit)) + assert paths.root() == explicit.resolve() + assert workspace_mod.source() == "environment" + + +def test_a_remembered_folder_beats_the_installed_directory(tmp_path, pointer, no_env, restart): + chosen = tmp_path / "chosen" + chosen.mkdir() + workspace_mod.select(chosen) + restart() + assert paths.root() == chosen.resolve() + assert workspace_mod.source() == "remembered" + + +def test_with_nothing_remembered_the_root_is_where_the_code_lives(pointer, no_env): + assert paths.root() == workspace_mod.code_dir() + assert workspace_mod.source() == "default" + + +def test_selecting_sets_the_environment_so_subprocesses_agree(tmp_path, pointer, monkeypatch): + """The CLIs run as subprocesses and inherit this environment. A switch that + only updated `paths` would leave the agent's Bash tools reading the folder + the UI just left.""" + chosen = tmp_path / "chosen" + chosen.mkdir() + workspace_mod.select(chosen) + assert os.environ["GRAD_ROOT"] == str(chosen.resolve()) + + +# --------------------------------------------------------------------------- +# what survives a restart +# --------------------------------------------------------------------------- +def test_the_choice_is_remembered_across_a_fresh_read(tmp_path, pointer, no_env): + chosen = tmp_path / "chosen" + chosen.mkdir() + workspace_mod.select(chosen) + + workspace_mod._cache = None # a new process + assert workspace_mod.remembered() == chosen.resolve() + + +def test_a_remembered_folder_that_no_longer_exists_is_not_returned( + tmp_path, pointer, no_env, restart +): + """The folder may have been deleted, renamed, or live on a drive that is not + mounted today. Returning it would send every ledger read somewhere that + cannot be created.""" + gone = tmp_path / "gone" + gone.mkdir() + workspace_mod.select(gone) + gone.rmdir() + + restart() + assert workspace_mod.remembered() is None + assert paths.root() == workspace_mod.code_dir() + + +def test_a_corrupt_pointer_is_not_a_startup_failure(pointer, no_env): + pointer.write_text("{not json", encoding="utf-8") + workspace_mod._cache = None + assert workspace_mod.remembered() is None + assert paths.root() == workspace_mod.code_dir() + + +@pytest.mark.parametrize("garbage", ["null", '"a string"', "[]", '{"root": 7}', '{"root": ""}']) +def test_a_hand_edited_pointer_yields_no_root_rather_than_a_traceback(pointer, no_env, garbage): + pointer.write_text(garbage, encoding="utf-8") + workspace_mod._cache = None + assert workspace_mod.remembered() is None + + +def test_a_pointer_that_cannot_be_written_still_switches_this_process(tmp_path, monkeypatch): + """A system-wide install is read-only. The switch has to apply now even if + it cannot be remembered for next time.""" + chosen = tmp_path / "chosen" + chosen.mkdir() + monkeypatch.setattr( + workspace_mod, "pointer_path", lambda: tmp_path / "nope" / "deeper" / "p.json" + ) + monkeypatch.setattr(workspace_mod, "_cache", None, raising=False) + assert workspace_mod.select(chosen) == chosen.resolve() + assert os.environ["GRAD_ROOT"] == str(chosen.resolve()) + workspace_mod._cache = None + + +# --------------------------------------------------------------------------- +# the recent list +# --------------------------------------------------------------------------- +def test_the_folder_you_leave_is_what_you_can_get_back_to(tmp_path, pointer, monkeypatch): + """Recording only the destination leaves the history empty exactly when it + matters: after one switch its only entry is where you already are, which the + menu filters out. Switching back is the whole point of the list.""" + from ui import models + + here, there = tmp_path / "here", tmp_path / "there" + here.mkdir() + there.mkdir() + monkeypatch.setenv("GRAD_ROOT", str(here)) + + workspace_mod.select(there) + assert here.resolve() in workspace_mod.recent() + assert models.workspaces_model()["recent"] == [str(here.resolve())] + + +def test_recent_folders_are_most_recent_first_and_deduplicated(tmp_path, pointer, no_env): + made = [] + for name in ("a", "b", "c"): + folder = tmp_path / name + folder.mkdir() + made.append(folder.resolve()) + workspace_mod.select(folder) + workspace_mod.select(made[0]) # back to the first + + listed = workspace_mod.recent() + assert listed[:3] == [made[0], made[2], made[1]] + # `a` was visited twice and appears once. The tail is the folder the first + # switch left, which is the installed directory here. + assert listed.count(made[0]) == 1 + assert listed[3:] == [workspace_mod.code_dir()] + + +def test_the_recent_list_is_capped(tmp_path, pointer, no_env): + for index in range(workspace_mod.MAX_RECENT + 4): + folder = tmp_path / f"w{index}" + folder.mkdir() + workspace_mod.select(folder) + assert len(workspace_mod.recent()) <= workspace_mod.MAX_RECENT + + +def test_a_recent_folder_that_was_deleted_is_dropped_from_the_list(tmp_path, pointer, no_env): + keep, gone = tmp_path / "keep", tmp_path / "gone" + keep.mkdir() + gone.mkdir() + workspace_mod.select(gone) + workspace_mod.select(keep) + assert gone.resolve() in workspace_mod.recent() + gone.rmdir() + assert gone.resolve() not in workspace_mod.recent() + assert keep.resolve() in workspace_mod.recent() + + +# --------------------------------------------------------------------------- +# validation -- the value comes from a text field +# --------------------------------------------------------------------------- +def test_a_blank_folder_is_refused_with_a_fix(): + with pytest.raises(UsageError) as caught: + workspace_mod.validate(" ") + assert caught.value.fix + + +def test_a_file_is_refused_as_a_workspace(tmp_path): + target = tmp_path / "notes.txt" + target.write_text("hi", encoding="utf-8") + with pytest.raises(UsageError, match="not a folder"): + workspace_mod.validate(target) + + +def test_a_missing_folder_is_refused_unless_creating(tmp_path): + missing = tmp_path / "does" / "not" / "exist" + with pytest.raises(UsageError, match="does not exist"): + workspace_mod.validate(missing) + assert workspace_mod.validate(missing, create=True) == missing.resolve() + assert missing.is_dir() + + +def test_surrounding_quotes_and_whitespace_are_tolerated(tmp_path): + """Copying a path out of a file manager brings the quotes with it.""" + folder = tmp_path / "with space" + folder.mkdir() + assert workspace_mod.validate(f' "{folder}" ') == folder.resolve() + + +def test_a_home_relative_path_is_expanded(monkeypatch, tmp_path): + monkeypatch.setenv("USERPROFILE", str(tmp_path)) # Windows + monkeypatch.setenv("HOME", str(tmp_path)) # everywhere else + (tmp_path / "grad").mkdir() + assert workspace_mod.validate("~/grad") == (tmp_path / "grad").resolve() + + +def test_validate_does_not_switch_anything(tmp_path, monkeypatch): + """`validate` is called to check a field; only `select` may move the app.""" + monkeypatch.setenv("GRAD_ROOT", str(tmp_path / "current")) + (tmp_path / "current").mkdir() + other = tmp_path / "other" + other.mkdir() + workspace_mod.validate(other) + assert os.environ["GRAD_ROOT"] == str(tmp_path / "current") + + +# --------------------------------------------------------------------------- +# the menu's model +# --------------------------------------------------------------------------- +def test_the_menu_model_lists_projects_and_marks_the_current_one(workspace, pointer): + from core import budget as budget_mod + from ui import models + + budget_mod.create("proj-a", title="Scaling laws", budget={"gpu_usd": 100.0}) + budget_mod.create("proj-b", title="Optimisers", budget={}) + budget_mod.set_current("proj-b") + + model = models.workspaces_model() + assert model["current_project"] == "proj-b" + by_id = {p["id"]: p for p in model["projects"]} + assert by_id["proj-b"]["current"] is True + assert by_id["proj-a"]["current"] is False + assert by_id["proj-a"]["spend"].startswith("gpu ") + # A project with no ceilings says so rather than showing an empty meter. + assert by_id["proj-b"]["spend"] == "no ceilings" + + +def test_the_menu_model_opens_on_an_empty_workspace(workspace, pointer): + """It is the panel that has to render when the workspace is wrong -- that is + what it is for.""" + from ui import models + + model = models.workspaces_model() + assert model["projects"] == [] + assert model["root"] == str(workspace) + + +def test_the_current_folder_is_not_offered_as_somewhere_to_go(tmp_path, pointer, no_env): + from ui import models + + first, second = tmp_path / "first", tmp_path / "second" + first.mkdir() + second.mkdir() + workspace_mod.select(first) + workspace_mod.select(second) + + model = models.workspaces_model() + assert model["root"] == str(second.resolve()) + assert str(second.resolve()) not in model["recent"] + assert str(first.resolve()) in model["recent"] diff --git a/tools/jobs.py b/tools/jobs.py index 1185bea..27c95a4 100644 --- a/tools/jobs.py +++ b/tools/jobs.py @@ -14,6 +14,7 @@ import argparse import datetime as _dt import json +import sys import time from pathlib import Path from typing import Any @@ -674,18 +675,17 @@ def _download_artifacts(r: ls.Run, dest: Path) -> None: # setup runs at import time -- listing it here would read all four entries out # of Windows Credential Manager on every `jobs.py` invocation, including # `collect` and `ceilings`. -CREDENTIAL_NAMES = ( - credentials.HF_TOKEN, - credentials.OPENROUTER_KEY, - credentials.VOYAGE_KEY, - credentials.S2_KEY, - credentials.CONTEXT7_KEY, -) +CREDENTIAL_NAMES = credentials.ALL def _credential_args(p: argparse.ArgumentParser) -> None: p.add_argument("action", choices=["status", "set", "delete"]) p.add_argument("name", nargs="?", help=f"one of: {', '.join(CREDENTIAL_NAMES)}") + p.add_argument( + "--stdin", + action="store_true", + help="read the value from stdin rather than prompting (for the workspace UI)", + ) @cli.command("credential", "inspect or set stored credentials (values are never printed)", setup=_credential_args) @@ -710,9 +710,23 @@ def cmd_credential(args: argparse.Namespace) -> dict[str, Any]: if args.action == "delete": credentials.delete(args.name) return {"deleted": args.name} - import getpass + if args.name not in CREDENTIAL_NAMES: + raise UsageError( + f"unknown credential {args.name!r}", + fix=f"one of: {', '.join(CREDENTIAL_NAMES)}", + ) + + if args.stdin: + # A pipe, not an argument. The value is a token, and an argv is visible + # to anything that can list processes -- which on a shared machine is + # everything. `getpass` is the same guarantee for a human at a terminal; + # this is the guarantee for a caller that has no terminal to prompt at, + # which is what the workspace's credential panel is. + value = sys.stdin.read().strip() + else: + import getpass - value = getpass.getpass(f"value for {args.name} (not echoed): ") + value = getpass.getpass(f"value for {args.name} (not echoed): ") if not value: raise UsageError("empty value", fix="run it again and paste the token") credentials.set_(args.name, value) diff --git a/tools/paper_ingest.py b/tools/paper_ingest.py index 934b2f2..2dc839b 100644 --- a/tools/paper_ingest.py +++ b/tools/paper_ingest.py @@ -49,10 +49,15 @@ # arXiv # --------------------------------------------------------------------------- def _fetch(url: str, *, timeout: float) -> bytes: - try: - import httpx # noqa: PLC0415 - except ImportError as exc: - raise ConfigError("httpx is not installed", fix="pip install httpx") from exc + """arXiv's e-print endpoint, through `core/http.py`'s accessor. + + Not a bare `import httpx`. Every other outbound request in this project goes + through `http._httpx()`, and the suite's "no network" guarantee is + implemented by replacing exactly that function -- so a second import site is + a hole in it, and a test that reached this one would hang rather than fail. + It also gets the same ImportError message for free. + """ + httpx = http._httpx() # noqa: SLF001 - the module's own accessor, see above try: resp = httpx.get(url, timeout=timeout, follow_redirects=True) except Exception as exc: # noqa: BLE001 diff --git a/tools/paper_search.py b/tools/paper_search.py index 9a101f1..caa2122 100644 --- a/tools/paper_search.py +++ b/tools/paper_search.py @@ -1,7 +1,7 @@ """grad-paper-search -- the five-stage retrieval funnel (HANDOFF §5). | 0 | Query expansion | Haiku: 1 question -> ~5 keyword queries + 1 HyDE abstract | quota | - | 1 | Retrieve | S2 snippets + local index (RRF) + citation expansion | free | + | 1 | Retrieve | Asta snippets + local index (RRF) + citation expansion | free | | 2 | Rerank | voyageai/rerank-2.5 -> top ~50 | credits| | 3 | Triage | Haiku reads all 50 in one call, returns ~15 with a reason | quota | | 4 | Select | The main agent reads the 15 | quota | @@ -19,26 +19,60 @@ import re from typing import Any -from core import config as config_mod, corpus, haiku, http, paths, quota_log +from core import config as config_mod, corpus, credentials, haiku, http, paths, quota_log from core.cli import Cli, main -from core.errors import GradError, UsageError +from core.errors import GradError, UpstreamError, UsageError from core.ledger_store import now_iso cli = Cli( "grad-paper-search", - "Search the literature (Semantic Scholar) and the local index, rerank, and triage.", + "Search the literature (Ai2 Asta / Semantic Scholar) and the local index, rerank, triage.", epilog=( - "Discovery and recall are different problems. Tier 1 (S2) finds papers you have\n" - "not read; tier 2 (the local index) answers 'where did I see that lemma'.\n" + "Discovery and recall are different problems. Tier 1 finds papers you have not\n" + "read; tier 2 (the local index) answers 'where did I see that lemma'.\n" "A local index cannot do discovery by construction, which is why both exist.\n\n" + "Tier 1 defaults to Asta, which serves the same Semantic Scholar corpus over MCP\n" + "without an institutional email. --tier1 s2 uses the REST API directly.\n\n" "The retriever sets the ceiling: expansion and citation expansion buy more than\n" "reranker shopping does." ), ) +#: `tier1` value -> the clients it selects, in the order they are queried. +TIER1_SOURCES = ("asta", "s2", "both", "none") + + +def tier1_clients(cfg: Any, override: str | None = None) -> list[tuple[str, Any]]: + """The discovery clients for this run, named so a trace can say which spoke. + + Both reach the same Semantic Scholar corpus and both answer in the same + vocabulary (`core/http.py:_row`), so a candidate found by either fuses to + one entry. What differs is whether the door opens: S2's own API no longer + issues keys to free-domain addresses, so a personal account falls back to + the shared anonymous pool. + """ + chosen = str(override or cfg.get("retrieval", "tier1", "asta")).lower() + if chosen not in TIER1_SOURCES: + raise UsageError( + f"unknown tier-1 source {chosen!r}", + fix=f"one of: {', '.join(TIER1_SOURCES)}", + ) + out: list[tuple[str, Any]] = [] + if chosen in ("asta", "both"): + out.append(("asta", http.Asta(cfg))) + if chosen in ("s2", "both"): + out.append(("s2", http.SemanticScholar(cfg))) + return out + + def _search_args(p: argparse.ArgumentParser) -> None: p.add_argument("question", help="the research question, in words") + p.add_argument( + "--tier1", + choices=list(TIER1_SOURCES), + help="which discovery client to use (default from config; asta unless changed)", + ) p.add_argument("--top", type=int, help="how many to return (default from config)") p.add_argument("--candidates", type=int, help="stage-1 candidate ceiling") p.add_argument("--no-expand", action="store_true", help="skip stage 0 (Haiku query expansion)") @@ -77,31 +111,41 @@ def cmd_search(args: argparse.Namespace) -> dict[str, Any]: # -- stage 1: retrieve --------------------------------------------------- candidates: dict[str, dict[str, Any]] = {} rankings: list[list[dict[str, Any]]] = [] + #: Search calls that *failed*, kept apart from the trace's other warnings. + #: An empty local index is not a failure and must not be reported as one. + upstream_failures: list[str] = [] + + tier1 = tier1_clients(cfg, args.tier1) if not args.local_only else [] + trace["stages"]["1_sources"] = [name for name, _ in tier1] - if not args.local_only: - s2 = http.SemanticScholar(cfg) - per_query = max(5, ceiling // max(1, len(queries) * 2)) + if tier1: + per_query = max(5, ceiling // max(1, len(queries) * 2 * len(tier1))) for query in queries: - for fn in (s2.snippet_search, s2.paper_search): - try: - hits = fn(query, limit=per_query) - except GradError as exc: - trace.setdefault("warnings", []).append(str(exc)) - continue - rankings.append(hits) - for hit in hits: - candidates.setdefault(hit["id"], hit) - if not args.no_citations: - seeds = [c for c in list(candidates.values())[:5] if c.get("paper_id")] - for seed in seeds: - for direction in ("citations", "references"): + for name, client in tier1: + for verb in ("snippet_search", "paper_search"): try: - hits = s2.neighbours(seed["paper_id"], direction=direction, limit=10) - except GradError: + hits = getattr(client, verb)(query, limit=per_query) + except GradError as exc: + trace.setdefault("warnings", []).append(f"{name}.{verb}: {exc}") + upstream_failures.append(f"{name}.{verb}: {exc}") continue rankings.append(hits) for hit in hits: candidates.setdefault(hit["id"], hit) + if not args.no_citations: + seeds = [c for c in list(candidates.values())[:5] if c.get("paper_id")] + for seed in seeds: + for name, client in tier1: + for direction in ("citations", "references"): + try: + hits = client.neighbours( + seed["paper_id"], direction=direction, limit=10 + ) + except GradError: + continue + rankings.append(hits) + for hit in hits: + candidates.setdefault(hit["id"], hit) if not args.no_local: local = _local_ranked(args.question, hyde, cfg, trace) @@ -118,8 +162,33 @@ def cmd_search(args: argparse.Namespace) -> dict[str, Any]: trace["stages"]["1_retrieve"] = {"rankings": len(rankings), "candidates": len(pool)} if not pool: - return {"question": args.question, "results": [], "trace": trace, - "note": "no candidates; try --no-expand to see the raw query, or widen --candidates"} + # A run that found nothing is the run the funnel view exists to explain + # -- "why is the obviously relevant paper not in here" -- so it gets a + # trace like any other. Returning before writing one left exactly the + # interesting failures invisible. + _write_trace(log_name, trace, []) + warnings = list(dict.fromkeys(trace.get("warnings") or [])) + if upstream_failures and not rankings: + # Every retrieval call failed. That is an upstream failure, not an + # empty result set, and the difference matters more here than + # anywhere else in this tool: `ok: true` with no results reads as + # "the literature has nothing on this", which is a conclusion nobody + # should draw from a rate limit. + raise UpstreamError( + "every retrieval call failed, so the search returned nothing: " + + "; ".join(dict.fromkeys(upstream_failures)), + fix=_tier1_fix(trace["stages"]["1_sources"]), + ) + return { + "question": args.question, + "results": [], + "trace": trace, + "trace_log": str(paths.notes_dir() / "funnel" / f"{log_name}.json"), + # The old note here recommended `--no-expand`, which is advice for a + # cause this branch cannot distinguish and sent anyone following it + # to a second empty run. + "note": "; ".join(warnings) or "no candidates; widen --candidates or rephrase", + } # -- stage 2: rerank ----------------------------------------------------- ranked = pool @@ -187,6 +256,31 @@ def apply_rerank(pool: list[dict[str, Any]], scored: list[dict[str, Any]]) -> li ] +def _tier1_fix(sources: list[str]) -> str: + """What to actually do when discovery is down, per source. + + This used to say "store a Semantic Scholar API key -- it is free", which + stopped being true: Ai2 no longer accept key requests from free-domain email + addresses, so for a personal account that instruction has no ending. Advice + that cannot be followed is worse than no advice, because it is followed + first and the real fix is found second. + """ + if sources == ["s2"]: + return ( + "Semantic Scholar's own API only issues keys to institutional addresses, so " + "this is the shared anonymous pool. Switch to Ai2's Asta, which serves the " + "same corpus and does not require one: " + "python -m tools.paper_search search '' --tier1 asta --json " + "(or set [retrieval] tier1 = \"asta\" in config/grad.toml)" + ) + return ( + "retry -- discovery is rate limited, not broken. A key raises Asta's limits and " + "is requested from a form rather than reviewed: " + f"python -m tools.jobs credential set {credentials.ASTA_KEY}. " + "Meanwhile --local-only searches what is already ingested." + ) + + def _local_ranked(question: str, hyde: str | None, cfg: Any, trace: dict[str, Any]) -> list[dict[str, Any]]: """Tier 2, fused across FTS5 and vectors before joining the global pool.""" path = paths.corpus_sqlite() diff --git a/ui/app.py b/ui/app.py index 4279bb2..f06853d 100644 --- a/ui/app.py +++ b/ui/app.py @@ -21,10 +21,11 @@ Two implementation details are the difference between this feeling like a tool and feeling like a demo, and both survive from the first version: - * **Buffered flush.** Tokens go into a buffer and a `ui.timer` flushes at - ~15 Hz, rather than re-rendering a markdown element per token. - * **Split tail.** The streaming message lives in its own element, separate - from the settled transcript above it, so only the tail re-renders. + * **Buffered flush.** Tokens accumulate in the turn's blocks and a `ui.timer` + flushes at ~15 Hz, rather than re-rendering a markdown element per token. + * **Split tail.** The streaming turn lives in its own element, separate from + the settled transcript above it, so only the tail re-renders -- and inside + the tail, only the block that moved. Notebooks render, they do not rebuild: JupyterLab already exists and is better at editing. Building a notebook editor is the single easiest way to burn a month @@ -42,9 +43,7 @@ from typing import Any from core import config as config_mod, paths -from ui import katex, kit, shell, state as state_mod - -SESSION_PREFIX = "ui_session" +from ui import katex, kit, sessions, shell, state as state_mod ROLES = ("user", "assistant") STATIC_URL = "/grad-static" @@ -58,7 +57,7 @@ def static_dir() -> Path: class Session: - """Owns the `ClaudeSDKClient` and the token buffer. + """Owns the `ClaudeSDKClient` and the turn in flight. The UI holds no logic of its own beyond this: everything else it shows is read from the ledger or produced by the CLIs. @@ -71,9 +70,21 @@ class Session: def __init__(self, key: str = "default") -> None: self.key = key self.client: Any = None - self.buffer: str = "" - self.settled: list[dict[str, str]] = [] + #: The turn in flight, as `agent.TurnStream` blocks: prose, and the tool + #: calls between it. The chat window's timer draws from here, so a card + #: for a running command is on screen while it runs. + self.blocks: list[dict[str, Any]] = [] + self.settled: list[dict[str, Any]] = [] self.busy = False + #: Which named session this is. `sessions.LEGACY_ID` on an upgraded + #: machine, because the transcript that was already there is one. + self.session_id: str = sessions.LEGACY_ID + self.title: str = "" + self.created_at: str | None = None + #: The *SDK's* id for this conversation, which is what `resume` takes. + #: Ours names the file; this one is what makes reopening continue rather + #: than merely redisplay. See the note in `ui/sessions.py`. + self.sdk_session_id: str | None = None self._task: asyncio.Task[None] | None = None async def start(self) -> None: @@ -84,9 +95,113 @@ async def start(self) -> None: cfg = config_mod.load() agent.preflight_environment() - self.client = ClaudeSDKClient(options=agent.build_options(cfg)) + options = agent.build_options(cfg, resume=self.sdk_session_id) + self.client = ClaudeSDKClient(options=options) await self.client.__aenter__() + # -- named sessions ----------------------------------------------------- + def adopt(self) -> None: + """Take a session for this client, and load it. + + The most recent one *no other window is in*, or a new one when they are + all taken. Two clients on one session is two writers on one file -- + `_persist` writes the whole thing, so the loser's turns disappear -- and + `Session` has been one-per-client since it was written, for exactly that + reason. What changed is only that the file is now chosen rather than + derived from the client's own key, so the claim has to be explicit. + + Opening the most recent rather than a blank one is deliberate: reopening + the app is not a request to start over. + """ + chosen = sessions.most_recent(self.key) + if chosen is None: + # A fresh workspace, or every existing session already open in + # another window. Either way this client needs one nobody else can + # be in, and a **fresh id is the only thing that guarantees that**. + # + # This used to reach for `LEGACY_ID` when the listing was empty, on + # the theory that a first session should be named where the + # pre-sessions code wrote. That was a race with no upside: `adopt` + # writes no file, so two clients connecting to an empty workspace + # both see an empty listing, both pick the same fixed name, and the + # second one's claim fails -- putting two writers on one transcript + # before either had said anything. And the theory was empty as well: + # this branch only runs when the listing *is* empty, which means + # there is no legacy file to keep writing to. `listing()` would have + # found it, and `most_recent` would have returned it. + # + # A new id cannot collide, so there is no claim to lose here. + chosen = sessions.new_id() + sessions.claim(chosen, self.key) + self.session_id = chosen + self.restore() + + async def release(self) -> None: + """Hand the session back and shut the client down, on disconnect.""" + sessions.release(self.key) + await self.close() + + async def open_session(self, session_id: str) -> str: + """Switch to a stored session: its transcript, and its conversation. + + The client is dropped rather than reused. `resume` is fixed when the + client is built, so a client that is already running is already bound to + another conversation -- keeping it would show one session's transcript + while the next turn continued a different one, which is the single most + confusing thing this could do. + """ + if not sessions.is_id(session_id): + return f"not a session id: {session_id}" + if self.busy: + return "a turn is still running — interrupt it first" + if session_id == self.session_id: + return f"already in {self.title or session_id}" + if not sessions.claim(session_id, self.key): + # Refused rather than shared. Both windows would write the whole + # file on every turn, and the loser's turns would vanish with + # nothing on screen to say they had. + return "another window has that session open" + + self._persist() + sessions.release(self.key) + sessions.claim(session_id, self.key) + await self.close() + self.session_id = session_id + self.blocks = [] + self.settled.clear() + self.restore() + if self.sdk_session_id: + return f"resumed {self.title or session_id}" + return ( + f"opened {self.title or session_id} — the transcript is here, but the agent " + "has no memory of it; the next turn starts fresh" + ) + + async def new_session(self, title: str = "") -> str: + """Start a clean conversation, keeping the one being left. + + The old behaviour -- one file, forever -- meant the only way to start + clean was to delete the record, and in this project the record is where + the reasoning behind an expectation lives. + """ + if self.busy: + return "a turn is still running — interrupt it first" + self._persist() + sessions.release(self.key) + await self.close() + self.session_id = sessions.new_id() + sessions.claim(self.session_id, self.key) + self.title = title.strip() + self.created_at = None + self.sdk_session_id = None + self.blocks = [] + self.settled.clear() + # Written immediately: an empty session that exists is listable, and a + # new session that vanished because nothing was said in it yet would be + # a control that silently did nothing. + self._persist() + return "new session" + async def close(self) -> None: """Exit the client context entered by `start`. @@ -101,19 +216,50 @@ async def close(self) -> None: except Exception: # noqa: BLE001 - shutdown must not raise on the way out pass + async def rebind(self) -> None: + """Re-point this session at whatever the workspace root is now. + + `path()` is derived from `paths.data_dir()`, so the transcript *file* + follows a root switch by itself -- but two things do not. `settled` is + in memory, so the new workspace would open showing the old one's + conversation; and the SDK client's `cwd` was fixed when it was built, so + the agent would keep reading and writing in the folder you just left. + Dropping the client is enough for the second: `ask` starts a new one on + the next turn, from the environment as it is by then. + """ + await self.close() + self.blocks = [] + self.busy = False + self.settled.clear() + # Sessions live under the root's data directory, so the one that was + # open belongs to the folder just left -- and the claim on it does too. + sessions.release(self.key) + self.adopt() + async def ask(self, prompt: str, on_settle: Any) -> None: await self.start() import agent # noqa: PLC0415 self.settled.append({"role": "user", "text": prompt}) self.busy = True - self.buffer = "" + # The turn lands in the stream's blocks as it arrives; the chat window's + # ~15 Hz timer is what turns that into something on screen. The same list + # the stream appends to, not a copy -- a snapshot taken here would never + # gain a tool card, and the block being written is the block being drawn. + stream = agent.TurnStream() + self.blocks = stream.blocks try: await self.client.query(prompt) async for message in self.client.receive_response(): - text = agent._text_of(message) # noqa: SLF001 - one helper, deliberately shared - if text: - self.buffer += text + # Captured from the stream rather than asked for: the SDK + # assigns it, and this is the id `resume` takes when the session + # is reopened. Every message carrying one carries the same one, + # so the first is enough -- but a resumed conversation can be + # given a *new* id by the CLI, so the latest wins. + sdk_id = getattr(message, "session_id", None) + if isinstance(sdk_id, str) and sdk_id: + self.sdk_session_id = sdk_id + stream.feed(message) except Exception as exc: # noqa: BLE001 - the transcript must say why a turn died # Otherwise the turn settles as an empty message: the prompt looks # unanswered and there is nothing on screen to say why. Only the @@ -121,18 +267,34 @@ async def ask(self, prompt: str, on_settle: Any) -> None: # message can carry a URL with a token in it, a header, or a path, # and both of those destinations are readable long after the fact. log.exception("session turn failed") - self.buffer += ( + # Whatever streamed before the failure is kept: a turn that died + # half-way is more legible with its half than without it -- and a + # tool card left mid-flight says which call it died on. + stream.note( f"\n\n**the session failed:** `{type(exc).__name__}` " "(details are in the app log)" ) finally: self.busy = False - settled_text = self.buffer - self.buffer = "" - if settled_text: - self.settled.append({"role": "assistant", "text": settled_text}) + # The first thing asked is what the session is about, and naming it + # from the prompt beats leaving every session called "empty session" + # until someone renames it by hand. An explicit title is never + # overwritten. + if not self.title: + self.title = sessions.title_from(prompt) + record = { + "role": "assistant", + "text": stream.text, + "blocks": list(stream.blocks), + } + self.blocks = [] + # `blocks`, not `text`: a turn that only ran commands and said + # nothing still happened, and dropping it would leave the transcript + # claiming the prompt went unanswered. + if record["blocks"]: + self.settled.append(record) self._persist() - await on_settle(settled_text) + await on_settle(record) def interrupt(self) -> None: """Interrupt the turn in flight, if there is one. @@ -153,25 +315,39 @@ async def _interrupt() -> None: self._task = asyncio.create_task(_interrupt()) def path(self) -> Path: - return paths.data_dir() / f"{SESSION_PREFIX}-{self.key}.jsonl" + return sessions.path_for(self.session_id) def _persist(self) -> None: """Closing the window should not be destructive.""" - path = self.path() - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text( - "\n".join(json.dumps(m, ensure_ascii=False) for m in self.settled), encoding="utf-8" - ) + try: + sessions.write( + self.session_id, + self.settled, + title=self.title, + created_at=self.created_at, + sdk_session_id=self.sdk_session_id, + ) + except (OSError, ValueError): + log.exception("could not persist session %s", self.session_id) def restore(self) -> None: - """Read the transcript back, keeping only records that render. + """Read a session back: its metadata, then the records that render. The file is on disk between runs, so a record is not necessarily one we wrote: the chat window subscripts `role` and `text`, and a line that is a bare string or is missing `text` would take the whole page down at build - time. + time. `blocks` is optional for the same reason in reverse -- transcripts + written before tool calls were drawn have none, and those still open. + + The `meta` line is skipped by that same filter rather than by a special + case: its `role` is absent, so it is not a record that renders. """ path = self.path() + meta = sessions.read_meta(path) + self.title = str(meta.get("title") or "") + self.created_at = meta.get("created_at") + sdk_id = meta.get("sdk_session_id") + self.sdk_session_id = sdk_id if isinstance(sdk_id, str) and sdk_id else None if not path.exists(): return for line in path.read_text(encoding="utf-8").splitlines(): @@ -184,7 +360,23 @@ def restore(self) -> None: and record.get("role") in ROLES and isinstance(record.get("text"), str) ): - self.settled.append({"role": record["role"], "text": record["text"]}) + kept: dict[str, Any] = {"role": record["role"], "text": record["text"]} + blocks = _drawable_blocks(record.get("blocks")) + if blocks: + kept["blocks"] = blocks + self.settled.append(kept) + + +def _drawable_blocks(value: Any) -> list[dict[str, Any]]: + """Blocks off disk that the chat window can actually draw. + + Same reason `restore` filters records: this file outlives the version that + wrote it, so `blocks` is untrusted input. A block with no `kind` would reach + the renderer's dispatch and take the page down at build time. + """ + if not isinstance(value, list): + return [] + return [b for b in value if isinstance(b, dict) and isinstance(b.get("kind"), str)] def build() -> None: @@ -212,12 +404,12 @@ def index() -> None: from nicegui import context # noqa: PLC0415 - page scope, not import scope session = Session(_client_key()) - session.restore() + session.adopt() workspace = state_mod.Workspace(session, _current_project()) # Per client, not `app.on_shutdown`: that would accumulate one handler # per connection and hold every session's subprocess open until the app # itself exits. - context.client.on_disconnect(session.close) + context.client.on_disconnect(session.release) shell.build(workspace) @@ -233,7 +425,12 @@ def _serve_static(nicegui_app: Any) -> None: directory = static_dir() directory.mkdir(parents=True, exist_ok=True) (directory / "fonts").mkdir(parents=True, exist_ok=True) - nicegui_app.add_static_files(STATIC_URL, str(directory)) + # `max_cache_age` defaults to an hour, which for a desktop app serving its + # own 16 KB of JavaScript over loopback buys nothing and costs a confusing + # hour: after an edit to `tiling.js`, a restarted app keeps running the old + # copy out of the webview's disk cache. The fonts beside it are large but + # never change, and they are the reason this is not zero. + nicegui_app.add_static_files(STATIC_URL, str(directory), max_cache_age=60) try: from tools import wiki as wiki_tool # noqa: PLC0415 - optional diff --git a/ui/kit.py b/ui/kit.py index 56380d2..227bffa 100644 --- a/ui/kit.py +++ b/ui/kit.py @@ -66,9 +66,27 @@ def _tone_class(tone: str | None) -> str: def escape(value: Any) -> str: + """HTML-escape, for text going into an element's content.""" return _html.escape("" if value is None else str(value), quote=False) +def attr(value: Any) -> str: + """A value safe to interpolate into a `props('name="…"')` string. + + Deliberately *not* `escape`. Props are parsed by NiceGUI and then bound as + attributes client-side, so nothing decodes entities on the way: an escaped + apostrophe would reach the screen as a literal `'` in the tooltip. + + What actually has to go is the double quote, which closes the value early + and takes every attribute after it with it -- silently, because the parser + has no reason to complain. That matters because these values are not all + constants: a preflight remedy and a lineage bar's candidate id are ledger + text, and ledger text can hold a quote. Newlines go for the same reason. + """ + collapsed = " ".join(str("" if value is None else value).split()) + return collapsed.replace('"', "'") + + def el(tag: str, classes: str = "", *, style: str = "") -> Any: """A bare container element with our classes on it.""" element = _ui().element(tag) @@ -127,7 +145,7 @@ def button( """A 2px-bordered square button. `tone` picks the fill, never a gradient.""" element = text(value, f"grad-btn {BUTTON_TONES.get(tone, '')} {classes}".strip(), tag="button") if title: - element.props(f'title="{escape(title)}"') + element.props(f'title="{attr(title)}"') if disabled: element.props("disabled").classes("disabled") elif on_click is not None: diff --git a/ui/layout.py b/ui/layout.py index 4e245c1..c614949 100644 --- a/ui/layout.py +++ b/ui/layout.py @@ -214,11 +214,34 @@ def close(self, window: str) -> Layout: def toggle(self, window: str) -> Layout: return self.close(window) if self.is_open(window) else self.open(window) - def move(self, window: str, column_index: int, slot_index: int | None = None) -> Layout: - """Retile: pull a window out and drop it into another column. - - `column_index == len(columns)` appends a new column, which is what a - drag past the right edge means. + def move( + self, + window: str, + column_index: int, + slot_index: int | None = None, + *, + new_column: bool = False, + ) -> Layout: + """Retile: pull a window out and drop it at a named position. + + `slot_index` is where in the target column it lands -- `None` appends, + which is what a drop with no vertical opinion means. `new_column` splits + a fresh column in at `column_index` rather than adding to the one already + there; `column_index == len(columns)` means the same thing at the right + edge, which is where a drag past the last pane ends up. + + Two corrections that are invisible until they are wrong: + + * **The cap is counted after the pull, not before.** Dragging the only + window out of a column empties it, and an empty column is dropped by + `normalise` -- so that drag can create a column without ever exceeding + `MAX_COLUMNS`. Counting the columns that still hold something is what + lets the gesture through while still refusing a genuine fourth. + * **Moving down inside one column shifts its own target.** The browser + computes `slot_index` against a column that still contains the dragged + window; by the time we insert, the pull has shifted everything after it + left by one. Without the adjustment, dragging a pane one place down + moves it two. """ found = self.locate(window) if not found: @@ -226,16 +249,56 @@ def move(self, window: str, column_index: int, slot_index: int | None = None) -> ci, si = found slot = self.columns[ci].slots.pop(si) slot.fraction = 1.0 + column_index = max(0, min(column_index, len(self.columns))) - if column_index == len(self.columns): - self.columns.append(Column([slot])) + wants_column = new_column or column_index == len(self.columns) + # Columns that still hold a window -- see the docstring. + live = sum(1 for c in self.columns if c.slots) + + if wants_column and live < MAX_COLUMNS: + self.columns.insert(column_index, Column([slot])) else: - target = self.columns[column_index].slots - index = len(target) if slot_index is None else max(0, min(slot_index, len(target))) - target.insert(index, slot) + # At the cap (or asked for a column we cannot make), the drop lands + # in the nearest real column rather than being refused: a gesture + # that visibly picked a pane up has to put it down somewhere. + if not self.columns: + self.columns = [Column([slot])] + else: + column_index = min(column_index, len(self.columns) - 1) + target = self.columns[column_index].slots + # Clamped against the column as the *browser* saw it -- one + # longer when the window came from this same column, because it + # was still in it when the boundary was picked. Clamping to the + # shortened list first and then correcting for the pull applies + # the same subtraction twice, and a drop past the last pane + # lands second-to-last. + limit = len(target) + (1 if ci == column_index else 0) + index = limit if slot_index is None else max(0, min(slot_index, limit)) + if ci == column_index and si < index: + index -= 1 + target.insert(index, slot) self.focused = window return self.normalise() + def swap(self, a: str, b: str) -> Layout: + """Exchange two windows, leaving the panes where they are. + + The slots keep their fractions and the windows trade places, rather than + each window carrying its size across with it. Dropping the ledger onto + the chat should put the ledger where the chat was, at the chat's size -- + not reflow the whole shell around a pane that just arrived. + """ + if a == b: + return self + first, second = self.locate(a), self.locate(b) + if not first or not second: + return self + (ac, as_), (bc, bs) = first, second + self.columns[ac].slots[as_].window = b + self.columns[bc].slots[bs].window = a + self.focused = a + return self.normalise() + def resize_columns(self, fractions: Iterable[float], *, total_px: int | None = None) -> Layout: values = list(fractions) if len(values) != len(self.columns): diff --git a/ui/models.py b/ui/models.py index d33ef79..9061d2d 100644 --- a/ui/models.py +++ b/ui/models.py @@ -28,7 +28,7 @@ import json import re from pathlib import Path -from typing import Any, Callable +from typing import Any, Callable, Iterable from core import paths @@ -100,6 +100,66 @@ def _tokens(value: Any) -> str: } +def workspaces_model() -> dict[str, Any]: + """What the project menu shows: this folder, the recent ones, the projects. + + Every reader is wrapped, because this is the one panel that has to render + when the workspace is *wrong* -- an empty folder, a ledger that will not + parse, a drive that is not mounted. A menu that cannot open because the + workspace it is meant to let you leave is broken is the one failure mode it + must not have. + """ + from core import budget as budget_mod, workspace as workspace_mod + + root, root_error = _safe(lambda: str(paths.root()), "") + recent, _ = _safe(lambda: [str(p) for p in workspace_mod.recent()], []) + current, _ = _safe(budget_mod.current_project) + records, projects_error = _safe(budget_mod.projects, {}) + + projects = [] + for project_id, record in sorted((records or {}).items()): + state, _ = _safe(lambda pid=project_id: budget_mod.status(pid), {}) + projects.append( + { + "id": project_id, + "title": _short(record.get("title") or "", 60), + "status": record.get("status") or "open", + "current": project_id == current, + "spend": _spend_line(state or {}), + } + ) + return { + "root": root, + # The rule that picked it -- "why is it still pointing there?" is + # otherwise unanswerable from inside the app. + "source": _safe(workspace_mod.source, "default")[0], + # The one already open is not offered as somewhere to go. + "recent": [p for p in (recent or []) if p != root], + "projects": projects, + "current_project": current, + "error": root_error or projects_error, + } + + +def _spend_line(state: dict[str, Any]) -> str: + """One line per project: what it has spent against what it may. + + A project with no ceilings is the common case -- they are optional -- and it + says so rather than rendering an empty bar, which would read as "nothing + spent" when it means "nothing to exceed". + """ + resources = state.get("resources") or {} + parts: list[str] = [] + for name, render in (("gpu_usd", _usd), ("quota_tokens", _tokens), ("credits_usd", _usd)): + entry = resources.get(name) or {} + ceiling = entry.get("ceiling") + if not ceiling: + continue + spent = render(entry.get("spent", 0)) + parts.append(f"{name.split('_')[0]} {spent}/{render(ceiling)}") + return " · ".join(parts) or "no ceilings" + + def header_model(*, agent_state: str = "idle", step: int | None = None) -> dict[str, Any]: """The workspace title bar: project, agent state, session quota strip. @@ -197,6 +257,8 @@ def status_model() -> dict[str, Any]: """ from core import ledger_store as ls + from ui import tasks as tasks_mod + kernel, _ = _safe(_kernel_state, "no kernel") runs, runs_error = _safe(ls.runs, []) uncollected = [r for r in (runs or []) if not r.collected] @@ -205,6 +267,10 @@ def status_model() -> dict[str, Any]: "kernel": kernel, "queued": len(uncollected), "gpu": len([r for r in uncollected if not r.is_smoke]), + # Local subprocesses, counted apart from the remote runs beside them. + # A wiki rebuild and a GPU job are both "running" and are not remotely + # the same fact -- one is this machine's CPU, the other is money. + "tasks": len(tasks_mod.running()), "error": runs_error, } @@ -218,6 +284,169 @@ def _kernel_state() -> str: return "lab stopped" +# --------------------------------------------------------------------------- +# 0. chat sessions +# --------------------------------------------------------------------------- +def sessions_model(current: str | None = None) -> dict[str, Any]: + """The stored conversations, and which one is open. + + Read here rather than in the window for the reason every other window reads + a model: a file read belongs on this side of the line, and it makes the + picker testable without a browser. `chat` has no entry in `MODEL_BUILDERS` + -- its state is the live session, not a file, and the poll must not redraw + it -- so this is called directly, the way `workspaces_model` is. + """ + from ui import sessions as sessions_mod + + listed, error = _safe(sessions_mod.listing, []) + rows = list(listed or []) + for row in rows: + # Held by *another* window. Opening it there too would put two writers + # on one file, so the picker says so rather than letting the refusal + # arrive as a surprise on click. + held = sessions_mod.holder(row["id"]) + row["held_elsewhere"] = bool(held) and row["id"] != current + return { + "rows": rows, + "current": current, + "count": len(rows), + # Reopening a session whose SDK id was never recorded shows the + # transcript without continuing the conversation. Counted so the window + # can say so rather than let it be discovered. + "transcript_only": len([r for r in rows if not r["resumable"]]), + "error": error, + } + + +# --------------------------------------------------------------------------- +# 0a. credentials +# --------------------------------------------------------------------------- +#: What each credential unlocks, and whether the system works without it. The +#: text matters as much as the flag: "missing" is not the same fact for a token +#: that gates GPU submission as for one that raises a rate limit. +CREDENTIAL_NOTES: dict[str, tuple[str, bool]] = { + "hf_token": ("Hugging Face Jobs — submitting and collecting runs", True), + "openrouter_key": ("the reranker, funnel stage 2 (costs credits)", False), + "voyage_key": ("embeddings for the local index (costs credits)", False), + "asta_api_key": ("raises Asta's rate limits; discovery works without it", False), + "s2_api_key": ("Semantic Scholar direct — only issued to institutional addresses", False), + "context7_key": ("raises Context7's rate limits; lookups work without it", False), + "claude_oauth_token": ("the funnel's Haiku stages, when the agent runs them", True), +} + + +def credentials_model() -> dict[str, Any]: + """Which credentials are stored. Values are never read, let alone returned. + + The point of the panel this feeds is that storing a credential was the one + thing the workspace could not do: `jobs.py credential set` prompts with + `getpass`, which needs a terminal, so the four commands in the README's + install section were the reason to keep a shell open beside the app. + """ + from core import credentials as credentials_mod + + present, error = _safe(credentials_mod.status, {}) + rows = [] + for name, stored in (present or {}).items(): + purpose, required = CREDENTIAL_NOTES.get(name, ("", False)) + rows.append( + { + "name": name, + "stored": bool(stored), + "purpose": purpose, + "required": required, + "tone": "ok" if stored else ("broken" if required else "neutral"), + "state": "STORED" if stored else ("MISSING" if required else "not set"), + } + ) + return { + "rows": rows, + "missing_required": [r["name"] for r in rows if r["required"] and not r["stored"]], + "error": error, + # The store itself, so "nothing is stored" and "nothing can be stored" + # are distinguishable on screen. + "service": getattr(credentials_mod, "SERVICE", "grad"), + } + + +# --------------------------------------------------------------------------- +# 0b. background tasks +# --------------------------------------------------------------------------- +def tasks_model() -> dict[str, Any]: + """Local commands the workspace started, newest first. + + Deliberately not merged into `queue_model`. Both lists hold things that are + "running", and that is where the resemblance stops: a wiki rebuild is this + machine's CPU for two minutes, a GPU job is money against a ceiling that a + gate refuses at. The queue window's own docstring makes the same argument in + the other direction about campaign candidates, and one table showing both + would make each one harder to read for no gain. + + The output tail is included whole. It is bounded at the source + (`tasks.TAIL_LINES`), and the poll's fingerprint is what turns "a line + arrived" into a redraw -- so a task that is quiet costs one comparison. + """ + from ui import tasks as tasks_mod + + rows = [] + for task in tasks_mod.all_tasks(): + rows.append( + { + "id": task.id, + "label": task.label, + "command": "python -m " + " ".join(task.argv), + "state": task.state, + "tone": tasks_mod.STATE_TONE.get(task.state, "neutral"), + "running": task.running, + "elapsed": _duration(task.elapsed), + "exit_code": task.exit_code, + "stoppable": task.running, + # Named so the button can say what stopping will actually do: + # asking the tool, or signalling it. See `tasks.cancel`. + "halt": ("python -m " + " ".join(task.halt)) if task.halt else None, + "message": tasks_mod.task_message(task), + "tail": _tail_runs(task.tail), + "dropped": task.dropped, + } + ) + return { + "rows": rows, + "running": len([r for r in rows if r["running"]]), + "finished": len([r for r in rows if not r["running"]]), + "empty_fix": ( + "nothing has been started from the workspace yet — VERIFY, RE-CHECK, " + "REBUILD and BUILD PDF all run here" + ), + } + + +def _tail_runs(tail: Iterable[tuple[str, str]]) -> list[tuple[str, str]]: + """Consecutive lines of the same kind, joined into one block. + + A tail is up to `tasks.TAIL_LINES` long, and a `
` per line would be 400
+    elements per task rebuilt on every poll that moved one of them. Runs collapse
+    that to two or three in practice, while keeping stderr distinguishable and --
+    the part a naive "all stdout, then all stderr" split would lose -- keeping
+    every line in the order the command emitted it.
+    """
+    runs: list[tuple[str, list[str]]] = []
+    for tag, line in tail:
+        if runs and runs[-1][0] == tag:
+            runs[-1][1].append(line)
+        else:
+            runs.append((tag, [line]))
+    return [(tag, "\n".join(lines)) for tag, lines in runs]
+
+
+def _duration(seconds: float) -> str:
+    seconds = max(0.0, float(seconds))
+    if seconds < 60:
+        return f"{seconds:.0f}s"
+    if seconds < 3600:
+        return f"{int(seconds // 60)}m {int(seconds % 60):02d}s"
+    return f"{int(seconds // 3600)}h {int((seconds % 3600) // 60):02d}m"
+
+
 # ---------------------------------------------------------------------------
 # 1. notebook
 # ---------------------------------------------------------------------------
diff --git a/ui/registry.py b/ui/registry.py
index 6fd707a..9ff1ef8 100644
--- a/ui/registry.py
+++ b/ui/registry.py
@@ -1,9 +1,9 @@
 """The window registry: the one list the whole shell is derived from.
 
-The opener strip, the layout presets, the `⌘K` palette, the persisted layout's
-validation and the status bar's count all read this tuple. Adding a twelfth
-window is adding one `WindowSpec` and one module -- if it is ever more than
-that, something has grown a second list and the two will drift.
+The `⋯` menu, the persisted layout's validation and the status bar's count all
+read this tuple. Adding a thirteenth window is adding one `WindowSpec` and one
+module -- if it is ever more than that, something has grown a second list and
+the two will drift.
 
 `module` is resolved with `importlib` at first render rather than imported here,
 for the reason the rest of the app imports lazily: `ui.registry` has to stay
@@ -25,6 +25,8 @@ class WindowSpec:
     id: str
     name: str
     module: str
+    #: One line saying what the window is for. It is the menu row's caption and
+    #: its tooltip, and the title bar's subtitle when the window defines none.
     hint: str
     #: In the arrangement a fresh workspace opens with. The mock's opening
     #: state: chat and notebook side by side, ledger over quota on the right.
@@ -46,6 +48,7 @@ class WindowSpec:
     WindowSpec("preflight", "preflight", "ui.windows.preflight", "the checklist that blocks a submission"),
     WindowSpec("funnel", "funnel", "ui.windows.funnel", "retrieval, stage by stage"),
     WindowSpec("queue", "queue", "ui.windows.queue", "runs and GPU jobs"),
+    WindowSpec("tasks", "tasks", "ui.windows.tasks", "commands running on this machine"),
 )
 
 BY_ID: dict[str, WindowSpec] = {w.id: w for w in WINDOWS}
diff --git a/ui/sessions.py b/ui/sessions.py
new file mode 100644
index 0000000..870ccb9
--- /dev/null
+++ b/ui/sessions.py
@@ -0,0 +1,278 @@
+"""Named chat sessions: starting a new one, and coming back to an old one.
+
+There was one conversation per client key, in one file, forever. Everything the
+agent had ever been asked was in it, and the only way to start clean was to
+delete the file -- which took the record with it. That is the wrong trade for
+this project in particular: a session is where the reasoning behind an
+expectation lives, and the ledger entry it produced points back at nothing.
+
+**A session is a file, and the file is the record.** No index, no database. The
+id is the filename, so listing is a glob and nothing can disagree with anything.
+A session's first line is a `meta` record and the rest are transcript records;
+`app.Session.restore` already ignores any line whose `role` is not a real role,
+so the meta line costs nothing there and the format stays one thing.
+
+**The legacy transcript is already a session.** It was `ui_session-default.jsonl`
+and the scheme here is `ui_session-.jsonl`, so the file that exists on an
+upgraded machine is a session called `default` with no migration step at all.
+Its title is derived from its first user message, which is what a session that
+was never named should be called anyway.
+
+**Two ids, and they are not interchangeable.** Ours names the file. The SDK's --
+recorded here when a turn reports one -- is what `ClaudeAgentOptions.resume`
+takes, and it is what makes reopening a session continue the *conversation*
+rather than merely redisplay it. A session whose SDK id is unknown (an older
+transcript, a session whose first turn failed) still opens: the transcript is
+shown and the next turn starts a fresh conversation under the same file. That
+degradation is deliberate and is reported, because "the agent remembers this"
+and "you can read this" are different promises.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import secrets
+from pathlib import Path
+from typing import Any
+
+from core import paths
+from core.ledger_store import now_iso
+
+PREFIX = "ui_session"
+#: The id of the conversation that existed before sessions did.
+LEGACY_ID = "default"
+#: What an id may contain. Ids reach a filename, and one of them comes from a
+#: file already on disk rather than from `new_id`.
+ID_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}")
+#: How much of the first user message becomes the fallback title.
+TITLE_CHARS = 60
+
+
+def sessions_dir() -> Path:
+    return paths.data_dir()
+
+
+def new_id() -> str:
+    """Sortable, so a glob comes back in a sensible order before anything reads
+    an mtime, plus four hex digits because two sessions in one second is a
+    double-click rather than an impossibility."""
+    stamp = re.sub(r"[^0-9]", "", now_iso())[:14]
+    return f"{stamp}-{secrets.token_hex(2)}"
+
+
+def is_id(value: Any) -> bool:
+    return isinstance(value, str) and bool(ID_RE.fullmatch(value))
+
+
+def path_for(session_id: str) -> Path:
+    """The file behind an id, refusing anything that is not one.
+
+    The id reaches a filename and can come off disk or out of a click handler,
+    so it is validated here rather than trusted -- the same reason
+    `state.layout_path` sanitises a project id.
+    """
+    if not is_id(session_id):
+        raise ValueError(f"not a session id: {session_id!r}")
+    return sessions_dir() / f"{PREFIX}-{session_id}.jsonl"
+
+
+def id_of(path: Path) -> str:
+    return path.name[len(PREFIX) + 1 : -len(".jsonl")]
+
+
+def read_meta(path: Path) -> dict[str, Any]:
+    """The header record, the title, and how many messages are in the file.
+
+    Stops at the header when the header carries everything -- which it does for
+    any session this version wrote, because `write` records the count alongside
+    the title. That matters because `listing()` calls this once per session: a
+    scan-to-the-end here would make listing cost the total size of every
+    transcript in the workspace, and transcripts grow with the conversations
+    that are most worth coming back to.
+
+    The full scan is still the fallback, for the one case that needs it: a file
+    written before this format existed has no header, so its title is its first
+    user message and its count is its lines. That is one file, once.
+    """
+    meta: dict[str, Any] = {}
+    title = ""
+    messages = 0
+    if not path.exists():
+        return meta
+    try:
+        with open(path, encoding="utf-8") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                try:
+                    record = json.loads(line)
+                except json.JSONDecodeError:
+                    continue
+                if not isinstance(record, dict):
+                    continue
+                if record.get("type") == "meta":
+                    meta = {k: v for k, v in record.items() if k != "type"}
+                    if meta.get("title") and isinstance(meta.get("messages"), int):
+                        return meta
+                    continue
+                messages += 1
+                if not title and record.get("role") == "user":
+                    title = str(record.get("text") or "").strip()
+    except OSError:
+        return meta
+    meta.setdefault("title", "")
+    if not meta["title"]:
+        meta["title"] = title_from(title) or "empty session"
+    meta["messages"] = messages
+    return meta
+
+
+def title_from(text: str) -> str:
+    """A session's name, taken from the first thing asked in it."""
+    flattened = " ".join(text.split())
+    if len(flattened) <= TITLE_CHARS:
+        return flattened
+    return flattened[: TITLE_CHARS - 1] + "…"
+
+
+def listing() -> list[dict[str, Any]]:
+    """Every session, most recently written first.
+
+    Sorted on mtime rather than on the id, because `default` predates the
+    timestamped scheme and because resuming a session is the thing that makes it
+    recent. A file that cannot be read is skipped rather than raised on -- one
+    damaged transcript must not make the picker unopenable.
+    """
+    out: list[dict[str, Any]] = []
+    directory = sessions_dir()
+    if not directory.exists():
+        return out
+    for path in directory.glob(f"{PREFIX}-*.jsonl"):
+        session_id = id_of(path)
+        if not is_id(session_id):
+            continue
+        meta = read_meta(path)
+        try:
+            modified = path.stat().st_mtime
+        except OSError:
+            modified = 0.0
+        out.append(
+            {
+                "id": session_id,
+                "title": meta.get("title") or "empty session",
+                "created_at": meta.get("created_at"),
+                "sdk_session_id": meta.get("sdk_session_id"),
+                # Whether reopening continues the conversation or only shows it.
+                "resumable": bool(meta.get("sdk_session_id")),
+                "messages": int(meta.get("messages") or 0),
+                "modified": modified,
+            }
+        )
+    out.sort(key=lambda s: s["modified"], reverse=True)
+    return out
+
+
+def delete(session_id: str) -> bool:
+    path = path_for(session_id)
+    try:
+        path.unlink()
+        return True
+    except FileNotFoundError:
+        return False
+    except OSError:
+        return False
+
+
+def write(
+    session_id: str,
+    records: list[dict[str, Any]],
+    *,
+    title: str = "",
+    created_at: str | None = None,
+    sdk_session_id: str | None = None,
+) -> None:
+    """The whole session: one meta line, then the transcript.
+
+    Rewritten wholesale rather than appended to, because that is what the
+    transcript already did and because a session is small. It is *not* routed
+    through `core/jsonl.py`: that module's contract is the append-only ledgers,
+    which are multi-writer and must never lose a line. A transcript has exactly
+    one writer -- the client that owns the session -- and is replaced in full.
+    """
+    path = path_for(session_id)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    meta = {
+        "type": "meta",
+        "id": session_id,
+        "title": title,
+        "created_at": created_at or now_iso(),
+        "sdk_session_id": sdk_session_id,
+        # Written here so `read_meta` can answer from the header alone. The
+        # count is free at this point -- the records are in hand -- and it is
+        # what keeps listing a workspace from costing a read of every transcript
+        # in it.
+        "messages": len(records),
+    }
+    lines = [json.dumps(meta, ensure_ascii=False)]
+    lines += [json.dumps(record, ensure_ascii=False) for record in records]
+    path.write_text("\n".join(lines), encoding="utf-8")
+
+
+# ---------------------------------------------------------------------------
+# who has which session open
+# ---------------------------------------------------------------------------
+#: Sessions held by a live client, in this process, keyed by **path**. A session
+#: is written wholesale by the client that owns it, so two clients on one
+#: session is two writers on one file and the later `_persist` silently discards
+#: the other's turns. `app.Session` was always one-per-client for exactly this
+#: reason; the claim keeps that true now that the *file* is chosen rather than
+#: derived from the client's own key.
+#:
+#: By path rather than by id because an id is only unique within a workspace:
+#: every root has a `default`, and one client switching roots would otherwise
+#: find another client's claim on a file it has never seen.
+_claimed: dict[str, str] = {}
+
+
+def _key(session_id: str) -> str:
+    return str(path_for(session_id))
+
+
+def claim(session_id: str, owner: str) -> bool:
+    """Take a session for one client. False if another client already has it."""
+    key = _key(session_id)
+    held = _claimed.get(key)
+    if held is not None and held != owner:
+        return False
+    _claimed[key] = owner
+    return True
+
+
+def release(owner: str) -> None:
+    for key in [k for k, held in _claimed.items() if held == owner]:
+        del _claimed[key]
+
+
+def holder(session_id: str) -> str | None:
+    return _claimed.get(_key(session_id))
+
+
+def reset_claims() -> None:
+    """Drop every claim. For tests -- module state outlives a fixture."""
+    _claimed.clear()
+
+
+def most_recent(owner: str | None = None) -> str | None:
+    """The session to open on a cold start, or None for a new workspace.
+
+    With an `owner`, the most recent one *nobody else is in*. A second window
+    opening the first one's conversation would not merely show it twice: both
+    would write the whole file on every turn, and the loser's turns would
+    disappear with nothing to say they had.
+    """
+    for row in listing():
+        if owner is None or claim(row["id"], owner):
+            return row["id"]
+    return None
diff --git a/ui/shell.py b/ui/shell.py
index 42219cf..2b3920b 100644
--- a/ui/shell.py
+++ b/ui/shell.py
@@ -1,4 +1,4 @@
-"""The workspace shell: title bar, opener strip, tiling area, status bar.
+"""The workspace shell: title bar, tiling area, status bar.
 
 The one thing worth understanding before changing anything here is how a window
 survives a retile.
@@ -36,7 +36,6 @@ def build(workspace: Workspace) -> None:
     with kit.el("div", "grad-app"):
         with kit.el("div", "grad-shell"):
             appbar = kit.el("div", "grad-appbar")
-            opener = kit.el("div", "grad-opener")
             tiles = kit.el("div", "grad-tiles")
             statusbar = kit.el("div", "grad-statusbar")
         # Detached window roots wait here between tilings. `display: none`
@@ -44,17 +43,13 @@ def build(workspace: Workspace) -> None:
         # window's state with it.
         attic = kit.el("div", "", style="display: none")
 
-    palette = _command_palette(ui, workspace)
+    windows = _windows_menu(ui, workspace)
+    projects = _project_menu(ui, workspace)
 
     def draw_appbar() -> None:
         appbar.clear()
         with appbar:
-            _appbar(workspace, palette)
-
-    def draw_opener() -> None:
-        opener.clear()
-        with opener:
-            _opener(workspace)
+            _appbar(workspace, windows, projects)
 
     def draw_status() -> None:
         statusbar.clear()
@@ -103,25 +98,23 @@ def retile() -> None:
                             _frame(workspace, slot.window, roots, bars, attic, draw_window)
 
     workspace.bind_chrome(draw_appbar)
-    workspace.bind_chrome(draw_opener)
     workspace.bind_chrome(draw_status)
     workspace.bind_retile(retile)
 
     draw_appbar()
-    draw_opener()
     draw_status()
     retile()
 
     # One poll for the whole workspace; see the note at the top of ui/state.py.
     ui.timer(POLL_SECONDS, workspace.tick)
 
-    _bind_client_events(ui, workspace)
+    _bind_client_events(ui, workspace, windows)
 
 
 # ---------------------------------------------------------------------------
 # chrome
 # ---------------------------------------------------------------------------
-def _appbar(workspace: Workspace, palette: Any) -> None:
+def _appbar(workspace: Workspace, windows: Any, projects: Any) -> None:
     header = workspace.header()
     session = header["session"]
 
@@ -131,7 +124,13 @@ def _appbar(workspace: Workspace, palette: Any) -> None:
 
     with kit.el("div", "grad-appbar-cell"):
         kit.text("project", "dim")
-        kit.text(header["project"], "", style="font-weight: 700")
+        kit.button(
+            f"{header['project']} ▾",
+            tone="ghost",
+            classes="grad-appbar-btn",
+            title="switch project, or open another workspace folder",
+            on_click=projects.open,
+        )
 
     with kit.el("div", "grad-appbar-cell"):
         state = header["agent_state"]
@@ -164,13 +163,17 @@ def _appbar(workspace: Workspace, palette: Any) -> None:
         kit.text(f"· resets {session['resets_in']}", "dim")
 
     with kit.el("div", "grad-appbar-cell right"):
-        kit.button("⌘K", tone="ghost", classes="grad-appbar-btn", on_click=palette.open)
+        # One control, not three. There used to be an always-visible strip of
+        # eleven window names, a `⌘K` palette that listed the same eleven, and a
+        # `LAYOUTS ▾` button whose caret promised a menu it did not have. All
+        # three were derived from `registry.WINDOWS`; this is the one that is
+        # left, and `⌘K` still opens it.
         kit.button(
-            "LAYOUTS ▾",
+            "⋯",
             tone="ghost",
-            classes="grad-appbar-btn",
-            title="tile ⌥1 · stack ⌥2 · full ⌥3",
-            on_click=lambda: workspace.preset("tile"),
+            classes="grad-appbar-btn grad-dots",
+            title="windows and arrangement (⌘K)",
+            on_click=windows.open,
         )
 
 
@@ -181,45 +184,393 @@ def _used_share(session: dict[str, Any]) -> float:
     return max(0.0, min(1.0, float(session.get("used_usd", 0.0)) / float(ceiling)))
 
 
-def _opener(workspace: Workspace) -> None:
-    kit.text("OPEN A WINDOW →", "grad-opener-hint")
-    open_ids = set(workspace.layout.windows)
-    for window in registry.WINDOWS:
-        is_open = window.id in open_ids
-        cell = kit.text(window.name, f"grad-opener-cell {'open' if is_open else ''}".strip(), tag="button")
-        cell.props(f'title="{kit.escape(window.hint)}"')
-        cell.on("click", lambda _=None, wid=window.id: workspace.toggle(wid))
-    kit.spacer()
-    kit.text("tile ⌥1 · stack ⌥2 · full ⌥3", "grad-opener-hint")
-
-
 def _statusbar(workspace: Workspace) -> None:
     status = workspace.status()
     kit.text(status["cwd"], "dim", tag="span")
     kit.text(status["kernel"], "", tag="span")
     kit.text(f"queue {status['queued']} · gpu {status['gpu']}", "", tag="span")
+    if status.get("tasks"):
+        # Only when something is actually running: a permanent "tasks 0" is
+        # noise in a bar that is read at a glance.
+        kit.text(f"tasks {status['tasks']}", "count", tag="span")
     if workspace.notice:
         kit.text(workspace.notice, "", tag="span")
     kit.spacer()
-    kit.text("⌥drag to retile", "dim", tag="span")
+    kit.text("drag a title bar to move · drop on another to swap", "dim", tag="span")
     kit.text(f"{len(workspace.layout.windows)} open", "count", tag="span")
 
 
-def _command_palette(ui: Any, workspace: Workspace) -> Any:
-    """`⌘K`: open a window by name. The opener strip, without the mouse."""
+class _Menu:
+    """A dialog whose body is rebuilt each time it opens.
+
+    `ui.dialog` builds its contents once. These menus list projects, folders and
+    open windows, and all three change *because of* what the dialog does --
+    create a project and the list it was read from is already stale, open a
+    window and the mark beside its name is wrong. Redrawing on open is cheaper
+    than binding every row to the poll, and it cannot go stale between the click
+    and the dialog appearing.
+
+    `draw` is handed the menu so a control *inside* it can call `redraw` after
+    changing what the menu is listing -- which is what lets the window menu stay
+    open across several toggles instead of closing after each one.
+    """
+
+    def __init__(self, dialog: Any, draw: Any) -> None:
+        self._dialog = dialog
+        self._draw = draw
+
+    def open(self) -> None:
+        self.redraw()
+        self._dialog.open()
+
+    def redraw(self) -> None:
+        self._draw(self)
+
+    def close(self) -> None:
+        self._dialog.close()
+
+
+def _project_menu(ui: Any, workspace: Workspace) -> _Menu:
+    """The workspace menu: which folder, which project, and how to change both."""
     with ui.dialog() as dialog, kit.el("div", "grad-app"):
-        with kit.el("div", "grad-card", style="background: var(--grad-paper); min-width: 420px"):
-            kit.text("OPEN A WINDOW", "head ink")
-            with kit.el("div", "body"):
-                for window in registry.WINDOWS:
-                    with kit.row("grad-row"):
+        body = kit.el(
+            "div", "grad-card", style="background: var(--grad-paper); min-width: 540px"
+        )
+
+    return _Menu(dialog, lambda menu: _draw_project_menu(ui, workspace, body, menu))
+
+
+def _draw_project_menu(ui: Any, workspace: Workspace, body: Any, menu: Any) -> None:
+    model = workspace.workspaces()
+    body.clear()
+
+    def act(coro: Any, what: str) -> None:
+        """Close first, then run: `reload` redraws the shell underneath, and a
+        dialog still open over it would be showing the workspace it just left."""
+        menu.close()
+        workspace.spawn(coro, what)
+
+    with body:
+        kit.text("WORKSPACE", "head ink")
+        with kit.el("div", "body"):
+            kit.error_strip(model.get("error"))
+            kit.kv([("folder", model["root"]), ("chosen by", model["source"])])
+
+            with kit.row("", gap=6).style("margin-top: 10px"):
+                folder = (
+                    ui.input(placeholder="path to another workspace folder")
+                    .props("borderless dense")
+                    .classes("field")
+                    .style("flex: 1 1 auto; padding: 0 8px")
+                )
+                kit.button(
+                    "BROWSE…",
+                    tone="neutral",
+                    title="pick a folder (needs the desktop window)",
+                    on_click=lambda: workspace.spawn(_browse(workspace, folder), "folder picker"),
+                )
+                kit.button(
+                    "OPEN",
+                    tone="primary",
+                    title="switch this app to that folder",
+                    on_click=lambda: act(
+                        workspace.switch_root(folder.value or "", create=True), "workspace switch"
+                    ),
+                )
+            kit.text(
+                "a folder that does not exist yet is created; the agent's tools follow it",
+                "grad-caption",
+            )
+
+            if model["recent"]:
+                kit.text("RECENT", "grad-caption").style("margin-top: 12px")
+                for path in model["recent"]:
+                    with kit.row("grad-row", gap=6):
                         kit.button(
-                            window.name.upper(),
+                            "OPEN",
                             tone="neutral",
-                            on_click=lambda _=None, wid=window.id: (workspace.open(wid), dialog.close()),
+                            on_click=lambda _=None, p=path: act(
+                                workspace.switch_root(p), "workspace switch"
+                            ),
                         )
-                        kit.text(window.hint, "grad-caption")
-    return dialog
+                        kit.text(path, "grad-caption")
+
+            # -- projects ---------------------------------------------------
+            kit.text("PROJECTS IN THIS FOLDER", "grad-caption").style("margin-top: 16px")
+            if not model["projects"]:
+                kit.text("none yet — the first one is created below", "grad-empty")
+            for project in model["projects"]:
+                with kit.row("grad-row", gap=6):
+                    kit.button(
+                        "IN USE" if project["current"] else "USE",
+                        tone="active" if project["current"] else "neutral",
+                        disabled=project["current"] or project["status"] == "closed",
+                        on_click=lambda _=None, pid=project["id"]: act(
+                            workspace.use_project(pid), "project switch"
+                        ),
+                    )
+                    kit.text(project["id"], "", style="font-weight: 700")
+                    kit.text(project["title"], "grad-caption")
+                    kit.spacer()
+                    if project["status"] == "closed":
+                        kit.chip("CLOSED", "neutral")
+                    kit.text(project["spend"], "grad-caption")
+
+            # -- a new one --------------------------------------------------
+            kit.text("NEW PROJECT", "grad-caption").style("margin-top: 16px")
+            with kit.row("", gap=6):
+                project_id = (
+                    ui.input(placeholder="id, e.g. proj-scaling-w2")
+                    .props("borderless dense")
+                    .classes("field")
+                    .style("flex: 0 0 220px; padding: 0 8px")
+                )
+                title = (
+                    ui.input(placeholder="what this research is")
+                    .props("borderless dense")
+                    .classes("field")
+                    .style("flex: 1 1 auto; padding: 0 8px")
+                )
+                kit.button(
+                    "CREATE",
+                    tone="primary",
+                    on_click=lambda: act(
+                        workspace.create_project(project_id.value or "", title.value or ""),
+                        "project create",
+                    ),
+                )
+            kit.text(
+                "created with no ceilings — set them below once it is selected",
+                "grad-caption",
+            )
+
+            _ceilings(ui, workspace, model, menu)
+            _credentials(ui, workspace, menu)
+
+
+#: The three ceilings a project carries, and the unit each is counted in.
+#: `tools.budget raise` takes one flag per resource; this is that list, in the
+#: order the quota window draws them.
+CEILINGS = (
+    ("gpu-usd", "GPU $", "dollars of remote compute"),
+    ("quota-tokens", "tokens", "subscription tokens, all roles"),
+    ("credits-usd", "credits $", "reranker and embeddings"),
+)
+
+
+def _ceilings(ui: Any, workspace: Workspace, model: dict[str, Any], menu: _Menu) -> None:
+    """Move a ceiling on the selected project.
+
+    A logged event, not a setting: `budget raise` appends to the ledger, so the
+    history of what was raised and when survives. The UI runs the same command
+    for the same reason every other button does.
+    """
+    current = next((p for p in model["projects"] if p["current"]), None)
+    if current is None:
+        return
+
+    kit.text("CEILINGS", "grad-caption").style("margin-top: 16px")
+    fields: dict[str, Any] = {}
+    with kit.row("", gap=6):
+        for flag, caption, hint in CEILINGS:
+            field = (
+                ui.input(placeholder=caption)
+                .props("borderless dense")
+                .classes("field")
+                .style("flex: 1 1 0; padding: 0 8px")
+            )
+            field.props(f'title="{kit.attr(hint)}"')
+            fields[flag] = field
+
+        def raise_them() -> None:
+            argv = ["tools.budget", "raise", current["id"]]
+            for flag, field in fields.items():
+                if (field.value or "").strip():
+                    argv += [f"--{flag}", str(field.value).strip()]
+            if len(argv) == 3:
+                workspace.say("no ceiling given — fill one of the three fields")
+                return
+            menu.close()
+            workspace.spawn(workspace.run_and_reload(*argv, "--json"), "ceiling raise")
+
+        kit.button("RAISE", tone="primary", on_click=raise_them)
+    kit.text(
+        f"a logged event on {current['id']} — leave a field blank to leave that ceiling alone",
+        "grad-caption",
+    )
+
+
+def _credentials(ui: Any, workspace: Workspace, menu: _Menu) -> None:
+    """Store the credentials the README's install section lists.
+
+    This is the one thing the workspace genuinely could not do: `credential set`
+    prompts with `getpass`, which needs a terminal, so a fresh machine needed a
+    shell open beside the app to become usable. The value goes down a pipe
+    rather than in an argument -- see `Workspace.set_credential`.
+
+    Values are never shown, and there is nothing here that could show one: the
+    CLI does not print them and `credentials.status()` returns booleans.
+    """
+    model = workspace.credentials()
+    kit.text("CREDENTIALS", "grad-caption").style("margin-top: 16px")
+    kit.error_strip(model.get("error"))
+
+    for row in model["rows"]:
+        with kit.row("grad-row", gap=6):
+            kit.chip(row["state"], row["tone"])
+            kit.text(row["name"], "grad-mono", tag="span")
+            kit.text(row["purpose"], "grad-caption", tag="span")
+            kit.spacer()
+            value = (
+                ui.input(placeholder="paste to set")
+                .props("borderless dense type=password")
+                .classes("field")
+                .style("flex: 0 0 200px; padding: 0 8px")
+            )
+
+            def store(_=None, name=row["name"], field=value) -> None:
+                pasted, field.value = field.value or "", ""
+                workspace.spawn(workspace.set_credential(name, pasted), "credential set")
+                menu.redraw()
+
+            def forget(_=None, name=row["name"]) -> None:
+                workspace.spawn(workspace.delete_credential(name), "credential delete")
+                menu.redraw()
+
+            kit.button("SET", tone="neutral", on_click=store)
+            kit.button("✕", tone="neutral", disabled=not row["stored"], title="forget it",
+                       on_click=forget)
+
+    kit.text(
+        "stored in Windows Credential Manager, never in the workspace and never in the "
+        "agent's environment — they are fetched at the moment of use",
+        "grad-caption",
+    )
+
+
+def folder_dialog_type() -> int:
+    """`dialog_type` for "pick a folder", as something that survives pickling.
+
+    Native mode runs pywebview in a **separate process** and marshals this call
+    over a `multiprocessing` queue, so every argument has to pickle. The obvious
+    constant does not: `webview.FOLDER_DIALOG` is a deprecated `proxy_tools.Proxy`
+    whose repr is `20` but whose type is a proxy around a function, and pickling
+    it fails with *"Can't pickle : it's not the same
+    object as webview.FOLDER_DIALOG"*.
+
+    That failure is also **uncatchable from here**: it is raised in the queue's
+    own feeder thread, so it prints a traceback and leaves the awaited call
+    hanging rather than raising where `_browse` could handle it. Sending a value
+    that pickles is the only real fix, which is why this returns a plain `int`
+    rather than the `FileDialog.FOLDER` enum member -- `create_file_dialog`
+    declares the parameter as `int` and passes it straight through, so the
+    narrowest thing that can cross a process boundary is the right one to send.
+    """
+    try:
+        import webview  # noqa: PLC0415
+
+        return int(webview.FileDialog.FOLDER)
+    except (ImportError, AttributeError):
+        # Older pywebview, where the constant existed only as the proxy above --
+        # whose value was this same 20.
+        return 20
+
+
+async def _browse(workspace: Workspace, field: Any) -> None:
+    """The native folder picker, when there is a native window to hang it on.
+
+    Only `ui.run(native=True)` has one; the documented browser fallback does
+    not, and neither does a second tab. So this fills the text field rather than
+    switching directly -- the typed path is the mechanism, and the picker is a
+    convenience on top of it that is allowed to be unavailable.
+    """
+    import logging  # noqa: PLC0415
+
+    try:
+        from nicegui import app as nicegui_app  # noqa: PLC0415
+
+        window = getattr(getattr(nicegui_app, "native", None), "main_window", None)
+        if window is None:
+            raise RuntimeError("no native window")
+        chosen = await window.create_file_dialog(dialog_type=folder_dialog_type())
+    except Exception as exc:  # noqa: BLE001 - an unavailable picker is not an error
+        logging.getLogger("grad.ui").debug("folder picker unavailable", exc_info=exc)
+        workspace.say("no folder picker here — type the path instead")
+        return
+    if chosen:
+        field.value = chosen[0] if isinstance(chosen, (list, tuple)) else str(chosen)
+
+
+#: The arrangements `apply_preset` knows, with the chord the browser sends for
+#: each. Listed here rather than in `layout.py` because the caption and the
+#: shortcut are chrome; the moves themselves are the layout's.
+PRESET_ROWS = (
+    ("tile", "TILE", "⌥1", "a column each, up to three"),
+    ("stack", "STACK", "⌥2", "one column, everything stacked"),
+    ("full", "FULL", "⌥3", "the focused window, the rest at the edge"),
+)
+
+
+def _windows_menu(ui: Any, workspace: Workspace) -> _Menu:
+    """`⋯` and `⌘K`: which windows are open, and how they are arranged.
+
+    This is the only opener. It replaced a permanent strip of eleven names,
+    which cost 34px of vertical space to show a list that is read once a session
+    and a state -- open or closed -- that the mark beside each name carries just
+    as well.
+
+    It does not close on a toggle. Opening three windows is three clicks, and a
+    menu that dismissed itself after each one would be three trips back to the
+    same button; `menu.redraw()` re-reads the layout in place so the marks stay
+    honest without the dialog going away.
+    """
+    with ui.dialog() as dialog, kit.el("div", "grad-app"):
+        body = kit.el("div", "grad-card", style="background: var(--grad-paper); min-width: 460px")
+
+    return _Menu(dialog, lambda menu: _draw_windows_menu(workspace, body, menu))
+
+
+def _draw_windows_menu(workspace: Workspace, body: Any, menu: _Menu) -> None:
+    open_ids = set(workspace.layout.windows)
+    body.clear()
+
+    with body:
+        with kit.row("head ink", gap=9):
+            kit.text("WINDOWS", "", tag="span")
+            kit.spacer()
+            kit.text(f"{len(open_ids)} of {len(registry.WINDOWS)} open", "", tag="span")
+
+        with kit.el("div", "body"):
+            for window in registry.WINDOWS:
+                is_open = window.id in open_ids
+                row = kit.el("button", f"grad-menu-row {'open' if is_open else ''}".strip())
+                row.props(f'title="{kit.attr(window.hint)}"')
+                row.on(
+                    "click",
+                    lambda _=None, wid=window.id: (workspace.toggle(wid), menu.redraw()),
+                )
+                with row:
+                    # A filled square for open, an empty one for closed. The
+                    # opener strip said the same thing by inverting the whole
+                    # cell, which is louder than a list of eleven can carry.
+                    kit.text("■" if is_open else "□", "mark", tag="span")
+                    kit.text(window.name, "name", tag="span")
+                    kit.text(window.hint, "hint", tag="span")
+
+            kit.text("ARRANGEMENT", "grad-caption").style("margin-top: 14px")
+            for name, caption, chord, hint in PRESET_ROWS:
+                row = kit.el("button", "grad-menu-row")
+                row.props(f'title="{kit.attr(hint)}"')
+                row.on("click", lambda _=None, p=name: (workspace.preset(p), menu.close()))
+                with row:
+                    kit.text(chord, "mark", tag="span")
+                    kit.text(caption, "name", tag="span")
+                    kit.text(hint, "hint", tag="span")
+
+            kit.text(
+                "drag a title bar to move a window · drop it on another to swap them",
+                "grad-caption",
+            ).style("margin-top: 12px")
 
 
 # ---------------------------------------------------------------------------
@@ -312,8 +663,8 @@ def _on_close(ui: Any, window_id: str) -> None:
 # ---------------------------------------------------------------------------
 # events from the browser
 # ---------------------------------------------------------------------------
-def _bind_client_events(ui: Any, workspace: Workspace) -> None:
-    """The four gestures `tiling.js` sends back once they have settled."""
+def _bind_client_events(ui: Any, workspace: Workspace, windows: _Menu) -> None:
+    """The gestures `tiling.js` sends back once they have settled."""
 
     def on_resize(event: Any) -> None:
         data = getattr(event, "args", {}) or {}
@@ -328,8 +679,26 @@ def on_resize(event: Any) -> None:
     def on_retile(event: Any) -> None:
         data = getattr(event, "args", {}) or {}
         window_id = data.get("window")
-        if isinstance(window_id, str) and window_id in registry.BY_ID:
-            workspace.retile(window_id, int(data.get("column") or 0))
+        if not isinstance(window_id, str) or window_id not in registry.BY_ID:
+            return
+        slot = data.get("slot")
+        workspace.retile(
+            window_id,
+            int(data.get("column") or 0),
+            int(slot) if isinstance(slot, (int, float)) else None,
+            new_column=bool(data.get("new_column")),
+        )
+
+    def on_swap(event: Any) -> None:
+        """Both ids are checked against the registry before either is used: this
+        arrives from the browser, and `swap` writes whatever it is handed
+        straight into a slot."""
+        data = getattr(event, "args", {}) or {}
+        a, b = data.get("a"), data.get("b")
+        if not (isinstance(a, str) and isinstance(b, str)):
+            return
+        if a in registry.BY_ID and b in registry.BY_ID:
+            workspace.swap(a, b)
 
     def on_preset(event: Any) -> None:
         data = getattr(event, "args", {}) or {}
@@ -337,5 +706,9 @@ def on_preset(event: Any) -> None:
 
     ui.on("grad_resize", on_resize)
     ui.on("grad_retile", on_retile)
+    ui.on("grad_swap", on_swap)
     ui.on("grad_preset", on_preset)
-    ui.on("grad_palette", lambda _: None)
+    # `⌘K` opened nothing at all before: the browser emitted this and the
+    # handler was a no-op, because the palette it was meant to open was bound to
+    # its own button and never to the chord it advertised.
+    ui.on("grad_palette", lambda _: windows.open())
diff --git a/ui/state.py b/ui/state.py
index 735930a..1fddcdf 100644
--- a/ui/state.py
+++ b/ui/state.py
@@ -22,12 +22,12 @@
 import asyncio
 import json
 import logging
-import sys
 from pathlib import Path
 from typing import Any, Callable
 
 from core import jsonl, paths
 from ui import layout as layout_mod, models, registry
+from ui.tasks import envelope_message, run_tool
 
 log = logging.getLogger("grad.ui")
 
@@ -95,9 +95,21 @@ def save_layout(project: str | None, value: layout_mod.Layout) -> None:
     "preflight": lambda w: models.preflight_model(),
     "funnel": lambda w: models.funnel_model(w.selection.get("funnel.trace")),
     "queue": lambda w: models.queue_model(),
+    "tasks": lambda w: models.tasks_model(),
 }
 
 
+def current_project() -> str | None:
+    """The selected project, or None. Never raises: an unreadable project file
+    means an unnamed workspace, not an app that will not open."""
+    from core import budget as budget_mod  # noqa: PLC0415
+
+    try:
+        return budget_mod.current_project()
+    except Exception:  # noqa: BLE001 - see the docstring
+        return None
+
+
 def _fingerprint(value: Any) -> str:
     """Cheap change detection. Sorted keys so dict order cannot fake a change."""
     try:
@@ -212,8 +224,14 @@ def preset(self, name: str) -> None:
             return
         self._after_layout_change()
 
-    def retile(self, window_id: str, column: int) -> None:
-        self.layout.move(window_id, column)
+    def retile(
+        self, window_id: str, column: int, slot: int | None = None, *, new_column: bool = False
+    ) -> None:
+        self.layout.move(window_id, column, slot, new_column=new_column)
+        self._after_layout_change()
+
+    def swap(self, a: str, b: str) -> None:
+        self.layout.swap(a, b)
         self._after_layout_change()
 
     def resize(self, axis: str, fractions: list[float], *, column: int | None = None, total_px: int | None = None) -> None:
@@ -234,6 +252,139 @@ def _after_layout_change(self) -> None:
         for redraw in self._chrome:
             _guard(redraw, "chrome")
 
+    # -- the workspace itself -----------------------------------------------
+    def reload(self) -> None:
+        """Re-read everything derived from the root or the current project.
+
+        Both a folder switch and a project switch land here, because the same
+        things are stale either way: the layout is stored per project *under*
+        the root, and every window's model is a read of a file beneath it.
+
+        The windows are redrawn explicitly rather than left to the poll. A
+        retile reuses live roots -- that is what keeps a drag from wiping the
+        transcript -- so without this the panes would be rearranged for the new
+        workspace while still showing the old one's contents until something
+        happened to change a fingerprint.
+        """
+        self.project = current_project()
+        self.layout = load_layout(self.project)
+        self.models.clear()
+        self._fingerprints.clear()
+        self.selection.clear()
+        self.agent_state = "idle"
+        self.step = None
+        if self._retile is not None:
+            _guard(self._retile, "retile")
+        # A copy: `retile` unbinds windows the new layout does not have.
+        for window_id, redraw in list(self._redraw.items()):
+            _guard(redraw, window_id)
+        for redraw in self._chrome:
+            _guard(redraw, "chrome")
+
+    async def switch_root(self, folder: str, *, create: bool = False) -> None:
+        """Point the whole app at another workspace folder.
+
+        Three things have to move together, and missing any one of them leaves
+        the app half-switched in a way that is hard to see:
+
+        * **the paths**, via `GRAD_ROOT` -- which also carries to every CLI the
+          UI and the agent shell out to, since they inherit this environment;
+        * **the config cache**, because `config/grad.toml` moved with the root;
+        * **the session**, because its transcript file is derived from the root
+          and its SDK client's working directory was fixed when it was built.
+          A session left alone would keep the old workspace's conversation on
+          screen and keep running the agent's tools in the old directory.
+        """
+        from core import config as config_mod, workspace as workspace_mod  # noqa: PLC0415
+
+        try:
+            chosen = workspace_mod.select(folder, create=create)
+        except Exception as exc:  # noqa: BLE001 - a bad path is a message, not a crash
+            log.debug("workspace switch refused", exc_info=exc)
+            self.say(getattr(exc, "message", None) or str(exc))
+            return
+
+        paths.ensure_workspace()
+        config_mod._cache.clear()  # noqa: SLF001 - the config path moved with the root
+        rebind = getattr(self.session, "rebind", None)
+        if rebind is not None:
+            await rebind()
+        self.reload()
+        self.say(f"workspace: {chosen}")
+
+    async def create_project(self, project_id: str, title: str) -> None:
+        """Create a project and select it, by running the same command the agent
+        would (§10) -- so it lands in the same ledger and reads back the same."""
+        payload = await run_tool(
+            "tools.budget", "new", "--id", project_id, "--title", title, "--use", "--json"
+        )
+        self.say(envelope_message(payload))
+        if payload.get("ok"):
+            self.reload()
+
+    async def use_project(self, project_id: str) -> None:
+        payload = await run_tool("tools.budget", "use", project_id, "--json")
+        self.say(envelope_message(payload))
+        if payload.get("ok"):
+            self.reload()
+
+    async def run_and_reload(self, *argv: str) -> None:
+        """Run a CLI, report it, and re-read everything derived from it.
+
+        For the buttons that change what the *whole workspace* is looking at --
+        a ceiling moved, a project created -- as opposed to one window's data,
+        which `invalidate` covers more cheaply.
+        """
+        payload = await run_tool(*argv)
+        self.say(envelope_message(payload))
+        if payload.get("ok"):
+            self.reload()
+
+    def workspaces(self) -> dict[str, Any]:
+        return models.workspaces_model()
+
+    def credentials(self) -> dict[str, Any]:
+        return models.credentials_model()
+
+    def sessions(self) -> dict[str, Any]:
+        return models.sessions_model(getattr(self.session, "session_id", None))
+
+    def rebuild_chat(self) -> None:
+        """Redraw the chat window, which the poll deliberately never touches.
+
+        Its state is the live session rather than a file, so a redraw costs the
+        transcript's scroll position -- which is exactly why the poll leaves it
+        alone. Switching session replaces the transcript wholesale, so here that
+        cost is the entire point.
+        """
+        redraw = self._redraw.get("chat")
+        if redraw is not None:
+            _guard(redraw, "chat")
+        for redraw in self._chrome:
+            _guard(redraw, "chrome")
+
+    async def set_credential(self, name: str, value: str) -> None:
+        """Store one credential, down a pipe rather than as an argument.
+
+        The same command the README tells you to run, with `--stdin` instead of
+        the `getpass` prompt -- because the prompt needs a terminal, and needing
+        a terminal for this was the only thing that forced one open beside the
+        app on a fresh machine.
+        """
+        if not value.strip():
+            self.say("nothing to store — paste the token first")
+            return
+        payload = await run_tool(
+            "tools.jobs", "credential", "set", name, "--stdin", "--json", stdin=value
+        )
+        # `envelope_message` and nothing else: the CLI never prints a value, and
+        # neither does this, but the notice is worth being explicit about.
+        self.say(f"{name}: {envelope_message(payload)}")
+
+    async def delete_credential(self, name: str) -> None:
+        payload = await run_tool("tools.jobs", "credential", "delete", name, "--json")
+        self.say(f"{name}: {envelope_message(payload)}")
+
     # -- selections ---------------------------------------------------------
     def select(self, key: str, value: Any, *, window: str | None = None) -> None:
         self.selection[key] = value
@@ -309,55 +460,7 @@ def _guard(fn: Callable[[], None], what: str) -> None:
         log.exception("redraw of %s failed", what)
 
 
-# ---------------------------------------------------------------------------
-# running the CLIs the buttons are bound to
-# ---------------------------------------------------------------------------
-async def run_tool(*argv: str, timeout: float = 900.0) -> dict[str, Any]:
-    """Run one of Grad's own CLIs and parse its JSON envelope.
-
-    Every button in the UI that *does* something does it by shelling out to the
-    same command the agent would run, with `--json`. That is deliberate: it
-    keeps the UI free of logic (§10), and it means anything the UI can do is
-    reproducible from a terminal and shows up in the same ledgers.
-    """
-    proc = await asyncio.create_subprocess_exec(
-        sys.executable,
-        "-m",
-        *argv,
-        cwd=str(paths.root()),
-        stdout=asyncio.subprocess.PIPE,
-        stderr=asyncio.subprocess.PIPE,
-    )
-    try:
-        out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout)
-    except asyncio.TimeoutError:
-        proc.kill()
-        await proc.wait()
-        return {"ok": False, "error": {"message": f"timed out after {timeout:.0f}s"}}
-
-    stdout = (out or b"").decode("utf-8", "replace").strip()
-    stderr = (err or b"").decode("utf-8", "replace").strip()
-    for line in reversed(stdout.splitlines()):
-        try:
-            payload = json.loads(line)
-        except json.JSONDecodeError:
-            continue
-        if isinstance(payload, dict):
-            return payload
-    return {
-        "ok": False,
-        "error": {"message": (stderr or stdout or "the command produced no output")[-2000:]},
-    }
-
-
-def envelope_message(payload: dict[str, Any]) -> str:
-    """The one line a status bar should show for a CLI result."""
-    if payload.get("ok"):
-        data = payload.get("data")
-        if isinstance(data, dict) and data.get("message"):
-            return str(data["message"])
-        return "done"
-    error = payload.get("error") or {}
-    message = error.get("message") or "the command failed"
-    fix = error.get("fix")
-    return f"{message}" + (f" — fix: {fix}" if fix else "")
+# `run_tool` and `envelope_message` moved to `ui/tasks.py`, next to `start` --
+# the two are the same decision made twice ("wait for this command" against
+# "watch it"), and having them in one module is what keeps the timeout on the
+# waiting one from being applied to something that should never have had one.
diff --git a/ui/static/tiling.js b/ui/static/tiling.js
index 066669a..97e1042 100644
--- a/ui/static/tiling.js
+++ b/ui/static/tiling.js
@@ -13,8 +13,10 @@
  *      JupyterLab -- kernel, scroll position, unsaved cells and all. The iframe
  *      therefore lives in a fixed overlay outside the pane tree and is flown to
  *      wherever its anchor currently is.
- *   3. Modifier chords. Alt+1/2/3 and Alt+drag are keyboard state the server
- *      never sees.
+ *   3. Dragging a title bar to retile. The drop indicator has to track the
+ *      pointer, and the hit test needs the live geometry of every pane -- both
+ *      at frame rate. Only the settled drop is sent back, as one event.
+ *   4. Modifier chords. Alt+1/2/3 is keyboard state the server never sees.
  *
  * Everything else -- what is open, what is focused, what persists -- stays in
  * Python, where it can be tested.
@@ -93,31 +95,187 @@
     if (handle) startDrag(handle, event);
   });
 
-  /* ------------------------------------------------- alt+drag to retile */
-  let dragging = null;
+  /* ------------------------------------------ drag a title bar to retile */
+  /* Visual Studio's gesture, under this app's constraint: every pane stays
+   * visible, so a drop has to name a *position* -- which column, and where in it
+   * -- not just a container. Three outcomes, chosen by where the pointer is:
+   *
+   *   over another window's title bar -> swap the two panes
+   *   near a column's left/right edge -> a new column at that edge
+   *   anywhere else over a column     -> insert at the nearest slot boundary
+   *
+   * The indicator is painted from the same hit test that produces the emitted
+   * event, so the line drawn is exactly what the server is asked for. Anything
+   * else and the drop lands somewhere the user was not shown.
+   */
+  const DRAG_THRESHOLD_PX = 5;  // under this a press is a click, not a drag
+  const EDGE_RATIO = 0.22;      // of a column's width, at each side
+  const EDGE_MAX_PX = 90;
+  const MAX_COLUMNS = 3;        // mirrors ui/layout.py
+
+  let drag = null;
+  let indicator = null;
+
+  const columnsOf = () => {
+    const tiles = document.querySelector('.grad-tiles');
+    return tiles ? Array.from(tiles.querySelectorAll(':scope > .grad-column')) : [];
+  };
+  const slotsOf = (column) => Array.from(column.querySelectorAll(':scope > .grad-slot'));
+  const windowOf = (slot) => slot.querySelector('.grad-titlebar[data-window]')?.dataset.window;
+
+  /** Where a window sits right now, read from the DOM rather than remembered:
+   *  the pane tree is rebuilt by the server on every retile. */
+  const positionOf = (id) => {
+    const columns = columnsOf();
+    for (let c = 0; c < columns.length; c += 1) {
+      const slots = slotsOf(columns[c]);
+      for (let s = 0; s < slots.length; s += 1) {
+        if (windowOf(slots[s]) === id) return { column: c, slot: s, alone: slots.length === 1 };
+      }
+    }
+    return null;
+  };
+
+  const hitTest = (x, y, self) => {
+    const columns = columnsOf();
+    if (!columns.length) return null;
+
+    // 1. another window's title bar -> swap.
+    const under = document.elementFromPoint(x, y);
+    const bar = under?.closest?.('.grad-titlebar[data-window]');
+    if (bar && bar.dataset.window !== self) return { kind: 'swap', other: bar.dataset.window };
+
+    // 2. which column? Outside them all, the nearest one, remembering which side
+    //    -- a drop past the right edge is how you make a column, so it must not
+    //    be clamped into meaning "drop inside the last one".
+    let index = columns.findIndex((c) => {
+      const b = c.getBoundingClientRect();
+      return x >= b.left && x < b.right;
+    });
+    let outside = 0;
+    if (index < 0) {
+      outside = x < columns[0].getBoundingClientRect().left ? -1 : 1;
+      index = outside < 0 ? 0 : columns.length - 1;
+    }
+    const box = columns[index].getBoundingClientRect();
+
+    // 3. the edge bands -> a new column beside this one. The cap is counted the
+    //    way layout.py counts it: a window dragged out of a column it holds
+    //    alone takes that column with it, so the move can create one without
+    //    ever exceeding MAX_COLUMNS.
+    const here = positionOf(self);
+    const effective = columns.length - (here && here.alone ? 1 : 0);
+    const band = Math.min(EDGE_MAX_PX, box.width * EDGE_RATIO);
+    const left = outside < 0 || (outside === 0 && x < box.left + band);
+    const right = outside > 0 || (outside === 0 && x > box.right - band);
+    if ((left || right) && effective < MAX_COLUMNS) {
+      return {
+        kind: 'column',
+        column: left ? index : index + 1,
+        rect: { left: left ? box.left : box.right, top: box.top, height: box.height },
+      };
+    }
+
+    // 4. otherwise the slot boundary nearest the pointer. `slot` is a boundary
+    //    index into the column as it looks *now*, dragged window included;
+    //    layout.py corrects for the pull when the move stays in one column.
+    const slots = slotsOf(columns[index]);
+    let slot = slots.length;
+    for (let i = 0; i < slots.length; i += 1) {
+      const b = slots[i].getBoundingClientRect();
+      if (y < b.top + b.height / 2) { slot = i; break; }
+    }
+    const edge = slot < slots.length
+      ? slots[slot].getBoundingClientRect().top
+      : (slots.length ? slots[slots.length - 1].getBoundingClientRect().bottom : box.top);
+    return { kind: 'slot', column: index, slot, rect: { left: box.left, top: edge, width: box.width } };
+  };
+
+  const clearPaint = () => {
+    document.querySelectorAll('.grad-swap-target').forEach((el) => el.classList.remove('grad-swap-target'));
+    if (indicator) indicator.style.display = 'none';
+  };
+
+  const paint = (hit) => {
+    clearPaint();
+    if (!hit) return;
+    if (hit.kind === 'swap') {
+      const bar = document.querySelector(`.grad-titlebar[data-window="${CSS.escape(hit.other)}"]`);
+      if (bar) bar.classList.add('grad-swap-target');
+      return;
+    }
+    if (!indicator) {
+      indicator = document.createElement('div');
+      document.body.appendChild(indicator);
+    }
+    const vertical = hit.kind === 'column';
+    indicator.className = `grad-drop-indicator${vertical ? ' vertical' : ''}`;
+    indicator.style.display = 'block';
+    indicator.style.left = `${Math.round(hit.rect.left) - (vertical ? 2 : 0)}px`;
+    indicator.style.top = `${Math.round(hit.rect.top) - (vertical ? 0 : 2)}px`;
+    indicator.style.width = vertical ? '4px' : `${Math.round(hit.rect.width)}px`;
+    indicator.style.height = vertical ? `${Math.round(hit.rect.height)}px` : '4px';
+  };
+
+  const endDrag = (commit) => {
+    if (!drag) return;
+    const { active, hit, id, ghost } = drag;
+    drag.bar?.classList.remove('grad-drag-source');
+    ghost?.remove();
+    drag = null;
+    document.body.classList.remove('grad-dragging');
+    clearPaint();
+    if (!active || !commit || !hit) return;
+
+    if (hit.kind === 'swap') {
+      emit('grad_swap', { a: id, b: hit.other });
+      return;
+    }
+    if (hit.kind === 'column') {
+      emit('grad_retile', { window: id, column: hit.column, slot: null, new_column: true });
+      return;
+    }
+    // A drop onto its own position is not a move. Emitting it anyway would cost
+    // a layout write and a full rebuild of the pane tree for no visible change.
+    const here = positionOf(id);
+    if (here && here.column === hit.column && (hit.slot === here.slot || hit.slot === here.slot + 1)) return;
+    emit('grad_retile', { window: id, column: hit.column, slot: hit.slot, new_column: false });
+  };
 
   document.addEventListener('mousedown', (event) => {
-    if (!event.altKey) return;
+    if (event.button !== 0) return;
     const bar = event.target.closest?.('.grad-titlebar[data-window]');
     if (!bar) return;
-    dragging = bar.dataset.window;
-    document.body.style.cursor = 'grabbing';
+    // The focus, restore and close buttons live inside the bar. A press on one
+    // of them is that button's click, never the start of a drag.
+    if (event.target.closest?.('.grad-winctl')) return;
+    drag = { id: bar.dataset.window, bar, x: event.clientX, y: event.clientY, active: false, hit: null };
+  });
+
+  document.addEventListener('mousemove', (event) => {
+    if (!drag) return;
+    if (!drag.active) {
+      // A press that never travels is a click -- the title bar's own focus
+      // handler -- so the drag only starts once the pointer has committed to it.
+      if (Math.abs(event.clientX - drag.x) + Math.abs(event.clientY - drag.y) < DRAG_THRESHOLD_PX) return;
+      drag.active = true;
+      document.body.classList.add('grad-dragging');
+      drag.bar.classList.add('grad-drag-source');
+      drag.ghost = document.createElement('div');
+      drag.ghost.className = 'grad-drag-ghost';
+      drag.ghost.textContent = drag.id;
+      document.body.appendChild(drag.ghost);
+    }
+    drag.ghost.style.left = `${event.clientX + 14}px`;
+    drag.ghost.style.top = `${event.clientY + 14}px`;
+    drag.hit = hitTest(event.clientX, event.clientY, drag.id);
+    paint(drag.hit);
     event.preventDefault();
   });
 
-  document.addEventListener('mouseup', (event) => {
-    if (!dragging) return;
-    const window_id = dragging;
-    dragging = null;
-    document.body.style.cursor = '';
-    const column = event.target.closest?.('.grad-column');
-    const tiles = document.querySelector('.grad-tiles');
-    if (!tiles) return;
-    const columns = Array.from(tiles.querySelectorAll(':scope > .grad-column'));
-    // Past the right edge means "make a new column", which is why the index can
-    // legitimately equal the column count.
-    const index = column ? columns.indexOf(column) : columns.length;
-    emit('grad_retile', { window: window_id, column: index });
+  document.addEventListener('mouseup', () => endDrag(true));
+  document.addEventListener('keydown', (event) => {
+    if (event.key === 'Escape') endDrag(false);
   });
 
   /* ----------------------------------------------------------- shortcuts */
@@ -189,6 +347,25 @@
 
   window.gradReflow = reflowFrames;
 
+  /* ------------------------------------------------------- sticky transcript */
+  /* A turn appends to the bottom of the transcript for as long as it runs, and
+   * a reader watching it should not have to chase it with the scrollbar. But
+   * scrolling up to re-read a tool's output has to survive the next token, so
+   * this pins only while the reader is already at the bottom. Server-side this
+   * would be a `run_javascript` per flush, fifteen times a second. */
+  window.gradStickBottom = (id) => {
+    const el = document.getElementById(id);
+    if (!el || el.dataset.gradStuck) return;
+    el.dataset.gradStuck = '1';
+    const SLACK_PX = 80;
+    const atBottom = () => el.scrollHeight - el.scrollTop - el.clientHeight <= SLACK_PX;
+    let pinned = true;
+    el.addEventListener('scroll', () => { pinned = atBottom(); }, {passive: true});
+    const stick = () => { if (pinned) el.scrollTop = el.scrollHeight; };
+    new MutationObserver(stick).observe(el, {childList: true, subtree: true, characterData: true});
+    stick();
+  };
+
   window.addEventListener('resize', reflowFrames);
   window.addEventListener('scroll', reflowFrames, true);
   // The pane tree is rebuilt by the server on every retile, so the anchor is a
diff --git a/ui/tasks.py b/ui/tasks.py
new file mode 100644
index 0000000..fe206cb
--- /dev/null
+++ b/ui/tasks.py
@@ -0,0 +1,462 @@
+"""Background tasks: the CLIs the UI starts and does not wait for.
+
+Every button in the workspace that *does* something does it by running the same
+command the agent would (§10). Most of those return in under a second. Four do
+not -- `nb verify` on a fresh kernel, `preflight run`, `wiki map`, `report
+build` -- and the way they used to be run had three problems, all of which this
+module exists to close.
+
+**A wall clock that killed working commands.** `run_tool` awaited the process
+under a 900-second cap and killed it on expiry. But `verify_timeout_s` is 1800
+*per cell*, and preflight's `tests` and `dry_run` are 900 each -- so the cap sat
+*below* the runtime the configuration explicitly allows, and the UI's answer to
+a slow notebook was to kill it and report a timeout. A background task has no
+wall clock: it finishes, or you stop it.
+
+**One status line for every operation.** `Workspace.say` holds a single string,
+so two commands in flight overwrote each other and neither left a trace. Tasks
+have their own list, their own state and their own output.
+
+**Nothing to watch.** A campaign or a wiki rebuild was opaque from click to
+envelope. Output is streamed here as it arrives, in a bounded tail.
+
+**Stopping is asked for, not inflicted.** `terminate()` reaches the CLI and
+nothing it spawned -- and `nb verify` spawns its kernel *detached*, precisely so
+it outlives the CLI. Killing the parent would leave that kernel holding the VRAM
+the verify was meant to free. So a task may carry the tool's own stop verb
+(`nb stop`, `evolve halt`), which is tried first and given a grace period; the
+signal is the fallback, not the mechanism.
+
+The registry is module-level rather than per-`Workspace` on purpose. A
+`Workspace` belongs to one connected client, and a task must survive a browser
+reload -- it is a process on this machine, not a view of one.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import sys
+import time
+from collections import deque
+from dataclasses import dataclass, field
+from typing import Any, Callable, Iterable
+
+from core import paths
+
+log = logging.getLogger("grad.ui")
+
+#: Lines of output kept per task. The tail is a window onto a running command,
+#: not a log: the log is whatever the command writes to the workspace.
+TAIL_LINES = 400
+#: A single line longer than this is cut. A progress bar that never emits a
+#: newline would otherwise buffer without bound.
+MAX_LINE_CHARS = 4000
+#: Finished tasks kept before the oldest are dropped. Enough to look back over a
+#: session; not so many that the window becomes a history.
+KEEP_FINISHED = 40
+#: How long a tool's own stop verb is given before the signal is used instead.
+HALT_GRACE_S = 20.0
+#: Between `terminate()` and `kill()`.
+KILL_GRACE_S = 5.0
+
+RUNNING = "running"
+OK = "ok"
+FAILED = "failed"
+CANCELLED = "cancelled"
+
+#: `state -> the one accent it is drawn in`, matching the rest of the app: a
+#: dashed border while it runs, because an outcome that has not happened yet is
+#: not a green one.
+STATE_TONE = {RUNNING: "dashed", OK: "ok", FAILED: "broken", CANCELLED: "attention"}
+
+
+@dataclass
+class Task:
+    """One local command, running or finished."""
+
+    id: str
+    label: str
+    argv: tuple[str, ...]
+    #: The tool's own graceful stop, if it has one. See the module docstring.
+    halt: tuple[str, ...] | None = None
+    state: str = RUNNING
+    started_at: float = field(default_factory=time.monotonic)
+    finished_at: float | None = None
+    exit_code: int | None = None
+    #: The last JSON object the command printed on stdout -- the §8 envelope,
+    #: tracked as the lines arrive rather than by re-reading the output, so a
+    #: task that printed a gigabyte still costs one dict.
+    envelope: dict[str, Any] | None = None
+    tail: deque[tuple[str, str]] = field(default_factory=lambda: deque(maxlen=TAIL_LINES))
+    #: Lines that fell out of the tail. Reported rather than silently dropped.
+    dropped: int = 0
+    #: Called once, when the task settles. See `start`.
+    on_done: Any = None
+    #: Set by `cancel` before it signals, read by `_run` when the process goes.
+    stopping: bool = False
+    _process: Any = None
+    #: The coroutine driving this task, held so it cannot be collected.
+    #: asyncio keeps only a *weak* reference to a running task, so a bare
+    #: `create_task` whose result nobody holds can vanish part-way through --
+    #: the same failure `Workspace.spawn` exists to close, and here it would
+    #: strand a process with no pump and nothing left to settle it.
+    _driver: Any = None
+
+    @property
+    def running(self) -> bool:
+        return self.state == RUNNING
+
+    @property
+    def elapsed(self) -> float:
+        return (self.finished_at or time.monotonic()) - self.started_at
+
+    def append(self, line: str, tag: str = "out") -> None:
+        if len(line) > MAX_LINE_CHARS:
+            line = line[:MAX_LINE_CHARS] + f" … +{len(line) - MAX_LINE_CHARS:,} characters"
+        if len(self.tail) == self.tail.maxlen:
+            self.dropped += 1
+        self.tail.append((tag, line))
+        if tag == "out":
+            self._remember_envelope(line)
+
+    def note(self, line: str) -> None:
+        """A line from the workspace itself -- that a stop was asked for, say."""
+        self.append(line, tag="note")
+
+    def _remember_envelope(self, line: str) -> None:
+        stripped = line.strip()
+        if not stripped.startswith("{"):
+            return
+        try:
+            payload = json.loads(stripped)
+        except json.JSONDecodeError:
+            return
+        if isinstance(payload, dict):
+            self.envelope = payload
+
+
+_tasks: dict[str, Task] = {}
+_counter = 0
+
+
+def all_tasks() -> list[Task]:
+    """Newest first, which is the order the window wants."""
+    return list(reversed(_tasks.values()))
+
+
+def get(task_id: str) -> Task | None:
+    return _tasks.get(task_id)
+
+
+def running() -> list[Task]:
+    return [t for t in _tasks.values() if t.running]
+
+
+def clear_finished() -> int:
+    """Drop every settled task. Returns how many went."""
+    gone = [tid for tid, task in _tasks.items() if not task.running]
+    for tid in gone:
+        del _tasks[tid]
+    return len(gone)
+
+
+def reset() -> None:
+    """Empty the registry. For tests -- module state outlives a fixture."""
+    global _counter
+
+    _tasks.clear()
+    _counter = 0
+
+
+def _register(task: Task) -> Task:
+    _tasks[task.id] = task
+    finished = [tid for tid, other in _tasks.items() if not other.running]
+    for tid in finished[: max(0, len(finished) - KEEP_FINISHED)]:
+        del _tasks[tid]
+    return task
+
+
+def _next_id() -> str:
+    global _counter
+
+    _counter += 1
+    return f"task-{_counter}"
+
+
+# ---------------------------------------------------------------------------
+# starting one
+# ---------------------------------------------------------------------------
+def start(
+    label: str,
+    *argv: str,
+    halt: Iterable[str] | None = None,
+    on_done: Callable[[Task], None] | None = None,
+) -> Task:
+    """Run one of Grad's own CLIs in the background and return its handle.
+
+    Returns as soon as the process is *registered*, not when it is spawned: the
+    caller is a click handler and the window that lists tasks reads the registry
+    on the next poll, so a task has to exist the moment the click is over --
+    otherwise a two-second window opens in which nothing on screen says anything
+    happened.
+
+    `on_done` runs once the task settles, however it settled -- a verify has to
+    write its record whether it passed, failed or was stopped, because "we do
+    not know" is a different notebook state from "it was fine before".
+    """
+    task = _register(Task(_next_id(), label, tuple(argv), tuple(halt) if halt else None))
+    task.on_done = on_done
+    task.note(f"$ python -m {' '.join(argv)}")
+    task._driver = asyncio.get_event_loop().create_task(_run(task))  # noqa: SLF001 - its own field
+    return task
+
+
+async def drained(task: Task) -> Task:
+    """Wait until this task's driver has finished with the process.
+
+    Later than `state`: a stop that lands before the process is even spawned
+    settles the task immediately, while `_run` still has to spawn, terminate and
+    drain it. Nothing in the UI waits for this -- the poll shows the state -- but
+    a caller that needs the process to be *gone* does.
+    """
+    driver = task._driver  # noqa: SLF001 - its own field
+    if driver is not None:
+        await asyncio.gather(driver, return_exceptions=True)
+    return task
+
+
+async def _run(task: Task) -> None:
+    try:
+        task._process = await asyncio.create_subprocess_exec(  # noqa: SLF001 - its own field
+            sys.executable,
+            "-m",
+            *task.argv,
+            cwd=str(paths.root()),
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+        )
+    except OSError as exc:
+        task.note(f"could not start: {exc}")
+        _settle(task, FAILED, None)
+        return
+
+    process = task._process  # noqa: SLF001 - its own field
+    if task.stopping:
+        # Stopped while the spawn was still in flight. `cancel` found no process
+        # to signal and settled the task itself, so killing it lands here --
+        # without this, a task the workspace reported as stopped runs to
+        # completion unattended, with nothing on screen saying it is still going.
+        task.note("stopped while it was starting")
+        try:
+            process.terminate()
+        except (ProcessLookupError, OSError):
+            pass
+
+    await asyncio.gather(
+        _pump(process.stdout, task, tag="out"),
+        _pump(process.stderr, task, tag="err"),
+    )
+    code = await process.wait()
+    # The single place a started task settles. `cancel` sets `stopping` and then
+    # waits rather than settling the task itself: both coroutines are awake at
+    # once, and whichever wrote the state second would win. A stopped task
+    # reporting the signal's exit code as a failure is the wrong verdict on a
+    # command that did nothing wrong.
+    if task.stopping:
+        _settle(task, CANCELLED, code)
+    else:
+        _settle(task, OK if code == 0 else FAILED, code)
+
+
+async def _pump(stream: Any, task: Task, *, tag: str) -> None:
+    """Split a pipe into lines without `readline`.
+
+    `StreamReader.readline` raises once a single line passes its 64 KiB limit,
+    and a training log's progress line can. Chunks cannot hit that.
+    """
+    pending = ""
+    while True:
+        chunk = await stream.read(8192)
+        if not chunk:
+            break
+        pending += chunk.decode("utf-8", "replace")
+        *complete, pending = pending.split("\n")
+        for line in complete:
+            task.append(line.rstrip("\r"), tag)
+        if len(pending) > MAX_LINE_CHARS:
+            task.append(pending, tag)
+            pending = ""
+    if pending:
+        task.append(pending.rstrip("\r"), tag)
+
+
+def _settle(task: Task, state: str, code: int | None) -> None:
+    task.state = state
+    task.exit_code = code
+    task.finished_at = time.monotonic()
+    callback, task.on_done = task.on_done, None
+    if callback is None:
+        return
+    try:
+        callback(task)
+    except Exception as exc:  # noqa: BLE001 - a callback must not strand the task
+        log.exception("the completion callback for %s failed", task.id)
+        task.note(f"the workspace could not record this result: {type(exc).__name__}")
+
+
+# ---------------------------------------------------------------------------
+# stopping one
+# ---------------------------------------------------------------------------
+async def cancel(task_id: str) -> str:
+    """Stop a task, asking the tool first. Returns the line to put on screen.
+
+    The order matters more here than anywhere else in this module. `terminate()`
+    reaches the CLI and nothing it started, and `nb verify` starts its kernel
+    detached so it survives the CLI exiting -- so killing the parent would leave
+    a kernel holding VRAM with nothing left to shut it down. When a tool has a
+    verb for stopping itself, that verb *is* the cancel; the signal is what
+    happens when it does not work.
+    """
+    task = _tasks.get(task_id)
+    if task is None:
+        return f"no task {task_id}"
+    if not task.running:
+        return f"{task.label} already finished"
+
+    # Before anything else: `_run` reads this when the process goes, and it is
+    # what tells a stop apart from a failure.
+    task.stopping = True
+
+    if task.halt:
+        task.note(f"stopping: python -m {' '.join(task.halt)}")
+        payload = await run_tool(*task.halt, timeout=HALT_GRACE_S)
+        if not payload.get("ok"):
+            task.note(f"the tool's own stop failed: {envelope_message(payload)}")
+        if await _waits_for_exit(task, HALT_GRACE_S):
+            return f"{task.label} stopped"
+        task.note("it did not stop when asked; signalling")
+
+    process = task._process  # noqa: SLF001 - its own field
+    if process is None:
+        # Nothing to signal, and nothing that will ever settle it.
+        _settle(task, CANCELLED, None)
+        return f"{task.label} stopped before it started"
+
+    try:
+        process.terminate()
+    except (ProcessLookupError, OSError):
+        return f"{task.label} was already gone"
+    if await _waits_for_exit(task, KILL_GRACE_S):
+        return f"{task.label} stopped"
+    try:
+        process.kill()
+    except (ProcessLookupError, OSError):
+        pass
+    task.note("killed")
+    return f"{task.label} killed"
+
+
+async def _waits_for_exit(task: Task, seconds: float) -> bool:
+    """Poll for the process to go, so a grace period is a grace period.
+
+    `process.wait()` is not awaited here: `_run` is already awaiting it, and a
+    second waiter on the same transport is what turns a cancel into a hang.
+    """
+    process = task._process  # noqa: SLF001 - its own field
+    if process is None:
+        return True
+    deadline = time.monotonic() + seconds
+    while time.monotonic() < deadline:
+        if process.returncode is not None:
+            return True
+        await asyncio.sleep(0.1)
+    return process.returncode is not None
+
+
+# ---------------------------------------------------------------------------
+# running one and waiting for it
+# ---------------------------------------------------------------------------
+async def run_tool(*argv: str, timeout: float = 120.0, stdin: str | None = None) -> dict[str, Any]:
+    """Run one of Grad's own CLIs and parse its JSON envelope.
+
+    For the commands that answer immediately -- selecting a project, a verdict,
+    a status poll. Anything that can run for minutes goes through `start`
+    instead, because the timeout here is enforced by killing the process.
+
+    Every button in the UI that *does* something does it by running the same
+    command the agent would, with `--json`. That is deliberate: it keeps the UI
+    free of logic (§10), and it means anything the UI can do is reproducible
+    from a terminal and lands in the same ledgers.
+
+    `stdin` exists for exactly one caller: storing a credential. A token passed
+    as an argument is visible to anything that can list processes, so it goes
+    down a pipe instead and the CLI reads it with `--stdin`.
+    """
+    try:
+        proc = await asyncio.create_subprocess_exec(
+            sys.executable,
+            "-m",
+            *argv,
+            cwd=str(paths.root()),
+            stdin=asyncio.subprocess.PIPE if stdin is not None else None,
+            stdout=asyncio.subprocess.PIPE,
+            stderr=asyncio.subprocess.PIPE,
+        )
+    except OSError as exc:
+        return {"ok": False, "error": {"message": f"could not run the command: {exc}"}}
+    try:
+        out, err = await asyncio.wait_for(
+            proc.communicate(stdin.encode() if stdin is not None else None), timeout=timeout
+        )
+    except asyncio.TimeoutError:
+        proc.kill()
+        await proc.wait()
+        return {
+            "ok": False,
+            "error": {
+                "message": f"timed out after {timeout:.0f}s",
+                "fix": "long commands belong in the background — open the tasks window",
+            },
+        }
+
+    stdout = (out or b"").decode("utf-8", "replace").strip()
+    stderr = (err or b"").decode("utf-8", "replace").strip()
+    for line in reversed(stdout.splitlines()):
+        try:
+            payload = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(payload, dict):
+            return payload
+    return {
+        "ok": False,
+        "error": {"message": (stderr or stdout or "the command produced no output")[-2000:]},
+    }
+
+
+def envelope_message(payload: dict[str, Any]) -> str:
+    """The one line a status bar should show for a CLI result."""
+    if payload.get("ok"):
+        data = payload.get("data")
+        if isinstance(data, dict) and data.get("message"):
+            return str(data["message"])
+        return "done"
+    error = payload.get("error") or {}
+    message = error.get("message") or "the command failed"
+    fix = error.get("fix")
+    return f"{message}" + (f" — fix: {fix}" if fix else "")
+
+
+def task_message(task: Task) -> str:
+    """The one line a status bar should show for a finished task."""
+    if task.running:
+        return f"{task.label} running …"
+    if task.state == CANCELLED:
+        return f"{task.label} stopped"
+    if task.envelope is not None:
+        return f"{task.label}: {envelope_message(task.envelope)}"
+    if task.state == OK:
+        return f"{task.label} done"
+    return f"{task.label} failed (exit {task.exit_code})"
diff --git a/ui/tokens.py b/ui/tokens.py
index 63d99b9..0ebf67a 100644
--- a/ui/tokens.py
+++ b/ui/tokens.py
@@ -160,7 +160,21 @@ def _base() -> str:
 
 .grad-app, .grad-app * { box-sizing: border-box; border-radius: 0; }
 body { margin: 0; background: var(--grad-desk); }
+
+/* The page itself must never scroll: the shell is already sized to fill the
+   window exactly (`100vh` less its 14px margins), so any scrollbar here means
+   a wrapper is adding space the layout did not budget for. NiceGUI's
+   `.nicegui-content` adds it twice over -- `padding: 16px` made the document
+   32px taller than the window, and `align-items: flex-start` on the same
+   flex wrapper let `.grad-app` size to its widest pane's max-content rather
+   than to the window, which is what pushed the tiling area ~200px off the
+   right edge. `align-self` answers the second; the width and height pin the
+   app to the viewport so the shell's own arithmetic is the only thing
+   deciding its size. */
+html, body { height: 100%; overflow: hidden; }
+.nicegui-content { padding: 0 !important; gap: 0 !important; width: 100%; }
 .grad-app {
+    width: 100%; height: 100vh; align-self: stretch; overflow: hidden;
     background: var(--grad-desk);
     color: var(--grad-ink);
     font-family: var(--grad-font-sans);
@@ -220,22 +234,34 @@ def _shell() -> str:
 }
 .grad-appbar-btn:hover { background: var(--grad-paper); color: var(--grad-ink); }
 
-.grad-opener {
-    display: flex; align-items: stretch; background: var(--grad-paper-sunk);
-    border-bottom: var(--grad-border); font-family: var(--grad-font-mono);
-    font-size: 11px; font-weight: 700; letter-spacing: 0.08em; flex: 0 0 auto;
-    overflow-x: auto;
+/* The `⋯` button carries a menu, so it gets the caret's job: a little wider
+   than a word button, and legible as a target rather than as punctuation. */
+.grad-dots { font-size: 15px; line-height: 1; padding: 1px 9px 4px; letter-spacing: 0.1em; }
+
+/* One row of the `⋯` menu: a mark, a name, and what the window is for. Rows are
+   buttons so the keyboard reaches them, which means resetting the four
+   properties a `