From 09b5e7d598011ebe50f478d254977f8f0def28f2 Mon Sep 17 00:00:00 2001 From: Alexander Watson Date: Mon, 4 May 2026 22:08:28 -0700 Subject: [PATCH 1/2] fix(examples): harden multi-agent notepad 409 retry and improve docs Add decorrelated jitter to the GitHub Contents API retry loop so racing workers don't retry in lockstep, and widen the retry set to 5xx for transient server errors. Without jitter, N concurrent workers trip the same backoff schedule and re-collide on each beat. Also: - Extract the inline 220-line sandbox runner from demo.sh into a separate runner.sh. demo.sh stays as host orchestration; runner.sh is the per-sandbox program. demo.sh shrinks 460 to 240 lines. - Rewrite README to lead with "Why GitHub as a shared notepad?" and add a "Memory architecture variants" section covering pile, append-journal, and indexed-memory patterns. - Switch Quick Start to use \`gh auth token\` as the primary path. - Pass the worker slice arg explicitly instead of reading \$9 from the outer dispatcher. - Add .gitignore for personal run_demo.sh helpers. --- examples/multi-agent-notepad/.gitignore | 1 + examples/multi-agent-notepad/README.md | 245 +++++++++++++++--------- examples/multi-agent-notepad/demo.sh | 207 +------------------- examples/multi-agent-notepad/runner.sh | 235 +++++++++++++++++++++++ 4 files changed, 390 insertions(+), 298 deletions(-) create mode 100644 examples/multi-agent-notepad/.gitignore create mode 100755 examples/multi-agent-notepad/runner.sh diff --git a/examples/multi-agent-notepad/.gitignore b/examples/multi-agent-notepad/.gitignore new file mode 100644 index 0000000000..6a5d5848e9 --- /dev/null +++ b/examples/multi-agent-notepad/.gitignore @@ -0,0 +1 @@ +run_demo.sh diff --git a/examples/multi-agent-notepad/README.md b/examples/multi-agent-notepad/README.md index 12d0676318..a98acb8159 100644 --- a/examples/multi-agent-notepad/README.md +++ b/examples/multi-agent-notepad/README.md @@ -3,67 +3,99 @@ # Multi-Agent Shared Notepad Demo -Launch multiple coding agents in parallel OpenShell sandboxes and let them use -a shared markdown notepad. This version uses Codex as the agent runtime and -GitHub as the durable notes backend. +Run multiple Codex coding agents in parallel OpenShell sandboxes, with a +GitHub repository as the durable shared notepad they coordinate through. + +## Why GitHub as a shared notepad? + +Long-running agents that produce artifacts — research notes, memory entries, +reports, decision logs — need somewhere durable to write them. The store has +to survive individual sandbox death, accept many concurrent writers safely, +and stay inspectable after the fact. GitHub provides all three for free: + +- **Durable across sandbox death.** A sandbox can crash mid-run; the + committed artifact persists. +- **Concurrency control built in.** Every PUT must include the current file + SHA, and the branch ref serializes commits. Racing writers see HTTP 409 + and retry instead of silently overwriting each other. +- **Auditable.** Every write is a commit with author, message, diff, and + timestamp. Free observability. +- **Reviewable.** Humans review agent output with the same PR/diff tooling + they already use for code. +- **No new infra.** Most teams already have a GitHub org. + +The pattern works best when artifacts are markdown or other text (so diffs +are useful) and when write rates are moderate (commits per minute, not per +second). For higher rates or structured queries, graduate to a real +datastore — see [Beyond map/reduce](#beyond-mapreduce-memory-architecture-variants) +below. + +## What this demo shows + +The simplest useful shape: **map/reduce**. N worker agents fan out and write +one note each; one synthesis agent reads them and writes a summary. -This example demonstrates three OpenShell ideas together: +```text +runs//notes/agent-1.md ← worker 1 writes +runs//notes/agent-2.md ← worker 2 writes +... +runs//summary.md ← synthesis agent writes +``` -- multiple isolated agents can run at the same time, -- agents can coordinate through durable shared notes without sharing a - filesystem, -- Codex OAuth can use provider-backed placeholders instead of storing real - OAuth tokens in the sandbox filesystem. +Each worker gets a different research angle on the same topic. Workers share +neither filesystem nor container — only the GitHub repository. -GitHub is the backing store in this example because it is familiar, durable, -branch-aware, and easy to inspect. The pattern is the important part: separate -coding agents communicate by reading and writing scoped markdown notes. +The demo also exercises two OpenShell features: -## Prerequisites +- **Provider-backed credentials.** Sandboxes get placeholders, not the real + Codex OAuth tokens or the GitHub token. The proxy resolves them at the + network boundary. +- **Scoped network policy.** Each sandbox can only `GET` and `PUT` paths under + `/repos///contents/runs//**`. -- OpenShell CLI from current `main` with provider env lookup and `--upload` - support. If it is not on `PATH`, set `OPENSHELL_BIN` to the binary path. -- A running OpenShell gateway: +## Files in this example - ```bash - openshell gateway start - ``` +- `demo.sh` — host orchestration. Validates env, creates providers, launches + sandboxes, waits for completion. Read this to understand how the run is + driven from the host. +- `runner.sh` — the script that runs **inside each sandbox**. Bootstraps + Codex OAuth, calls `codex exec`, and writes the result to GitHub. Read this + to understand the agent side. +- `policy.template.yaml` — the network policy applied to every sandbox in + the run. Renders to a concrete policy with the configured owner, repo, and + run id. +- `prompts/worker.md`, `prompts/synthesis.md` — agent instructions. -- Local Codex sign-in on the host machine: - - ```bash - codex login - codex login status - ``` +## Prerequisites -- `jq` on the host machine -- A disposable or demo-only GitHub repository to use as the shared notepad -- A GitHub token with permission to write repository contents +- OpenShell CLI from current `main` (or set `OPENSHELL_BIN` to the binary + path) +- A running OpenShell gateway: `openshell gateway start` +- Local Codex sign-in on the host: `codex login` +- `gh` (GitHub CLI) signed in, **or** a GitHub PAT with `contents:write` +- `jq` on the host +- A disposable or demo-only GitHub repository -Use an empty repository or one created specifically for this demo. The script -writes under `runs//` and will update files at those paths if they -already exist. Do not point the demo at a production repository unless you are -comfortable with it creating and updating files under that prefix. +The demo writes under `runs//`. Use a repository created specifically +for this demo, or one you're comfortable with the demo creating files in. -## Quick Start +## Quick start ```bash export DEMO_GITHUB_OWNER= export DEMO_GITHUB_REPO= -export DEMO_GITHUB_TOKEN= +export DEMO_GITHUB_TOKEN="$(gh auth token)" bash examples/multi-agent-notepad/demo.sh ``` -If you use the GitHub CLI, you can use your signed-in GitHub session: - -```bash -export DEMO_GITHUB_TOKEN="$(gh auth token)" -``` +`gh auth token` returns a token with whatever scopes you logged in with — +usually broader than `contents:write`. If you'd rather use a scope-limited +PAT, set `DEMO_GITHUB_TOKEN` to that value instead. By default the script launches five worker agents and one synthesis agent in -the OpenShell `base` sandbox image, where Codex is preinstalled. -To run a faster smoke test: +the OpenShell `base` image, where Codex is preinstalled. To run a faster +smoke test: ```bash export DEMO_AGENT_COUNT=2 @@ -80,64 +112,93 @@ export DEMO_RUN_ID="$(date +%Y%m%d-%H%M%S)" export DEMO_KEEP_SANDBOXES=0 ``` -`DEMO_RUN_ID` is used in sandbox names and policy paths, so keep it to -lowercase letters, numbers, and `-`. -Use a fresh `DEMO_RUN_ID` for each run unless you intentionally want to update -the files from a previous run. +`DEMO_RUN_ID` is used in sandbox names and policy paths, so keep it +lowercase letters, numbers, and `-`. Use a fresh `DEMO_RUN_ID` per run unless +you intentionally want to overwrite a previous run's files. -`DEMO_BRANCH` is used in GitHub API calls and output links. For this demo, use -a simple branch name containing only letters, numbers, `.`, `_`, and `-`. +`DEMO_BRANCH` may contain only letters, numbers, `.`, `_`, and `-`. -If a worker fails, the script prints the relevant log tail and keeps full logs -in a temporary directory. Set `DEMO_KEEP_SANDBOXES=1` when you want to inspect -the sandboxes after the run; temporary providers are still removed. +If a worker fails, the script prints the relevant log tail and keeps full +logs in a temporary directory. Set `DEMO_KEEP_SANDBOXES=1` to inspect +sandboxes after the run; temporary providers are still removed. -## What It Creates +## How credential protection works -The demo creates a small shared notepad for one multi-agent run. Each worker -writes a note, then the synthesis agent reads those notes and writes a summary: +The host script reads your local Codex sign-in and creates a temporary +OpenShell provider for the OAuth tokens. It also creates a temporary +provider for the GitHub token. Sandboxes receive provider placeholders, not +the real credential values. -```text -runs//notes/agent-1.md -runs//notes/agent-2.md -runs//notes/agent-3.md -runs//notes/agent-4.md -runs//notes/agent-5.md -runs//summary.md -``` - -Each worker gets a different research angle for the same topic. Workers never -share a filesystem or container. The GitHub repository is the shared notepad -and coordination layer. - -This is not a general-purpose agent memory system. It is a simple markdown -notepad that isolated agents can use to exchange findings. - -If files for the same `DEMO_RUN_ID` already exist, the demo updates them in -place. +When `codex` or `curl` inside a sandbox sends an authorized request, the +OpenShell proxy resolves the placeholder at the network boundary and +forwards the request upstream with the real credential. The credential +values never sit in the sandbox filesystem. -## How Credential Protection Works +## Network policy -The host script uses your local Codex sign-in to create a temporary OpenShell -provider for Codex OAuth. It also creates a temporary provider for the GitHub -token. Sandboxes receive provider placeholders, not the real credential values. +The script renders `policy.template.yaml` for the configured GitHub +repository and run id. The policy allows: -When Codex or `curl` sends an authorized request, the OpenShell proxy resolves -the placeholder at the network boundary and forwards the request upstream with -the real credential. The credential values do not need to be copied into the -sandbox filesystem. - -## Network Policy - -The script renders `policy.template.yaml` for the configured GitHub repository -and run id. The policy allows: - -- Codex traffic to OpenAI and ChatGPT endpoints used by the community base image -- limited Codex plugin metadata reads from `github.com/openai/plugins.git` -- GitHub REST `GET` and `PUT` calls scoped to: - - ```text - /repos///contents/runs//** - ``` +- Codex traffic to OpenAI and ChatGPT endpoints used by the community base + image +- Limited Codex plugin metadata reads from `github.com/openai/plugins.git` +- GitHub REST `GET` and `PUT` calls scoped to + `/repos///contents/runs//**` The policy does not grant broad GitHub API access. + +## Beyond map/reduce: memory architecture variants + +The pile-and-reduce shape in this demo is one of several useful patterns +for using a Git repository as durable agent memory. Each is a different +tradeoff between contention, complexity, and what you can ask of the +artifact afterwards. + +### Pile (this demo) + +Each agent writes its own file at a unique path. A reducer reads them all +and writes a summary. + +- **Best for:** parallel exploration with bounded fan-in (research, + multi-perspective analysis, code review across files). +- **Contention:** low. Different files have independent SHAs, so the only + source of 409s is branch-ref locking when many commits land at once. The + retry loop in `put_contents` handles it. +- **Extension:** hierarchical reduce — k workers per reducer, log(N) levels — + for runs that exceed what one synthesis agent can chew through. + +### Append journal + +A long-lived agent (or small team) appends entries to a shared `journal.md` +across many sessions. Useful as chronological memory: "decisions made", +"things learned", "open questions". + +- **Best for:** a single agent or a small group continuously writing an + ordered log. +- **Contention:** high if many writers; the GET-PUT-409-retry pattern is + load-bearing here. For high-write-rate journals, switch to the Git Data + API — create blobs concurrently, build one tree, commit once — to avoid + branch-ref serialization. +- **Extension:** split by month (`journal/2026-05.md`) to bound file size and + spread contention across files. + +### Indexed memory + +Agents read and write keyed entries — `memory/.md` per key. Higher- +level agents look up "what do we know about X?" by reading the relevant +file directly. + +- **Best for:** keyed memory that survives across runs and is queried by + topic. +- **Contention:** per-key, which is usually what you want. Hot keys still + benefit from the retry loop. +- **Extension:** maintain a `index.md` listing all keys. The index becomes + the hot spot — update it lazily, or rebuild it from a directory listing + when needed. + +### When to graduate beyond GitHub + +GitHub stops being the right backend when you need sustained write rates +above roughly 10/sec, sub-100ms reads, structured queries, or vector search. +At that point reach for Postgres, a KV store, or a vector DB. The shape of +the agent code stays similar — only the storage primitive changes. diff --git a/examples/multi-agent-notepad/demo.sh b/examples/multi-agent-notepad/demo.sh index c1777ab8aa..09032c118b 100755 --- a/examples/multi-agent-notepad/demo.sh +++ b/examples/multi-agent-notepad/demo.sh @@ -126,212 +126,7 @@ render_policy() { write_runner() { cp "${PROMPTS_DIR}/worker.md" "${PROMPTS_UPLOAD_DIR}/worker.md" cp "${PROMPTS_DIR}/synthesis.md" "${PROMPTS_UPLOAD_DIR}/synthesis.md" - - cat > "$RUNNER_FILE" <<'EOF' -#!/usr/bin/env bash -set -euo pipefail - -MODE="$1" -OWNER="$2" -REPO="$3" -BRANCH="$4" -RUN_ID="$5" -AGENT_INDEX="${6:-0}" -AGENT_COUNT="${7:-0}" -TOPIC="${8:-}" - -api_url() { - printf 'https://api.github.com%s' "$1" -} - -github_request() { - local method="$1" - local path="$2" - local output="$3" - shift 3 - curl -sS \ - -X "$method" \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${DEMO_GITHUB_TOKEN}" \ - -H "X-GitHub-Api-Version: 2022-11-28" \ - "$@" \ - -o "$output" \ - -w "%{http_code}" \ - "$(api_url "$path")" -} - -render_template() { - local template="$1" - local slice="${2:-}" - node - "$template" "$AGENT_INDEX" "$AGENT_COUNT" "$TOPIC" "$slice" <<'NODE' -const fs = require("fs"); -const [templatePath, agentIndex, agentCount, topic, slice] = process.argv.slice(2); -let text = fs.readFileSync(templatePath, "utf8"); -text = text.replace(/^(?:\n)+\n?/, ""); -const replacements = { - "{{AGENT_INDEX}}": agentIndex, - "{{AGENT_COUNT}}": agentCount, - "{{TOPIC}}": topic, - "{{SLICE}}": slice, -}; -for (const [needle, value] of Object.entries(replacements)) { - text = text.split(needle).join(value); -} -process.stdout.write(text); -NODE -} - -bootstrap_codex_oauth() { - mkdir -p "$HOME/.codex" - node - <<'NODE' -const fs = require("fs"); -const path = `${process.env.HOME}/.codex/auth.json`; -const b64u = (obj) => Buffer.from(JSON.stringify(obj)).toString("base64url"); -const now = Math.floor(Date.now() / 1000); -const fakeIdToken = [ - b64u({ alg: "none", typ: "JWT" }), - b64u({ - iss: "https://auth.openai.com", - aud: "codex", - sub: "openshell-placeholder", - email: "placeholder@example.com", - iat: now, - exp: now + 3600, - }), - "placeholder", -].join("."); - -fs.writeFileSync(path, JSON.stringify({ - auth_mode: "chatgpt", - OPENAI_API_KEY: null, - tokens: { - id_token: fakeIdToken, - access_token: process.env.CODEX_AUTH_ACCESS_TOKEN, - refresh_token: process.env.CODEX_AUTH_REFRESH_TOKEN, - account_id: process.env.CODEX_AUTH_ACCOUNT_ID, - }, - last_refresh: new Date().toISOString(), -}, null, 2)); -NODE - chmod 600 "$HOME/.codex/auth.json" -} - -put_contents() { - local repo_path="$1" - local source_file="$2" - local message="$3" - local get_body put_body put_response status sha - get_body="$(mktemp)" - put_body="$(mktemp)" - put_response="$(mktemp)" - - status="$(github_request GET "/repos/${OWNER}/${REPO}/contents/${repo_path}?ref=${BRANCH}" "$get_body")" - if [[ "$status" == "200" ]]; then - sha="$(node -e 'const fs=require("fs"); const p=JSON.parse(fs.readFileSync(process.argv[1],"utf8")); process.stdout.write(p.sha || "");' "$get_body")" - elif [[ "$status" == "404" ]]; then - sha="" - else - echo "GitHub GET ${repo_path} failed with HTTP ${status}" >&2 - cat "$get_body" >&2 - return 1 - fi - - node - "$source_file" "$message" "$BRANCH" "$sha" > "$put_body" <<'NODE' -const fs = require("fs"); -const [file, message, branch, sha] = process.argv.slice(2); -const body = { - message, - branch, - content: fs.readFileSync(file).toString("base64"), -}; -if (sha) body.sha = sha; -process.stdout.write(JSON.stringify(body)); -NODE - - status="$(github_request PUT "/repos/${OWNER}/${REPO}/contents/${repo_path}" "$put_response" --data-binary "@${put_body}")" - if [[ "$status" != "200" && "$status" != "201" ]]; then - echo "GitHub PUT ${repo_path} failed with HTTP ${status}" >&2 - cat "$put_response" >&2 - return 1 - fi -} - -get_contents_file() { - local repo_path="$1" - local destination="$2" - local body status - body="$(mktemp)" - status="$(github_request GET "/repos/${OWNER}/${REPO}/contents/${repo_path}?ref=${BRANCH}" "$body")" - if [[ "$status" != "200" ]]; then - echo "GitHub GET ${repo_path} failed with HTTP ${status}" >&2 - cat "$body" >&2 - return 1 - fi - node - "$body" "$destination" <<'NODE' -const fs = require("fs"); -const [bodyPath, destination] = process.argv.slice(2); -const body = JSON.parse(fs.readFileSync(bodyPath, "utf8")); -fs.writeFileSync(destination, Buffer.from((body.content || "").replace(/\s/g, ""), "base64")); -NODE -} - -run_codex_to_file() { - local prompt_file="$1" - local output_file="$2" - codex exec \ - --skip-git-repo-check \ - --sandbox read-only \ - --ephemeral \ - --output-last-message "$output_file" \ - "$(cat "$prompt_file")" -} - -worker() { - local slice="$9" - local prompt_file output_file repo_path - bootstrap_codex_oauth - prompt_file="$(mktemp)" - output_file="$(mktemp)" - repo_path="runs/${RUN_ID}/notes/agent-${AGENT_INDEX}.md" - - render_template /sandbox/payload/prompts/worker.md "$slice" > "$prompt_file" - run_codex_to_file "$prompt_file" "$output_file" - put_contents "$repo_path" "$output_file" "Add agent ${AGENT_INDEX} note for ${RUN_ID}" - printf 'wrote %s\n' "$repo_path" -} - -synthesis() { - local notes_dir prompt_file output_file repo_path - bootstrap_codex_oauth - notes_dir="$(mktemp -d)" - prompt_file="$(mktemp)" - output_file="$(mktemp)" - repo_path="runs/${RUN_ID}/summary.md" - - for i in $(seq 1 "$AGENT_COUNT"); do - get_contents_file "runs/${RUN_ID}/notes/agent-${i}.md" "${notes_dir}/agent-${i}.md" - done - - render_template /sandbox/payload/prompts/synthesis.md "" > "$prompt_file" - { - printf '\n\n## Worker Notes\n\n' - for i in $(seq 1 "$AGENT_COUNT"); do - printf '\n\n---\n\n' - cat "${notes_dir}/agent-${i}.md" - done - } >> "$prompt_file" - - run_codex_to_file "$prompt_file" "$output_file" - put_contents "$repo_path" "$output_file" "Add multi-agent summary for ${RUN_ID}" - printf 'wrote %s\n' "$repo_path" -} - -case "$MODE" in - worker) worker "$@" ;; - synthesis) synthesis ;; - *) echo "unknown mode: $MODE" >&2; exit 2 ;; -esac -EOF + cp "${SCRIPT_DIR}/runner.sh" "$RUNNER_FILE" chmod +x "$RUNNER_FILE" } diff --git a/examples/multi-agent-notepad/runner.sh b/examples/multi-agent-notepad/runner.sh new file mode 100755 index 0000000000..d04186248d --- /dev/null +++ b/examples/multi-agent-notepad/runner.sh @@ -0,0 +1,235 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# This script runs INSIDE each OpenShell sandbox. +# +# It is uploaded to /sandbox/payload/demo-runner.sh by demo.sh and invoked as +# the sandbox entrypoint. It receives positional arguments from `run_sandbox()` +# in demo.sh and reads DEMO_GITHUB_TOKEN from the environment, where the +# OpenShell proxy resolves the provider placeholder at the network boundary. +# +# Modes: +# worker — render the worker prompt for a slice, run codex, PUT the note. +# synthesis — read every worker note, run the synthesis prompt, PUT summary. + +set -euo pipefail + +MODE="$1" +OWNER="$2" +REPO="$3" +BRANCH="$4" +RUN_ID="$5" +AGENT_INDEX="${6:-0}" +AGENT_COUNT="${7:-0}" +TOPIC="${8:-}" + +api_url() { + printf 'https://api.github.com%s' "$1" +} + +github_request() { + local method="$1" + local path="$2" + local output="$3" + shift 3 + curl -sS \ + -X "$method" \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${DEMO_GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "$@" \ + -o "$output" \ + -w "%{http_code}" \ + "$(api_url "$path")" +} + +render_template() { + local template="$1" + local slice="${2:-}" + node - "$template" "$AGENT_INDEX" "$AGENT_COUNT" "$TOPIC" "$slice" <<'NODE' +const fs = require("fs"); +const [templatePath, agentIndex, agentCount, topic, slice] = process.argv.slice(2); +let text = fs.readFileSync(templatePath, "utf8"); +text = text.replace(/^(?:\n)+\n?/, ""); +const replacements = { + "{{AGENT_INDEX}}": agentIndex, + "{{AGENT_COUNT}}": agentCount, + "{{TOPIC}}": topic, + "{{SLICE}}": slice, +}; +for (const [needle, value] of Object.entries(replacements)) { + text = text.split(needle).join(value); +} +process.stdout.write(text); +NODE +} + +bootstrap_codex_oauth() { + mkdir -p "$HOME/.codex" + node - <<'NODE' +const fs = require("fs"); +const path = `${process.env.HOME}/.codex/auth.json`; +const b64u = (obj) => Buffer.from(JSON.stringify(obj)).toString("base64url"); +const now = Math.floor(Date.now() / 1000); +const fakeIdToken = [ + b64u({ alg: "none", typ: "JWT" }), + b64u({ + iss: "https://auth.openai.com", + aud: "codex", + sub: "openshell-placeholder", + email: "placeholder@example.com", + iat: now, + exp: now + 3600, + }), + "placeholder", +].join("."); + +fs.writeFileSync(path, JSON.stringify({ + auth_mode: "chatgpt", + OPENAI_API_KEY: null, + tokens: { + id_token: fakeIdToken, + access_token: process.env.CODEX_AUTH_ACCESS_TOKEN, + refresh_token: process.env.CODEX_AUTH_REFRESH_TOKEN, + account_id: process.env.CODEX_AUTH_ACCOUNT_ID, + }, + last_refresh: new Date().toISOString(), +}, null, 2)); +NODE + chmod 600 "$HOME/.codex/auth.json" +} + +put_contents() { + local repo_path="$1" + local source_file="$2" + local message="$3" + local get_body put_body put_response status sha + local attempt=0 + local max_attempts=6 + local sleep_secs=1 + get_body="$(mktemp)" + put_body="$(mktemp)" + put_response="$(mktemp)" + + while (( attempt < max_attempts )); do + attempt=$((attempt + 1)) + + status="$(github_request GET "/repos/${OWNER}/${REPO}/contents/${repo_path}?ref=${BRANCH}" "$get_body")" + if [[ "$status" == "200" ]]; then + sha="$(node -e 'const fs=require("fs"); const p=JSON.parse(fs.readFileSync(process.argv[1],"utf8")); process.stdout.write(p.sha || "");' "$get_body")" + elif [[ "$status" == "404" ]]; then + sha="" + else + echo "GitHub GET ${repo_path} failed with HTTP ${status}" >&2 + cat "$get_body" >&2 + return 1 + fi + + node - "$source_file" "$message" "$BRANCH" "$sha" > "$put_body" <<'NODE' +const fs = require("fs"); +const [file, message, branch, sha] = process.argv.slice(2); +const body = { + message, + branch, + content: fs.readFileSync(file).toString("base64"), +}; +if (sha) body.sha = sha; +process.stdout.write(JSON.stringify(body)); +NODE + + status="$(github_request PUT "/repos/${OWNER}/${REPO}/contents/${repo_path}" "$put_response" --data-binary "@${put_body}")" + if [[ "$status" == "200" || "$status" == "201" ]]; then + return 0 + fi + + # Retry on 409 (concurrent writer) and 5xx (transient server error) + # with decorrelated jitter so racing workers don't retry in lockstep. + if [[ "$status" =~ ^(409|5[0-9][0-9])$ && $attempt -lt $max_attempts ]]; then + sleep "$(( sleep_secs + RANDOM % sleep_secs ))" + sleep_secs=$((sleep_secs * 2)) + continue + fi + + echo "GitHub PUT ${repo_path} failed with HTTP ${status} (attempt ${attempt}/${max_attempts})" >&2 + cat "$put_response" >&2 + return 1 + done +} + +get_contents_file() { + local repo_path="$1" + local destination="$2" + local body status + body="$(mktemp)" + status="$(github_request GET "/repos/${OWNER}/${REPO}/contents/${repo_path}?ref=${BRANCH}" "$body")" + if [[ "$status" != "200" ]]; then + echo "GitHub GET ${repo_path} failed with HTTP ${status}" >&2 + cat "$body" >&2 + return 1 + fi + node - "$body" "$destination" <<'NODE' +const fs = require("fs"); +const [bodyPath, destination] = process.argv.slice(2); +const body = JSON.parse(fs.readFileSync(bodyPath, "utf8")); +fs.writeFileSync(destination, Buffer.from((body.content || "").replace(/\s/g, ""), "base64")); +NODE +} + +run_codex_to_file() { + local prompt_file="$1" + local output_file="$2" + codex exec \ + --skip-git-repo-check \ + --sandbox read-only \ + --ephemeral \ + --output-last-message "$output_file" \ + "$(cat "$prompt_file")" +} + +worker() { + local slice="$1" + local prompt_file output_file repo_path + bootstrap_codex_oauth + prompt_file="$(mktemp)" + output_file="$(mktemp)" + repo_path="runs/${RUN_ID}/notes/agent-${AGENT_INDEX}.md" + + render_template /sandbox/payload/prompts/worker.md "$slice" > "$prompt_file" + run_codex_to_file "$prompt_file" "$output_file" + put_contents "$repo_path" "$output_file" "Add agent ${AGENT_INDEX} note for ${RUN_ID}" + printf 'wrote %s\n' "$repo_path" +} + +synthesis() { + local notes_dir prompt_file output_file repo_path + bootstrap_codex_oauth + notes_dir="$(mktemp -d)" + prompt_file="$(mktemp)" + output_file="$(mktemp)" + repo_path="runs/${RUN_ID}/summary.md" + + for i in $(seq 1 "$AGENT_COUNT"); do + get_contents_file "runs/${RUN_ID}/notes/agent-${i}.md" "${notes_dir}/agent-${i}.md" + done + + render_template /sandbox/payload/prompts/synthesis.md "" > "$prompt_file" + { + printf '\n\n## Worker Notes\n\n' + for i in $(seq 1 "$AGENT_COUNT"); do + printf '\n\n---\n\n' + cat "${notes_dir}/agent-${i}.md" + done + } >> "$prompt_file" + + run_codex_to_file "$prompt_file" "$output_file" + put_contents "$repo_path" "$output_file" "Add multi-agent summary for ${RUN_ID}" + printf 'wrote %s\n' "$repo_path" +} + +case "$MODE" in + worker) worker "${9:-}" ;; + synthesis) synthesis ;; + *) echo "unknown mode: $MODE" >&2; exit 2 ;; +esac From 8e8274d90e2931596f63366e9ce02229c6e4c83e Mon Sep 17 00:00:00 2001 From: Alexander Watson Date: Tue, 5 May 2026 07:57:53 -0700 Subject: [PATCH 2/2] docs updates --- examples/multi-agent-notepad/README.md | 2 +- examples/multi-agent-notepad/demo.sh | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/examples/multi-agent-notepad/README.md b/examples/multi-agent-notepad/README.md index a98acb8159..5cc1b3819d 100644 --- a/examples/multi-agent-notepad/README.md +++ b/examples/multi-agent-notepad/README.md @@ -13,7 +13,7 @@ reports, decision logs — need somewhere durable to write them. The store has to survive individual sandbox death, accept many concurrent writers safely, and stay inspectable after the fact. GitHub provides all three for free: -- **Durable across sandbox death.** A sandbox can crash mid-run; the +- **Durable across sandbox restarts.** A sandbox can crash mid-run; the committed artifact persists. - **Concurrency control built in.** Every PUT must include the current file SHA, and the branch ref serializes commits. Racing writers see HTTP 409 diff --git a/examples/multi-agent-notepad/demo.sh b/examples/multi-agent-notepad/demo.sh index 09032c118b..7a2e4bf2f3 100755 --- a/examples/multi-agent-notepad/demo.sh +++ b/examples/multi-agent-notepad/demo.sh @@ -100,7 +100,9 @@ validate_env() { [[ "$DEMO_GITHUB_OWNER" =~ ^[A-Za-z0-9_.-]+$ ]] || fail "DEMO_GITHUB_OWNER contains unsupported characters" [[ "$DEMO_GITHUB_REPO" =~ ^[A-Za-z0-9_.-]+$ ]] || fail "DEMO_GITHUB_REPO contains unsupported characters" [[ "$DEMO_BRANCH" =~ ^[A-Za-z0-9._-]+$ ]] || fail "DEMO_BRANCH may contain only letters, numbers, '.', '_', and '-'" + info "GitHub repo ${DEMO_GITHUB_OWNER}/${DEMO_GITHUB_REPO}@${DEMO_BRANCH} and token present" + info "checking OpenShell gateway is reachable..." "$OPENSHELL_BIN" status >/dev/null 2>&1 || fail "OpenShell gateway is not reachable; run: openshell gateway start" export CODEX_AUTH_ACCESS_TOKEN @@ -113,6 +115,7 @@ validate_env() { [[ -n "$CODEX_AUTH_ACCESS_TOKEN" ]] || fail "local Codex sign-in is missing an access token; run: codex login" [[ -n "$CODEX_AUTH_REFRESH_TOKEN" ]] || fail "local Codex sign-in is missing a refresh token; run: codex login" [[ -n "$CODEX_AUTH_ACCOUNT_ID" ]] || fail "local Codex sign-in is missing an account id; run: codex login" + info "local Codex OAuth tokens loaded from ~/.codex/auth.json" } render_policy() { @@ -221,6 +224,7 @@ print_results() { } main() { + step "Validating Codex and GitHub credentials" validate_env render_policy write_runner