Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions .github/workflows/review-swarm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
name: Review swarm

on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]

permissions:
contents: read
pull-requests: write

concurrency:
group: review-swarm-${{ github.event.pull_request.number }}
cancel-in-progress: true

jobs:
review:
if: github.event.pull_request.draft == false
# Ordering invariant: swarm 60 min < poll 65 min < job 75 min (10 min buffer).
timeout-minutes: 75
runs-on: ubuntu-latest
steps:
- name: Check out PR head
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
path: pr-head
fetch-depth: 0

- name: Check out immutable gate from main
uses: actions/checkout@v4
with:
ref: main
path: main-gate
sparse-checkout: |
workflows/review-swarm.yaml
.github/workflows/scripts

- name: Validate cloud authentication
env:
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: |
if [ -z "$RELAY_WORKSPACE_KEY" ]; then
echo "RELAY_WORKSPACE_KEY secret not configured; see README § Cloud review swarm" >&2
exit 1
fi

- name: Install Agent Relay
run: npm install --global agent-relay@11.8.2

- name: Prepare PR inputs
working-directory: pr-head
env:
GH_TOKEN: ${{ github.token }}
run: ../main-gate/.github/workflows/scripts/swarm-prepare.sh "${{ github.event.pull_request.number }}" ../main-gate

- name: Launch cloud swarm
id: launch
working-directory: pr-head
env:
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: |
response=$(agent-relay cloud run .review-gate/review-swarm.yaml --sync-code --json)
run_id=$(jq -er '.runId // .run_id' <<<"$response")
echo "run_id=$run_id" >> "$GITHUB_OUTPUT"
echo "Launched cloud run $run_id"

- name: Wait for cloud swarm
id: wait
if: steps.launch.outputs.run_id != ''
env:
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: |
# Ordering invariant: swarm 3600s < this 3900s poll < 75 min job.
# This observer never gates directly: it always records status and exits 0.
set +e
deadline=$((SECONDS + 3900))
status=timed_out
while [ "$SECONDS" -lt "$deadline" ]; do
response=$(agent-relay cloud status "${{ steps.launch.outputs.run_id }}" --json 2>/dev/null) || response='{}'
status=$(jq -r '.status // "unknown"' <<<"$response" 2>/dev/null) || status=unknown
case "$status" in completed|failed|cancelled|interrupted) break ;; esac
sleep 15
done
echo "swarm_status=$status" >> "$GITHUB_OUTPUT"
echo "Cloud run ended with status: $status"
exit 0

- name: Post fresh transcripts and verdict
if: always() && steps.launch.outputs.run_id != ''
env:
GH_TOKEN: ${{ github.token }}
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: main-gate/.github/workflows/scripts/swarm-post.sh "${{ steps.launch.outputs.run_id }}" "${{ github.event.pull_request.number }}" "$GITHUB_WORKSPACE/pr-head"

- name: Enforce completed swarm
if: steps.wait.outputs.swarm_status != 'completed'
run: |
echo "Review swarm did not complete: ${{ steps.wait.outputs.swarm_status }}" >&2
exit 1
57 changes: 57 additions & 0 deletions .github/workflows/scripts/swarm-post.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
set -euo pipefail

run_id=${1:?usage: swarm-post.sh RUN_ID PR_NUMBER PR_TREE}
pr=${2:?usage: swarm-post.sh RUN_ID PR_NUMBER PR_TREE}
pr_tree=${3:?usage: swarm-post.sh RUN_ID PR_NUMBER PR_TREE}
script_dir=$(cd "$(dirname "$0")" && pwd)
sync_started=$(date +%s)

agent-relay cloud sync "$run_id" --dir "$pr_tree"
. "$script_dir/swarm-verdict.sh"
swarm_extract_verdicts "$pr_tree/ops/reviews" "$pr" "$sync_started"

upsert_comment() {
local anchor=$1 body_file=$2 comment_id
comment_id=$(gh api --paginate "repos/{owner}/{repo}/issues/$pr/comments" \
--jq ".[] | select(.body | contains(\"$anchor\")) | .id" | head -n 1)
if [ -n "$comment_id" ]; then
gh api --method PATCH "repos/{owner}/{repo}/issues/comments/$comment_id" \
--raw-field "body=$(cat "$body_file")" >/dev/null
else
gh api --method POST "repos/{owner}/{repo}/issues/$pr/comments" \
--raw-field "body=$(cat "$body_file")" >/dev/null
fi
}

tmp_dir=$(mktemp -d)
trap 'rm -rf "$tmp_dir"' EXIT
for lens in maintainability history structure; do
upper=$(printf '%s' "$lens" | tr '[:lower:]' '[:upper:]')
eval "file=\${SWARM_${upper}_FILE}"
eval "verdict=\${SWARM_${upper}_VERDICT}"
body="$tmp_dir/$lens.md"
{
echo "<!-- swarm-lens: $lens -->"
echo "### Review swarm: $lens — $verdict"
echo
if [ -n "$file" ] && [ "$verdict" != STALE ]; then cat "$file"; else echo "No fresh transcript was produced for cloud run \`$run_id\`."; fi
} > "$body"
upsert_comment "<!-- swarm-lens: $lens -->" "$body"
done

marker="$tmp_dir/marker.md"
{
echo '<!-- review-swarm -->'
echo "### Review swarm: $SWARM_OVERALL"
echo
echo "Cloud run: \`$run_id\`"
for lens in maintainability history structure; do
upper=$(printf '%s' "$lens" | tr '[:lower:]' '[:upper:]')
eval "verdict=\${SWARM_${upper}_VERDICT}"
echo "- $lens: $verdict"
done
} > "$marker"
upsert_comment '<!-- review-swarm -->' "$marker"

[ "$SWARM_OVERALL" = PASSED ]
16 changes: 16 additions & 0 deletions .github/workflows/scripts/swarm-prepare.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
set -euo pipefail

pr=${1:?usage: swarm-prepare.sh PR_NUMBER GATE_DIR}
gate_dir=${2:?usage: swarm-prepare.sh PR_NUMBER GATE_DIR}

case "$pr" in *[!0-9]*|'') echo "PREPARE_FAILED: invalid PR number" >&2; exit 2 ;; esac
mkdir -p .review-target .review-gate
printf '%s\n' "$pr" > .review-target/pr-number
gh pr view "$pr" --json headRefName,headRefOid,title,url > .review-target/pr.json
gh pr diff "$pr" > .review-target/pr.diff
cp "$gate_dir/workflows/review-swarm.yaml" .review-gate/review-swarm.yaml
cp "$gate_dir/.github/workflows/scripts/swarm-verdict.sh" .review-gate/swarm-verdict.sh
git add -f .review-target/pr-number .review-target/pr.json .review-target/pr.diff \
.review-gate/review-swarm.yaml .review-gate/swarm-verdict.sh
echo "PREPARED PR #$pr ($(wc -l < .review-target/pr.diff) diff lines)"
36 changes: 36 additions & 0 deletions .github/workflows/scripts/swarm-verdict.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash

# Set SWARM_<LENS>_{FILE,VERDICT} and SWARM_OVERALL from persisted reviews.
# Filenames are timestamps, so lexical order identifies the newest transcript.
swarm_extract_verdicts() {
local reviews_dir=$1 pr=$2 min_mtime=${3:-0}
local lens upper file line mtime

SWARM_OVERALL=PASSED
for lens in maintainability history structure; do
upper=$(printf '%s' "$lens" | tr '[:lower:]' '[:upper:]')
file=$(find "$reviews_dir" -maxdepth 1 -type f \
-name "*-pr${pr}-${lens}.md" -printf '%f\n' 2>/dev/null | sort | tail -n 1)
if [ -z "$file" ]; then
line=MISSING
file=
else
file="$reviews_dir/$file"
mtime=$(stat -c %Y "$file")
if [ "$mtime" -lt "$min_mtime" ]; then
line=STALE
else
line=$(sed '/^[[:space:]]*$/d' "$file" | tail -n 1 | tr -d '\r')
line=$(printf '%s' "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
case "$line" in
REVIEW_PASSED) line=PASSED ;;
REVIEW_FAILED) line=FAILED ;;
*) line=UNCLEAR ;;
esac
fi
fi
eval "SWARM_${upper}_FILE=\$file"
eval "SWARM_${upper}_VERDICT=\$line"
[ "$line" = PASSED ] || SWARM_OVERALL=FAILED
done
}
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ dist/
.env
.agentworkforce/
.cargo-home/
.review-target

# Toolchains materialize inside the workspace in a cloud sandbox and must never
# be committed or delivered. Run f18ec684's patch carried .rustup-home/ files;
# ops/deliver-run.sh scrubs them too, but ignoring them is the durable fix.
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,11 @@ Nine gates, in `docs/RFC-0001` §3. Gate 1 first: a relayflow can run — the he
ladder survives `kill -9` at every boundary.

Private while we build. YC 2026-09-15 runs on this base.

## Cloud review swarm

The `review-swarm` GitHub Actions workflow requires a repository Actions secret
named `RELAY_WORKSPACE_KEY`. Obtain the key on a trusted machine with
`agent-relay workspace key`, then add its non-empty output under **Settings →
Secrets and variables → Actions → New repository secret**. The workflow checks
the secret before launching and fails closed when it is absent.
129 changes: 67 additions & 62 deletions ops/NEXT.md
Original file line number Diff line number Diff line change
@@ -1,87 +1,92 @@
# NEXT — work package for this tick

**Scope:** Build a minimal agent worker in the SDK. CODE task, SDK-side.

This run is pinned to **gate 3** and must not work on any other gate.
**Scope (from ops/TARGET.md):** **Track D: Cloud review-swarm redesign** — build `.github/workflows/review-swarm.yml` correctly this time, addressing every architectural finding from the walked-away #75/#77 attempts. Parallel to Track A (hn-monitor); different territory (`.github/` + `workflows/` — no overlap with `sdk/` work).

## Objective

Promote the throwaway worker the tests already build into a real SDK component
that can execute agent steps by running their declared CLI as a subprocess.
Build the cloud-hosted review swarm enforcement system that meets every PR with three independent reviewers (maintainability, history, structure), addressing all 9 non-negotiable requirements from prior PR rejections. This makes RFC-0001 §2 rule 7 ("every PR met by a review swarm") enforcement durable instead of laptop-dependent.

## Files in scope

## Context
- `.github/workflows/review-swarm.yml` — GitHub Actions trigger (create new)
- `.github/workflows/scripts/swarm-post.sh` — sync + verdict + post script (create new)
- `.github/workflows/scripts/swarm-prepare.sh` — launcher-side PR fetcher (create new)
- `.github/workflows/scripts/swarm-verdict.sh` — shared verdict extraction logic (create new)
- `workflows/review-swarm.yaml` — aggregate step refactored to use shared verdict logic
- `.gitignore` — drop the `.review-target` mask
- `README.md` — document `RELAY_WORKSPACE_KEY` secret + how to obtain it

Nothing in this repo can execute an agent step. Searching for `workerAttach` /
`step.complete` finds only TESTS (`sdk/tests/live-kernel.test.ts`,
`journal-client.test.ts`, `journal-client-loopback.ts`) and the protocol
definitions. `sdk/src/cli/run.ts` only OBSERVES worker leases and waits for one
that never arrives.
## Definition of done

The kernel's dispatch, lease and claim machinery is real and tested. The worker
side of the protocol is simply unimplemented, and that is what blocks gate 2
("a workload RUNS as a relayflow" — today a run can only be shown CREATED) and
gate 3 ("every claim/lease/retry served by the kernel").
All of the following must pass:

`sdk/tests/live-kernel.test.ts` around the `live-manual-agent` case (line 288)
shows the whole shape: connect, `hello`, `workerAttach` with pins, receive
`step.dispatch`, act, complete. The protocol is already proven there.
1. **Syntax validation:**
```
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/review-swarm.yml'))"
python3 -c "import yaml; yaml.safe_load(open('workflows/review-swarm.yaml'))"
bash -n .github/workflows/scripts/swarm-post.sh
bash -n .github/workflows/scripts/swarm-prepare.sh
bash -n .github/workflows/scripts/swarm-verdict.sh
```

## Files in scope
2. **Immutable gate verified:** `.github/workflows/review-swarm.yml` contains two separate `actions/checkout@v4` steps with different `path:` values — one for PR head, one for main's gate files.

- `sdk/src/worker.ts` — new file, the worker implementation
- `sdk/src/index.ts` — export the worker
- `sdk/tests/live-kernel.test.ts` OR a new test file — add a test that runs a
real flow with an agent step end to end against a live `relayflowd`, with
this worker attached, and asserts the step reaches `done`.
3. **Unified verdict logic:** Either:
- `scripts/swarm-verdict.sh` exists and both `workflows/review-swarm.yaml` aggregate step AND `.github/workflows/scripts/swarm-post.sh` source it, OR
- `workflows/review-swarm.yaml` aggregate step is trivial and `.github/workflows/scripts/swarm-post.sh` does all extraction

## Definition of done
4. **Auth preflight exists:** `.github/workflows/review-swarm.yml` contains a preflight step that validates `RELAY_WORKSPACE_KEY` is set and non-empty before launching cloud run.

ALL of the following must hold:
5. **Sticky transcripts verified:** `.github/workflows/scripts/swarm-post.sh` uses `<!-- swarm-lens: <lens> -->` HTML anchors and finds-by-anchor before posting (not creating duplicate comments on every push).

1. The worker in `sdk/src/worker.ts`, exported from `sdk/src/index.ts`
6. **No author whitelist:** `.github/workflows/review-swarm.yml` contains no `if: github.event.pull_request.user.login == ...` condition.

2. A test that runs a real flow with an agent step end to end against a live
`relayflowd`, with this worker attached, and asserts the step reaches
`done`. `sdk/tests/live-kernel.test.ts` already starts a daemon — follow
that pattern.
7. **Timeout ordering documented:** Comments in the code show:
- `workflows/review-swarm.yaml` `timeoutMs: 3600000` (60 min)
- Wait step poll deadline: 3900s (65 min)
- Job `timeout-minutes: 75` (65 + 10 min buffer)

3. **The worker must attach BEFORE the run starts.** A run that finds no worker
parks, and attaching afterwards does not re-drive it — `run.resume` is what
picks a parked run back up. That contract is pinned in the live-kernel
suite; do not fight it.
8. **Wait/post structure verified:**
- Wait step records `swarm_status` output, always exits 0
- Post step has `if: always() && steps.launch.outputs.run_id != ''`
- Fail step has `if: steps.wait.outputs.swarm_status != 'completed'`

4. The worker must:
- attach for `agent` steps with the pins it holds
- on `step.dispatch`, run the step's declared `cli` as a subprocess
- report the result back through the existing protocol (`step.complete`, and
the failure path when the CLI exits nonzero)
- nothing speculative: no retries of its own, no scheduling, no LLM calls.
The kernel owns retry and lease policy — do not reimplement it.
9. **SDK tests green:**
```
cd sdk && npm test
```
(Must show all tests passing with exit 0)

5. `cd sdk && npm test` must be green. Run it and paste the literal command and
output tail showing test counts.
10. **Git status clean:**
```
git status --porcelain
```
(Must show only the 7 files in scope, all staged)

6. `cd kernel && sh ../ops/cargo.sh test` must be green. Run it and paste the
literal command and output tail showing test counts.
## Out of scope

7. EVERY new test confirmed to FAIL against current code, with the literal
failing output quoted in the summary.
- `sdk/` — Track A owns that; do not modify
- `kernel/` — gate 1 done, no changes
- `ops/*` — chief owns briefs and state; do not modify
- Any GHA workflow other than review-swarm.yml
- Actually TESTING the workflow in CI (requires human to set `RELAY_WORKSPACE_KEY` secret)
- Implementing the `scripts/swarm-verdict.sh` verdict extraction (requirement #2 allows aggregate to stay in yaml)
- The `.review-target/{pr-number,pr.diff,pr.json}` fetch mechanism (requirement #6) — defer to implementation

8. As your LAST action, run `git status --porcelain` and paste it.
## Requirements summary (all 9 must be satisfied)

## Explicitly OUT of scope
1. **Immutable gate:** Two checkout steps with different paths — PR head vs main's gate files
2. **Unified verdict logic:** One source of truth for verdict extraction (shared script OR yaml-only)
3. **Auth preflight:** Validate `RELAY_WORKSPACE_KEY` before launch, fail-fast if missing
4. **Sticky transcripts:** HTML anchors, find-before-post, no duplicates
5. **No author whitelist:** All PRs reviewed
6. **Cloud fetch pattern:** GHA runner fetches PR diff/metadata, stages to `.review-target/`, git add -f
7. **Timeout ordering:** Job > poll > swarm, documented with comments
8. **Wait/post/fail structure:** Transcripts posted even on rejection, fail step gates merge
9. **Transcript freshness:** Sub-guard against stale transcripts (mtime check OR run-id binding)

- LLM steps — not in the gate 3 scope
- Retry logic in the worker — the kernel owns retry policy
- Scheduling or lease management — the kernel owns lease policy
- Optimizations, abstractions, or speculative features
- Changes to the kernel
- Changes to existing tests (except adding new test cases)
- Work on any gate other than gate 3
## How this package fits gate 3

## If blocked
Gate 3's done-when (RFC-0001 §3): "the cloud review swarm enforces rule 7 for every PR, not just when my laptop is on." This package builds the `.github/workflows/review-swarm.yml` trigger that makes that true. The local `~/AgentWorkforce/review-swarm-loop.sh` currently enforces it, but ends when the laptop session ends. This moves enforcement to GitHub Actions + cloud sandbox.

If gate 3 is genuinely unreachable from the current state, write
ops/NEEDS_HUMAN.md saying exactly why and still end with ASSESS_DONE. Do not
silently substitute different work: a run that reports progress on the wrong
gate is worse than one that reports it is blocked.
Gate 3 will be AMBER after this lands (infrastructure exists) and GREEN when a real PR is reviewed by the cloud swarm and the transcripts + verdict reach the PR correctly.
2 changes: 1 addition & 1 deletion sdk/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading