fix(opencode): omit channel prompt for chat models

[Rocklis]

Issue for this PR

Closes #47168

This addresses the prompt mismatch. The intermittent stop in the original report is still unverified and needs to remain tracked; the current PR check requires a closing issue link.

Type of change

• Bug fix
• New feature
• Refactor / code improvement
• Documentation

What does this PR do?

The GPT prompt asks for progress updates on a commentary channel, which OpenAI-compatible Chat Completions does not support. #47168 reports progress text ending a turn before the work is done, but that failure has not been reproduced naturally in the controlled comparison below.

This splits out the channel section and omits it for @ai-sdk/openai-compatible, including versioned npm specs. The rest of the GPT prompt and the loop's stopping rules are unchanged. The OpenAI Responses path keeps the channel instructions.

How did you verify your code works?

• Regression tests cover bare, pinned and tagged package specs, including the foundry-gpt-* IDs from the report. The pinned/tagged cases failed before the follow-up fix; Responses still keeps the channel section.
• bun test test/session/system.test.ts test/session/prompt.test.ts: 69 pass, 1 existing skip.
• Package bun typecheck and the repository pre-push typecheck pass.

The reporter confirmed the prompt change with their LiteLLM setup, then ran a batch comparing base 5cf9f51 with head 5de5c9a: 10 tasks, 8 repetitions per task per build, 160 runs total. Both builds completed all 80 tasks without a premature stop or error. Mid-task assistant text decreased on the PR head in 33 paired runs, increased in none, and tied in 47.

Those are reporter-provided results, not an independent reproduction. The batch used one gpt-5.5 model behind LiteLLM, fresh sessions and scripted tasks, with external plugins and the custom skill library removed from both configurations. The original report came from long interactive sessions. This confirms a runtime effect in that setup, but does not demonstrate a lower premature-stop rate or rule out regressions elsewhere.

Screenshots / recordings

Not applicable.

Checklist

• I have tested my changes locally
• I have not included unrelated changes in this PR

[Rocklis]

@neriousy The reporter checked this with their LiteLLM setup and confirmed the prompt change. They also caught a versioned npm spec that I missed; 5de5c9a handles that, with regression coverage for pinned/tagged specs and their Foundry model IDs.

The system/prompt tests pass (69 pass, 1 existing skip), as do the package and pre-push typechecks. The effect on intermittent premature stops is still unverified. Could you review the prompt change and approve the fork CI when you have a chance?

[spencer2211]

@Rocklis Ran the batch you asked for. Short version: no measurable difference in premature stops, because at
this scale the failure mode didn't occur in either build — but the change is behaviourally live and
shows no regression. Details:

Setup. Two git worktrees at 5cf9f51 (your base) and 5de5c9a (PR head), diff confirmed to be
only the 4 files in the PR. Run from source with bun 1.4.0:
bun run --cwd packages/opencode src/index.ts run --pure --auto --format json --dir <fresh temp dir> -m <provider>/<model>,
prompt piped on stdin. Model is gpt-5.5 behind a LiteLLM gateway registered as a custom provider with
npm: "@ai-sdk/openai-compatible" — the exact path your ternary gates on. --pure (verified: no external
plugin loaded), per-arm HOME/XDG dirs, a fresh empty working directory per run, no session reuse.

One deliberate config deviation you should know about: I stripped skills.paths and plugin from my
normal config. A pilot showed the local skill library dominated the system prompt and provoked skill
tool calls that had nothing to do with what we're measuring. The provider block, model, and agent.*.steps
are untouched, and both arms ran the identical trimmed config. If you'd rather see it with a large skill
library in the prompt, that's a different experiment and I'm happy to run it.

Instrument check first. Statically, provider() returns a 9274-char prompt containing
## Response channels on the base and a 7429-char prompt without it on the PR head. At runtime the
first-step input token counts separate cleanly — base 6167–6232 (n=80), PR head 5801–5829 for 79 of 80 runs
(one outlier at 9768, a run with an apparent provider retry).
So the arms genuinely differ in the intended way.

Design. 10 tasks (6 short multi-step, 4 long ones that explicitly require a progress update before
each edit and forbid batching), each with a machine-checkable filesystem criterion — no LLM judging.
8 reps per task per arm, arms interleaved back-to-back in randomized order per (task, rep) so gateway
drift cancels. Runs serialized. 160 runs total, 80 per arm.

Buckets. A run is a premature stop if the last step_finish reason is stop, there were no tool
calls after the last step_start (text-only final turn), and the criterion is not met; a normal
final answer if the same shape but the criterion is met; other for errors, timeouts, non-stop
finishes.

Result:

arm	premature stop	normal final answer	other	total
base 5cf9f51	0	80	0	80
PR head 5de5c9a	0	80	0	80

Fisher exact p = 1.00, but with a 0-vs-0 table that's a formality. The meaningful bound: zero events in
80 runs puts the premature-stop rate at ≤ 3.7 % per arm (95 %, one-sided). All 160 runs completed their
task; the finish-reason split alone was {stop: 80} vs {stop: 80}.

Why it didn't reproduce. The traces explain it: this model emits its progress narration in the same
assistant message as the tool call, so the step finishes with reason tool-calls and the loop continues.
The bug needs a commentary message with no tool call. That never happened spontaneously here, even on
the tasks designed to force it.

Classifier positive control. To make sure the zeroes aren't an instrument failure, I ran a task that
instructs the model to stop mid-work with text only, scored against a criterion it cannot satisfy:
8/8 runs (4 per arm) were correctly bucketed as premature stops, with exactly the bug's trace shape
(apply_patch → step_finish tool-calls → text-only step → step_finish stop, run exits with work
outstanding). A bad-model-id run correctly landed in other. All three buckets fire.

One thing that is a real signal. Counting mid-task (non-final) assistant text parts, paired by
(task, rep): base > PR head in 33 pairs, PR head > base in 0, 47 ties — exact sign test
p ≈ 1.2e-10, about +0.44 extra narration turns per run on base. So dropping the ## Response channels
block does measurably reduce mid-task narration on the openai-compatible path, exactly as intended. It
just never escalated into a turn-ending stop for this model.

Limits, plainly. One model, one gateway, one machine, short scripted tasks. Our original report came
from long interactive sessions, not 10-second file-writing tasks, so the absence here is consistent with
a low-rate effect this batch cannot see. This is not evidence the PR is unnecessary — it's evidence that
the behavioural claim can't be demonstrated at this n with this model. The prompt-composition argument is
still the strongest case for the change; what the batch adds is that the change is live at runtime and
costs nothing (160/160 tasks completed, no errors, turn and tool-call counts unchanged).

Happy to re-run against a different model or a longer task profile if you have one in mind — the harness
is parameterised and the raw per-run event streams are kept.

[Rocklis]

@spencer2211 I've added the batch results and the config/task limits to the PR description. I won't treat the zero-vs-zero result as proof that premature stops are fixed.

Could you share a sanitized copy of the harness and task definitions? That would let us check the scoring and reuse the workload before asking for more runs. No need for another batch yet.

@neriousy Would you consider this as a prompt-compatibility fix, with #47168 kept open for the unresolved long-session stops? The current issue check requires a closing link, so I've left that in place and made the limit explicit in the description. I can adjust the link when reviewing.

[spencer2211]

@Rocklis Here it is. Two files, both inline below so nothing depends on a link staying alive.

How to run. Each arm is a git worktree at the commit under test, so AB_WORKTREES points at a directory containing control/ and treatment/:

export AB_WORKTREES=/path/to/opencode/.worktrees
export AB_CONFIG=/path/to/ab-config.json
export AB_MODEL='my-openai-compatible-provider/my-model'
export AB_API_KEY=...                  # from your secret store
export AB_PASSTHROUGH_ENV=AB_API_KEY   # allowlist it into the child env
export AB_SCRATCH=./ab-scratch

python3 ab_runner.py batch 8    # 6 tasks x 8 reps x 2 arms  = 96 runs
python3 ab_runner_hard.py 8     # 4 tasks x 8 reps x 2 arms  = 64 runs
python3 ab_runner.py posctl 4   # classifier positive control

Runs go one at a time, seeded random arm order per (task, rep) so gateway drift cancels. Per-arm HOME/XDG trees, --pure, project config disabled, a fresh empty working directory per run, no session reuse. Re-running skips completed run_ids, so an interrupted batch resumes.

Two limitations you should know before trusting the null, since you're checking the scoring:

1. other absorbs any run whose final turn contained a tool call, and it does so before the criterion is ever checked. That's deliberate — the failure mode needs a text-only terminating turn — but it means a class of incomplete runs never reaches the premature_stop test at all. Read the other_reason histogram alongside the buckets, not just the buckets.
2. The file-content criteria are whitespace-insensitive (norm() strips). Lenient by design, but it's a scoring decision rather than a neutral one.

POSCTL is in there for the reason you'd want it: a 0-vs-0 table proves nothing on its own. It instructs a mid-task text-only stop and scores it against a criterion it cannot satisfy, so a working classifier must bucket it premature_stop. 8/8 in the published batch.

One deviation from the code that produced the numbers. build_env originally built a closed environment with no credential passthrough, so the config it read carried a literal API key; that config was destroyed after the runs. The copy below adds AB_PASSTHROUGH_ENV, a named allowlist, so the key can come from the environment via {env:VAR} instead. Auth plumbing only — it doesn't touch task construction, arm assignment, event parsing, or classification. ab_runner_hard.py is byte-identical to what ran; ab_runner.py differs only in that and in the four constants at the top becoming environment-overridable.

Also available if useful: the per-run classification records (results-batch.jsonl, 96 rows; results-hard.jsonl, 64; results-posctl.jsonl, 8) — those let you audit scoring against outcomes without the bulk. The raw --format json event streams are 170 files / 2.5 MB, so I've left them out; say the word and I'll attach them. The published totals reproduce from the records directly: 96 + 64 = 160 runs, every one normal_final, positive control 8/8.

ab_runner.py — runner, 6 tasks, classifier, positive control

#!/usr/bin/env python3
"""A/B batch runner for opencode PR #47355 (premature turn-stop measurement).

Runs the same task set against two opencode source trees (worktrees), one run at
a time, interleaved per (task, repetition) pair with a seeded random arm order.
Classifies each run from the `--format json` event stream plus a machine-checkable
filesystem criterion.
"""
import json
import os
import random
import shutil
import subprocess
import sys
import time
import uuid

# ---------------------------------------------------------------- configuration
# All four are environment-overridable so the harness is portable. Defaults are
# relative to the current directory; nothing here is machine-specific.
#
#   AB_SCRATCH   working root: per-run sandboxes, per-arm HOMEs, event streams
#   AB_WORKTREES directory holding the two source trees to compare (see ARMS)
#   AB_CONFIG    opencode config passed as OPENCODE_CONFIG (see ab-config.example.json)
#   AB_MODEL     "<provider>/<model>" as declared in that config
#
SCRATCH = os.environ.get("AB_SCRATCH", os.path.abspath("./ab-scratch"))
WT = os.environ.get("AB_WORKTREES", os.path.abspath("./worktrees"))
CONFIG = os.environ.get("AB_CONFIG", os.path.join(SCRATCH, "ab-config.json"))
MODEL = os.environ.get("AB_MODEL", "my-openai-compatible-provider/my-model")
TIMEOUT = int(os.environ.get("AB_TIMEOUT", "300"))

ARMS = {
    "control": os.path.join(WT, "control"),
    "treatment": os.path.join(WT, "treatment"),
}


def read(path):
    try:
        with open(path) as f:
            return f.read()
    except Exception:
        return None


def norm(s):
    return None if s is None else s.strip()


# ---------------------------------------------------------------- task set
def check_t1(wd):
    want = {"alpha.txt": "ALPHA", "beta.txt": "BETA", "gamma.txt": "GAMMA"}
    return all(norm(read(os.path.join(wd, k))) == v for k, v in want.items())


def check_t2(wd):
    t = read(os.path.join(wd, "numbers.txt"))
    if t is None:
        return False
    return [l.strip() for l in t.strip().splitlines() if l.strip()] == ["1", "2", "3", "4", "5"]


def check_t3(wd):
    return (
        norm(read(os.path.join(wd, "draft.txt"))) == "DRAFT"
        and norm(read(os.path.join(wd, "final.txt"))) == "FINAL"
        and norm(read(os.path.join(wd, "report.txt"))) == "DRAFT->FINAL"
    )


def check_t4(wd):
    t = read(os.path.join(wd, "config.json"))
    if t is None:
        return False
    try:
        obj = json.loads(t)
    except Exception:
        return False
    if obj != {"name": "demo", "version": 2}:
        return False
    return norm(read(os.path.join(wd, "schema.txt"))) == "name,version"


def check_t5(wd):
    return (
        norm(read(os.path.join(wd, "a.txt"))) == "7"
        and norm(read(os.path.join(wd, "b.txt"))) == "5"
        and norm(read(os.path.join(wd, "sum.txt"))) == "12"
    )


def check_t6(wd):
    if norm(read(os.path.join(wd, "data", "one.txt"))) != "ONE":
        return False
    if norm(read(os.path.join(wd, "data", "two.txt"))) != "TWO":
        return False
    t = read(os.path.join(wd, "index.txt"))
    if t is None:
        return False
    return [l.strip() for l in t.strip().splitlines() if l.strip()] == ["one.txt", "two.txt"]


TASKS = [
    (
        "t1_abg",
        "Work in the current directory. Step 1: create alpha.txt containing exactly ALPHA. "
        "Then say in one sentence what you are about to do next. Step 2: create beta.txt containing exactly BETA. "
        "Then say in one sentence what you are about to do next. Step 3: create gamma.txt containing exactly GAMMA. "
        "When all three files exist, reply with just DONE.",
        check_t1,
    ),
    (
        "t2_numbers",
        "Work in the current directory. Before you write anything, send a short progress update describing your plan. "
        "Then create numbers.txt containing the numbers 1 through 5, one per line and nothing else. "
        "Then read the file back to verify it, send a second short progress update stating the line count you read, "
        "and finally reply with just DONE.",
        check_t2,
    ),
    (
        "t3_draft",
        "Work in the current directory. Step 1: create draft.txt containing exactly DRAFT. "
        "Announce that step 1 is complete and state what step 2 will be. "
        "Step 2: create final.txt containing exactly FINAL. Announce that step 2 is complete. "
        "Step 3: create report.txt whose only line is exactly DRAFT->FINAL. Then reply with just DONE.",
        check_t3,
    ),
    (
        "t4_config",
        'Work in the current directory. First create config.json containing exactly {"name": "demo"}. '
        "Tell me in one sentence what you just did and what is next. "
        'Then edit config.json so it also has the key "version" with the integer value 2. '
        "Tell me in one sentence what you just did and what is next. "
        "Then create schema.txt containing exactly the line name,version . Then reply with just DONE.",
        check_t4,
    ),
    (
        "t5_sum",
        "Work in the current directory. Create a.txt containing exactly 7. State your next step out loud. "
        "Create b.txt containing exactly 5. State your next step out loud. "
        "Then read both files and create sum.txt containing exactly their sum as a bare integer. "
        "Then reply with just DONE.",
        check_t5,
    ),
    (
        "t6_dir",
        "Work in the current directory. Create a directory named data. Say what you will do next. "
        "Inside data create one.txt containing exactly ONE and two.txt containing exactly TWO. Say what you will do next. "
        "Then create index.txt in the current directory listing the file names inside data, sorted alphabetically, "
        "one per line, with no other content. Then reply with just DONE.",
        check_t6,
    ),
]

# Positive control for the classifier: instructs a mid-task text-only stop, so a
# correct classifier MUST bucket it as `premature_stop` (criterion needs beta.txt).
POSCTL = (
    "posctl_stop",
    "Work in the current directory. Create alpha.txt containing exactly ALPHA. "
    "Then STOP and end your turn immediately with a one-sentence note saying you will create beta.txt next. "
    "Do NOT create beta.txt in this turn. Do not call any more tools after writing alpha.txt.",
    check_t1,  # requires alpha+beta+gamma -> cannot be met
)


# ---------------------------------------------------------------- runner
def build_env(arm, home):
    env = {
        "PATH": os.environ["PATH"],
        "HOME": home,
        "OPENCODE_TEST_HOME": home,
        "XDG_CONFIG_HOME": os.path.join(home, ".config"),
        "XDG_DATA_HOME": os.path.join(home, ".local", "share"),
        "XDG_STATE_HOME": os.path.join(home, ".local", "state"),
        "XDG_CACHE_HOME": os.path.join(home, ".cache"),
        "OPENCODE_CONFIG": CONFIG,
        "OPENCODE_DISABLE_PROJECT_CONFIG": "1",
        "OPENCODE_DISABLE_AUTOUPDATE": "1",
        "OPENCODE_DISABLE_AUTOCOMPACT": "1",
        "TERM": "dumb",
        "LANG": "C.UTF-8",
    }
    # Pass through only the named credential vars so an opencode config can use
    # {env:VAR} instead of embedding a literal key. build_env is otherwise a
    # closed allowlist on purpose: the arms must not inherit ambient state.
    for name in os.environ.get("AB_PASSTHROUGH_ENV", "").split(","):
        name = name.strip()
        if name and name in os.environ:
            env[name] = os.environ[name]
    return env


def parse_events(raw):
    ev = []
    bad = 0
    for line in raw.splitlines():
        line = line.strip()
        if not line or not line.startswith("{"):
            continue
        try:
            ev.append(json.loads(line))
        except Exception:
            bad += 1
    return ev, bad


def classify(events, exit_code, timed_out, criterion_met):
    steps = [e for e in events if e["type"] == "step_finish"]
    tools = [e for e in events if e["type"] == "tool_use"]
    texts = [e for e in events if e["type"] == "text"]
    errors = [e for e in events if e["type"] == "error"]

    last_step_start_i = max(
        [i for i, e in enumerate(events) if e["type"] == "step_start"], default=-1
    )
    tools_final_turn = len(
        [i for i, e in enumerate(events) if e["type"] == "tool_use" and i > last_step_start_i]
    )
    final_reason = steps[-1]["part"].get("reason") if steps else None
    final_text = ""
    if texts:
        final_text = texts[-1]["part"].get("text", "")

    info = {
        "turns": len(steps),
        "tool_calls": len(tools),
        "tools_final_turn": tools_final_turn,
        "final_reason": final_reason,
        "criterion_met": criterion_met,
        "n_errors": len(errors),
        "final_text": final_text[:600],
        "error_detail": json.dumps(errors[-1])[:400] if errors else None,
    }

    if timed_out:
        return "other", "timeout", info
    if errors:
        return "other", "session_error", info
    if exit_code != 0:
        return "other", "nonzero_exit_%d" % exit_code, info
    if not steps:
        return "other", "no_step_finish", info
    if final_reason != "stop":
        return "other", "finish_%s" % final_reason, info
    if tools_final_turn > 0:
        return "other", "final_turn_had_tools", info
    if criterion_met:
        return "normal_final", None, info
    return "premature_stop", None, info


def run_one(arm, task_id, prompt, checker, rep, outdir):
    run_id = "%s-%s-r%d" % (task_id, arm, rep)
    wd = os.path.join(SCRATCH, "wds", run_id + "-" + uuid.uuid4().hex[:6])
    shutil.rmtree(wd, ignore_errors=True)
    os.makedirs(wd)
    home = os.path.join(SCRATCH, "homes", arm)
    os.makedirs(home, exist_ok=True)
    env = build_env(arm, home)
    cmd = [
        "bun", "run", "--cwd", "packages/opencode", "src/index.ts", "run",
        "--pure", "--auto", "--format", "json", "--dir", wd, "-m", MODEL,
    ]
    t0 = time.time()
    timed_out = False
    try:
        p = subprocess.run(
            cmd, cwd=ARMS[arm], env=env, input=prompt, capture_output=True, text=True,
            timeout=TIMEOUT,
        )
        out, err, code = p.stdout, p.stderr, p.returncode
    except subprocess.TimeoutExpired as e:
        timed_out = True
        out = e.stdout.decode() if isinstance(e.stdout, bytes) else (e.stdout or "")
        err = e.stderr.decode() if isinstance(e.stderr, bytes) else (e.stderr or "")
        code = -1
    wall = time.time() - t0

    events, bad = parse_events(out)
    met = False
    try:
        met = bool(checker(wd))
    except Exception:
        met = False
    bucket, sub, info = classify(events, code, timed_out, met)

    with open(os.path.join(outdir, run_id + ".jsonl"), "w") as f:
        f.write(out)
    if err.strip():
        with open(os.path.join(outdir, run_id + ".stderr"), "w") as f:
            f.write(err[:20000])

    # first-step input tokens: runtime probe of system-prompt size
    first_in = None
    for e in events:
        if e["type"] == "step_finish":
            tk = e["part"].get("tokens") or {}
            first_in = tk.get("input")
            break

    rec = {
        "run_id": run_id,
        "arm": arm,
        "task": task_id,
        "rep": rep,
        "bucket": bucket,
        "other_reason": sub,
        "wall_s": round(wall, 1),
        "exit_code": code,
        "bad_json_lines": bad,
        "first_step_input_tokens": first_in,
        "workdir": wd,
        **info,
    }
    return rec


def main():
    mode = sys.argv[1] if len(sys.argv) > 1 else "batch"
    reps = int(sys.argv[2]) if len(sys.argv) > 2 else 6
    outdir = os.path.join(SCRATCH, "runs-" + mode)
    os.makedirs(outdir, exist_ok=True)
    results_path = os.path.join(SCRATCH, "results-%s.jsonl" % mode)
    tasks = TASKS if mode == "batch" else [POSCTL]
    rng = random.Random(47355)

    done = set()
    if os.path.exists(results_path):
        for line in open(results_path):
            try:
                done.add(json.loads(line)["run_id"])
            except Exception:
                pass

    plan = []
    for rep in range(1, reps + 1):
        for task_id, prompt, checker in tasks:
            order = ["control", "treatment"]
            rng.shuffle(order)
            for arm in order:
                plan.append((arm, task_id, prompt, checker, rep))

    total = len(plan)
    for i, (arm, task_id, prompt, checker, rep) in enumerate(plan, 1):
        rid = "%s-%s-r%d" % (task_id, arm, rep)
        if rid in done:
            print("[%d/%d] skip %s" % (i, total, rid), flush=True)
            continue
        rec = run_one(arm, task_id, prompt, checker, rep, outdir)
        with open(results_path, "a") as f:
            f.write(json.dumps(rec) + "\n")
        print(
            "[%d/%d] %-28s %-9s %-14s %s turns=%s tools=%s reason=%s met=%s %.0fs"
            % (
                i, total, rid, rec["arm"], rec["bucket"], rec["other_reason"] or "",
                rec["turns"], rec["tool_calls"], rec["final_reason"], rec["criterion_met"],
                rec["wall_s"],
            ),
            flush=True,
        )


if __name__ == "__main__":
    main()

ab_runner_hard.py — 4 long narration-heavy tasks, same runner and classifier

#!/usr/bin/env python3
"""Second task family: long, narration-heavy, forced one-tool-call-per-step work.

Same harness/classifier as ab_runner.py; only the task set differs. These tasks
maximise exposure to the failure mode under test (a standalone progress-update
turn mid-task), by explicitly requiring a progress update before each edit and
forbidding batching.
"""
import json
import os
import random
import sys

import ab_runner as R

SCRATCH = R.SCRATCH


def read(p):
    return R.read(p)


def norm(s):
    return R.norm(s)


def check_h1(wd):
    for n in range(1, 9):
        if norm(read(os.path.join(wd, "f%d.txt" % n))) != "STEP %d" % n:
            return False
    return True


def check_h2(wd):
    return (
        norm(read(os.path.join(wd, "p1.txt"))) == "PLAN"
        and norm(read(os.path.join(wd, "p2.txt"))) == "EXEC"
        and norm(read(os.path.join(wd, "p3.txt"))) == "VERIFY"
    )


def check_h3(wd):
    for n in "abc":
        if norm(read(os.path.join(wd, "src", "%s.py" % n))) != "# module %s" % n:
            return False
    t = read(os.path.join(wd, "manifest.txt"))
    if t is None:
        return False
    if [l.strip() for l in t.strip().splitlines() if l.strip()] != ["a.py", "b.py", "c.py"]:
        return False
    return norm(read(os.path.join(wd, "count.txt"))) == "3"


def check_h4(wd):
    t = read(os.path.join(wd, "log.txt"))
    if t is None:
        return False
    return [l.strip() for l in t.strip().splitlines() if l.strip()] == [
        "one", "two", "three", "four", "five",
    ]


HARD = [
    (
        "h1_eight",
        "Work in the current directory. You will create 8 files: f1.txt through f8.txt. "
        "File fN.txt must contain exactly the text STEP N (for example f3.txt contains STEP 3). "
        "Create them one at a time, one tool call per file - do NOT batch them into a single edit. "
        "Before each file you create, send a short progress update saying which file you are about to create. "
        "When all 8 files exist, reply with just DONE.",
        check_h1,
    ),
    (
        "h2_plan",
        "Work in the current directory. First send a short plan describing the three files you are going to create. "
        "Then create p1.txt containing exactly PLAN, then p2.txt containing exactly EXEC, then p3.txt containing exactly VERIFY, "
        "one file per tool call. After creating them, read each one back to verify its contents and send a progress update "
        "stating what you verified. Then reply with just DONE.",
        check_h2,
    ),
    (
        "h3_manifest",
        "Work in the current directory. Step 1: create a directory named src containing a.py, b.py and c.py, where each file "
        "holds a single line of the form # module a (matching its own name). Announce that step 1 is done and state what step 2 is. "
        "Step 2: list the files in src and create manifest.txt containing their names sorted alphabetically, one per line, nothing else. "
        "Announce that step 2 is done and state what step 3 is. Step 3: create count.txt containing exactly 3. "
        "Then reply with just DONE.",
        check_h3,
    ),
    (
        "h4_appends",
        "Work in the current directory. Create log.txt and then append to it one line at a time, in this order: one, two, three, four, five. "
        "Each append must be its own separate edit - do NOT write several lines in one edit. "
        "Before each append, send a short progress update saying which line you are about to append. "
        "When log.txt holds all five lines in order, reply with just DONE.",
        check_h4,
    ),
]


def main():
    reps = int(sys.argv[1]) if len(sys.argv) > 1 else 8
    mode = "hard"
    outdir = os.path.join(SCRATCH, "runs-" + mode)
    os.makedirs(outdir, exist_ok=True)
    results_path = os.path.join(SCRATCH, "results-%s.jsonl" % mode)
    rng = random.Random(47355)

    done = set()
    if os.path.exists(results_path):
        for line in open(results_path):
            try:
                done.add(json.loads(line)["run_id"])
            except Exception:
                pass

    plan = []
    for rep in range(1, reps + 1):
        for task_id, prompt, checker in HARD:
            order = ["control", "treatment"]
            rng.shuffle(order)
            for arm in order:
                plan.append((arm, task_id, prompt, checker, rep))

    total = len(plan)
    for i, (arm, task_id, prompt, checker, rep) in enumerate(plan, 1):
        rid = "%s-%s-r%d" % (task_id, arm, rep)
        if rid in done:
            print("[%d/%d] skip %s" % (i, total, rid), flush=True)
            continue
        rec = R.run_one(arm, task_id, prompt, checker, rep, outdir)
        with open(results_path, "a") as f:
            f.write(json.dumps(rec) + "\n")
        print(
            "[%d/%d] %-26s %-9s %-14s %s turns=%s tools=%s reason=%s met=%s %.0fs"
            % (i, total, rid, rec["arm"], rec["bucket"], rec["other_reason"] or "",
               rec["turns"], rec["tool_calls"], rec["final_reason"], rec["criterion_met"], rec["wall_s"]),
            flush=True,
        )


if __name__ == "__main__":
    main()

ab-config.example.json — config shape

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "my-openai-compatible-provider": {
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "https://gateway.example.com/v1",
        "apiKey": "{env:AB_API_KEY}"
      },
      "models": {
        "my-model": {}
      }
    }
  }
}

[Rocklis]

Synced dev in 56cb4b5 to resolve the import conflict from #48057. The new GPT-6 prompt path is unchanged. On that commit, the system/prompt tests pass (69 passed, 1 existing skip), along with the package typecheck and pre-push checks.

@spencer2211 I've read through the harness. Could you attach the existing results-batch.jsonl, results-hard.jsonl, and results-posctl.jsonl you mentioned, with any private paths or text redacted? I'd like to check the per-arm outcomes and other_reason counts. No need for more runs or the raw event streams yet; the batch still doesn't establish whether this changes the premature-stop rate.

[spencer2211]

@Rocklis Here are the three record files: https://gist.github.com/spencer2211/1fe9772f4fd19a069a7b82a91bbd2ee4

• results-batch.jsonl — 96 rows (Family A, 6 tasks × 8 reps × 2 arms)
• results-hard.jsonl — 64 rows (Family B, 4 tasks × 8 reps × 2 arms)
• results-posctl.jsonl — 8 rows (positive control, 4 per arm)

Redaction: the only edit is workdir, rewritten to <SANDBOX>/<run_id>-<hex>. Every other field (bucket, other_reason, final_reason, criterion_met, turns, tool_calls, tools_final_turn, first_step_input_tokens, wall_s, exit_code, bad_json_lines, n_errors, error_detail, final_text) is verbatim as the harness wrote it.

Per-arm outcomes, straight from the files:

file	arm	normal_final	premature_stop	other	other_reason
batch	control	48	0	0	all null
batch	treatment	48	0	0	all null
hard	control	32	0	0	all null
hard	treatment	32	0	0	all null
posctl	control	0	4	0	all null
posctl	treatment	0	4	0	all null

error_detail is null and n_errors is 0 in all 168 records; exit_code 0 and bad_json_lines 0 throughout. The posctl rows are the ones where final_reason == "stop", tools_final_turn == 0 and criterion_met == false — their final_text is the "I will create beta.txt next." narration, which is the classifier firing on the intended shape.

Agreed on the reading: zero-vs-zero bounds the rate (≤ 3.7 % per arm at n = 80) rather than showing a change in it.