Amplifier Streaming Loop Orchestrator Module

Token-level streaming orchestration for real-time response delivery.

Prerequisites

• Python 3.11+
• UV - Fast Python package manager

Installing UV

# macOS/Linux/WSL
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Purpose

Provides streaming orchestration that delivers LLM responses token-by-token for improved perceived performance and user experience.

Contract

Module Type: Orchestrator Mount Point: orchestrators Entry Point: amplifier_module_loop_streaming:mount

Behavior

• Token-level streaming from provider
• Real-time response delivery
• Parallel tool execution: Multiple tool calls execute concurrently
• Deterministic context updates: Results added in original order
• Progressive rendering
• Interruptible generation

Durable completed-tool checkpoints

A host may register a zero-argument session.durable_checkpoint capability on the coordinator. The Loop invokes it once after a normal tool batch has settled and all results have been appended in their original order, before a subsequent provider request (including budget finalization). The host owns storage and must persist the complete canonical context before returning; count-only debouncing is not a durable-write guarantee.

The callable may be synchronous or awaitable. Literal False, an ordinary exception, or a non-callable registration fails the turn explicitly without another provider dispatch. Cancellation propagates with appended results intact; this does not add a cancellation-recovery policy. A missing capability preserves the behavior of existing hosts. The Loop itself does not write transcripts. This checkpoint is not the earlier tool:post event, which precedes result append.

Provider budget preflight

When a provider exposes the optional request_budget capability, the loop awaits an awaitable result (or accepts a synchronous result) before checking the fully assembled request for dispatch. An oversized request gets up to two smaller, retention-aware context views; required reminders and request-only injections are replayed without running hooks or draining pending state again. If that view still cannot fit, the loop raises locally and makes no SDK request. Providers without the capability retain the existing request and dispatch behavior.

If a provider exposes request_budget but returns literal None on the initial preflight, the loop treats that request as unavailable and uses the same normal dispatch path. Once a concrete budget has required a rebuild or output-reserve probe, None (or a removed capability) is a local capability-loss error: it never counts as a fit or permits an unchecked SDK request.

When context.request_retention advertises its optional hard_fit keyword, each provider-forced rebuild forwards hard_fit=True, allowing the context to target the provider's requested budget directly. The second rebuild runs only when the provider requests a strictly smaller budget. Older retention capabilities, uninspectable dynamic callables, and the generic context fallback keep their existing provider/retain_contents/token_budget assembly; the final preflight and provider's final payload guard remain the safety boundary. The existing orchestrator:provider_budget event exposes each preflight's attempt, result, estimate, allowance, and requested context budget to mounted observability consumers.

The same event also reports the absence of a native count, so a session that silently fell back to estimate-based behavior is visible rather than invisible. An unavailable report carries result: "unavailable", a mode of initial_fallback, post_concrete_failure, or measured, and a reason of capability_missing, no_decision, or measurement_absent; attempt is included only where the preflight naturally knows it. It carries no count, no fit, no provider exception, and no request or context data, and providers may separately report their own provider-specific reasons. These reports are diagnostic only: the compatibility fallback and the fail-closed capability-loss error both keep their existing behavior, a malformed advertised budget stays an error rather than becoming a fallback, and a provider that never advertised request_budget keeps its historical silent path.

Provider-count measured compaction

When both sides advertise the optional measured contracts — context.measured_request_view and request_budget:provider_count — the loop lets Context compact against the provider's native input count. Context returns the exact ChatRequest it counted, and the loop sends that same object after committing the selected retention transaction. This path is additive: contexts and providers without both capabilities keep the legacy request-budget preflight and estimate-based retention behavior.

If the measured Context getter explicitly accepts fit_output, the loop also offers its lossless output-reserve ladder after Context exhausts its eight legal reduction rungs and still measures a hard oversize. Each of at most six extra counts uses a deep clone of the frozen request, preserving tools, overlays, options, and tool choice. Only the output cap changes, plus a view-only warning below 10,000 output tokens that is included before recounting. The exact accepted counted request is dispatched; hooks, tools, and the system-prompt factory are not rerun.

A missing/unusable count during this fitting fails closed. A provider that does not honor the requested output cap also fails locally. An independent input ceiling can leave every rung oversized; protected content is never discarded to force a fit. Older measured getters that lack the keyword keep their existing behavior. Output fitting can help only when the provider's reported input allowance grows as its output reserve shrinks. For independent input ceilings, the bounded probes may add count calls without finding a fit; this change does not solve an input-only overflow.

For non-streaming foreground responses, the optional context.foreground_usage capability records only normalized successful response usage. A counted streaming request records its selected provider count. If a pre-chunk overflow requires a stream retry, that reading is marked stale rather than attributed to the replacement request. Streams without a provider count do not invent usage; an already-owned reading is marked stale.

Failed goal turns

A failed conversational turn ends the active /goal, flushes any pending error completion as final, and emits terminal goal progress without calling the evaluator, stall judge, or summary model. Task cancellation propagates after terminal goal progress; unlike a cooperative stop, it creates no completion. This applies to initial, continuation, and escalation turns. Cleanup diagnostics are best effort (including cancellation during those diagnostics); the original turn exception still propagates. No successful response, on-disk persistence, or absence of earlier provider calls is claimed by this cleanup.

Provider-reported overflow recovery

A provider may optionally expose synchronous recover_context_overflow to turn its own ContextLengthError into one smaller context target. The loop uses it only before a normal or finalization request has yielded an SDK chunk or returned a response, only when retention explicitly supports hard_fit, and only once for that outbound generation. The recovered view replays the already-resolved retention and request overlays without rerunning hooks or tools; a finalization retry keeps tool_choice="none".

Recovery feedback must be a strict budget dictionary: non-boolean integer fields, an observed input above its allowance, and a positive target smaller than the failed context estimate. A retry is preflighted with the same complete options and the same or lower wire output cap. A fitting preflight retries; None is an explicitly unproven retry authorized by the server rejection; an over-budget, malformed, cancelled, or second-overflow path propagates without another send. The generic loop neither parses provider error messages nor knows provider-private feedback formats. orchestrator:provider_overflow_recovery records only the scalar recovery result.

When a provider reports its effective output cap, an oversized compacted view is also preflighted with progressively smaller response reserves: 50%, 40%, 30%, 20%, and 10% of the original cap, then 1,000 tokens. These are local preflights: they do not resend a provider request or omit input. At caps below 10,000 tokens, a view-only system reminder asks the model to tell the user that the session is degraded and to recommend a new session for substantial work.

Configuration

[[orchestrators]]
module = "loop-streaming"
name = "streaming"
config = {
    max_iterations = -1,             # Maximum LLM calls for a single execute() turn
                                      # (-1 = unlimited, default). This is also the
                                      # mechanism a delegated child session's call
                                      # budget is enforced through -- see "Delegated-
                                      # session call budget (Layer 1)" below.
    budget_warn_ratio = 0.8,         # Fraction of max_iterations at which a one-shot
                                      # "start converging" system-reminder is injected
                                      # (see below). Inert when max_iterations is -1.
    goal_stall_threshold = 3,        # /goal: candidate threshold before a bounded
                                      # evidence judge allows one recovery turn
    goal_model_role = "fast",        # /goal: routing-matrix model role requested for
                                      # the evaluator/stall-judge/summary calls, via
                                      # the model_role_resolver coordinator capability
    goal_provider_preferences = [    # /goal: ordered {provider, model, config?}
        {provider = "anthropic", model = "claude-haiku-*"},        # fallback list, consulted ONLY when
        {provider = "openai", model = "gpt-?.?-luna*"},            # goal_model_role routing above didn't
        {provider = "openai", model = "gpt-?.?-mini*"},            # yield a usable, mounted provider (no
        {provider = "gemini", model = "gemini-*-flash-preview"},   # routing bundle installed, resolver
        {provider = "github-copilot", model = "claude-haiku-4.5"}, # returned no candidates, or resolved
        {provider = "github-copilot", model = "gpt-5.4-mini"},     # provider not mounted). Without this,
        {provider = "ollama", model = "*"},                        # that case falls through to the
    ],                                # session's expensive default model for every
                                      # evaluator call (one per turn) -- a cost
                                      # regression. Models are GLOB patterns, not
                                      # pinned versions, so a new release (e.g. the
                                      # next Haiku point release) is picked up
                                      # automatically the moment a provider lists it,
                                      # with no config change here. Shown above is
                                      # the built-in default (the routing matrix's
                                      # own "fast"-role membership) -- override to
                                      # change it.
    stream_delay = 0.0,              # Per-token artificial delay (seconds), for
                                      # human-facing typing animation (0.0 = off)
    extended_thinking = false,       # Enable extended thinking on the main
                                      # conversational turns (not the /goal internal
                                      # calls, which always disable it)
    min_delay_between_calls_ms = 0,  # Minimum delay between provider calls (rate
                                      # limiting; 0 = disabled)
}

Ephemeral injection mode (prompt-cache prefix fix)

config = {
    ephemeral_injection_mode = "persist",  # "tail" | "persist" (default "persist")
}

Per-iteration ephemeral tail messages -- hooks-status-context, hooks-todo-reminder, the compaction notice, and any other inject_context hook result -- are, by default ("persist"), written into canonical context via context.add_message(...), and only when the text differs from the last text this orchestrator persisted. When unchanged, no duplicate is added; between compactions and other request rewrites, this preserves the opportunity for an append-only request prefix. This matters because OpenAI's (and most providers') implicit/explicit prompt cache reuses only the longest true prefix of a prior request: the original "tail" behavior re-generates and re-appends these messages at the tail of every request, positionally displacing the assistant/tool turn that follows and truncating the reusable prefix -- pinning cache-hit share near the static system-prompt boundary and re-billing the entire growing transcript as a fresh cache write on every call.

Evidence for the default:

The historical observations below are not a guarantee of current long-history cache reuse. Request retention protects instruction delivery, not cache performance; measure reuse from the provider's raw total/read/write counters.

• OpenAI: a pre-registered 9-arm live probe found only the persist design (change-gated, canonical-context write) heals prefix reuse (98.9%); byte-stable tails and folding into the tool result do not heal it (both are still positional, not content, mismatches). In-vivo across 4 DTU eval waves (30+ runs), cache-read share recovered from ~9-11% to 89-97% on every persist run, cache-write dropped ~10x, and task quality was unchanged (all runs correct/passing). A real 6-turn session with "tail" (the old default) showed cache_read pinned flat at 63,060 tokens across every call while cache_write climbed monotonically -- 77.9M cache-write vs. 24.0M cache-read (3.24:1, inverted) -- an estimated $250-380 of that session's $452 total was avoidable re-write spend.
• Anthropic (the flip gate): n=3 DTU S1 runs with persist mode on, claude-opus-4 @ xhigh: 3/3 correct; cache-read share 90.9-92.5% (mean 91.9%) vs. the "tail" baseline's 83.1-87.8% (mean 86.2%) -- +5pts, favorable; cost $1.29 vs. $1.90 mean; wall time 257s vs. 331s mean; wire contract clean (append-only message list, persisted injections re-emitted only on change, valid cache_control breakpoints, zero provider errors, no thinking-block interaction issues).

The tradeoff this mode accepts (spec §5.2): once an injection is persisted, it is no longer "removed next turn" -- it becomes real, bounded history from that point on (still marked metadata.ephemeral=True, now meaning "machine-generated per-turn scaffolding", not "guaranteed absent next turn"), and it is re-emitted (persisted again) only when its content actually changes. Operators who need the original single-ephemeral-tail-message contract -- e.g. a custom hook whose injection text must never accumulate in history -- should set ephemeral_injection_mode = "tail" explicitly; that path remains fully supported and is byte-identical to the module's original, pre-this-feature behavior. An unknown value falls back to "persist" (the current default) with a logged warning.

Retention capability: when the active context exposes the callable context.request_retention capability, persist mode requires the current complete reminder envelope in every request view, including unchanged-body suppression, pending tool feedback, and bounded finalization. A fresh orchestrator also reuses an exact matching admitted persisted envelope after resume instead of adding a duplicate. Required content that cannot fit fails visibly before a provider request.

If the capability is unavailable, persist mode keeps legacy assembly and logs one warning per execute() that complete reminder retention is unavailable. Explicit tail mode is unchanged and does not receive this retention guarantee. None of these changes gives user-carried reminders native system/developer authority.

System-reminder envelope and placement (reminder-redesign-spec.md, W1)

config = {
    reminder_placement = "pre_user",  # "pre_user" | "tail" (default "pre_user")
}

Background: a captured production session showed a model obeying a bare, trailing <system-reminder> injection instead of the user's real request -- the reminder landed AFTER the user's message on the wire, and the model treated the last thing it saw as "the task" rather than supporting context. Two independent fixes address this:

1. The envelope. Every merged hook-injection blob this orchestrator writes is wrapped in <system-reminders>...</system-reminders> with an explicit instruction header telling the model these blocks are NOT from the user and NOT a request, and must never be treated as the task. This is not behind a flag -- it is the fix, and a flag would mean shipping a knob whose "off" position is the known-bad behavior. The envelope tag is deliberately <system-reminders> (not e.g. <injected-context>) so it shares the "<system-reminder" prefix that amplifier-foundation's is_real_user_message matcher (and amplifier-module-provider-openai's FM3 repair) already use -- one prefix match covers both the per-source blocks and this outer envelope.

2. Placement (reminder_placement). By default ("pre_user"), the turn's reminder block is written before the user's prompt -- in canonical history for ephemeral_injection_mode = "persist" (the block precedes the user message as real, append-only history), or spliced into the request VIEW for ephemeral_injection_mode = "tail" (nothing persisted; the splice happens once, at iteration 1, and is never repeated). This is achieved by hoisting iteration 1's provider:request emit to TURN START, before context.add_message adds the user's prompt -- the event payload carries "phase": "turn_start" so a hook that cares can discriminate; hooks that ignore the key behave exactly as before. reminder_placement = "tail" is the rollback lever: it skips the turn-start assembly entirely and restores the pre-this-feature ordering (block after the user message) -- the envelope, role pin, and metadata tag are still applied in "tail" mode; only the ORDERING reverts. An unknown value falls back to "pre_user" with a logged warning.

Role pinning. Every reminder message this orchestrator writes uses the literal role = "user", regardless of what a contributing hook requested via context_injection_role. HookResult.context_injection_role defaults to "system", and a hook that never sets it explicitly contributes system-role content; if the one hook in a chain that does set "user" is ever unmounted, deprioritized, or out-registered, the kernel's merge_inject_context_results ("first result wins" for role) would let a system-role blob through. amplifier-module-provider-anthropic hoists every role == "system" message into the single cached system block, and amplifier-module-provider-openai folds system content into instructions -- a per-turn-changing blob in that position rewrites the system prefix on every single turn. Pinning to "user" defuses this.

Known, accepted contract narrowing: a hook that requests context_injection_role = "assistant" (simulating agent self-talk) is now overridden to "user" like everything else. No hook shipped in this ecosystem does this today; if one needs to in the future, it needs a different mechanism than provider:request inject_context (this orchestrator logs at debug when a non-"user"/"system" role is overridden, so the narrowing is observable rather than invisible).

"Before", not "immediately before". In persist mode, the change-gate suppresses re-persisting an unchanged reminder block. On a turn where nothing changed, canonical history looks like [block N] [user N] [assistant] [tool] [assistant] [user N+1] -- the block is several messages back, not adjacent to user N+1. This is correct and intended, and is the same cache-prefix property ephemeral_injection_mode = "persist" exists to guarantee (see above). Do not "fix" this by disabling the change-gate -- that would defeat the whole cache-prefix benefit this mode provides.

Change-gate comparison basis. The change-gate always compares the RAW (pre-envelope) merged body against the last persisted body, never the enveloped string. This matters because the turn-start block uses the pre-user header variant and a later mid-loop block (same iteration's change-gate lineage) uses the tail variant -- two different headers wrapping potentially-identical content. Comparing enveloped strings would falsely detect a "change" the first time a turn transitions from its turn-start block to a mid-loop one, forcing a spurious extra persisted message on every multi-iteration turn.

Mid-loop (iterations >= 2) placement is unchanged: new content is still written at the tail (tail-variant envelope), the change-gate still suppresses unchanged content, and the pending-injection drain (from tool:post / a stashed prompt:submit result with append_to_last_tool_result) still joins ALL pending injections for one drain into a SINGLE enveloped message (or a single concatenation) rather than one message per injection.

Intended successor (not implemented here): the clean end state is a dedicated turn:reminders event that reminder-contributing hooks register on explicitly, replacing the current re-use of provider:request with a phase discriminator. That would require editing every reminder hook in the ecosystem; hoisting the existing provider:request emit (as done here) delivers the identical wire result today with zero hook edits. A future module version may introduce turn:reminders as the registration point of record.

Delegated-session call budget (Layer 1)

max_iterations doubles as the enforcement mechanism for a per-session-leg LLM-call budget (see microsoft/amplifier-foundation's tool-delegate module, which injects a value here via orchestrator_config when it spawns a child session -- this module has no concept of "delegation" itself; it only counts main-loop LLM calls against whatever max_iterations it was given, root session or child).

Exhaustion is a normal turn ending, not an error. A response that ends naturally (no tool call and no pending steer) returns immediately, including on iteration max_iterations; it makes no duplicate provider call and is not marked budget-exhausted. Only when the hard limit prevents a required continuation does the loop make one additional provider.complete() call with an injected <system-reminder source="orchestrator-loop-limit"> asking the agent to wrap up and summarize. Thus a budget of N permits at most N + 1 main-loop provider calls, while a natural completion uses exactly the calls it needed.

That final request retains the ordinary generic tool declarations (including provider-native declarations needed to validate preceding tool history), but sets the portable tool_choice="none". A compliant final response retains safe text/thinking blocks and provider metadata in the transcript while rendering normalized text; any unexpected tool call is neither dispatched nor persisted structurally, so finalization cannot extend the iteration budget or leave unpaired tool state.

ORCHESTRATOR_COMPLETE's payload always carries a metadata bag:

{
    "llm_calls": 301,             # actual main-loop provider calls, including any one finalization call (not goal-loop internal calls -- see below)
    "llm_call_budget": 300,       # the max_iterations this turn ran under, or None if unlimited
    "budget_exhausted": True,     # whether the budget prevented a needed continuation
    "resumable": True,            # whether this exit path guarantees the transcript was persisted
}

status gains a new value, "budget_exhausted", with precedence error > cancelled > budget_exhausted > success/incomplete -- budget exhaustion sits above success because the wrap-up call fills the response with the agent's own summary text, which would otherwise look identical to an ordinary completed turn.

At budget_warn_ratio (default 80%) of max_iterations, the loop emits orchestrator:budget_warning once per turn and injects a <system-reminder> telling the agent how many calls remain and to start converging. This is a single flat threshold, not an escalation ladder -- unlike hooks-progress-monitor (which escalates because it is guessing the agent is stuck), the budget here is a known fact the agent can act on directly. Both this message and the max-iteration wrap-up reminder above are wrapped in the <system-reminders> envelope (see above) and carry metadata.ephemeral = True -- without it, OpenAI's reasoning-replay cutoff (max(idx for non-ephemeral user)) would count either message as a REAL user turn and collapse the reasoning-replay window for the rest of the turn. The budget-warning message also carries metadata.persisted = True (it is written via context.add_message, genuine history); the max-iteration reminder does not (it is view-only, appended to the outgoing request but never persisted).

The /goal auto-continue loop's own internal calls (evaluator, stall judge, run summary -- emitted with iteration: 0) are not counted against max_iterations; they are separately bounded by goal_stall_threshold and are ~3 calls per goal turn. This keeps the budget coupled to real conversational turns, not goal-loop internals.

Both max_iterations and budget_warn_ratio default to today's behavior (unlimited, and an inert ratio) -- this feature is fully opt-in and ships with zero effect until a caller sets a budget.

Usage

# In amplifier configuration
[session]
orchestrator = "loop-streaming"

Perfect for:

• Interactive CLI applications
• Web UIs with progressive rendering
• Long-form content generation

Dependencies

• amplifier-core>=1.0.0

Contributing

Note

This project is not currently accepting external contributions, but we're actively working toward opening this up. We value community input and look forward to collaborating in the future. For now, feel free to fork and experiment!

Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit Contributor License Agreements.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.