diff --git a/docs/reference/shared-core/llm-provider-seam.md b/docs/reference/shared-core/llm-provider-seam.md index eddd8fa7..d8cdca91 100644 --- a/docs/reference/shared-core/llm-provider-seam.md +++ b/docs/reference/shared-core/llm-provider-seam.md @@ -561,13 +561,36 @@ the seam types are unchanged. The full metering design is in ## Fallback lives outside the adapter Adapters stay **dumb**: they normalize one provider and nothing more. Fallback -chains are **policy**, implemented in a small `withFallback(providers: -LlmProvider[])` runner (in `packages/core`/`packages/llm`): - -- Try `providers[0]`; on a **classified-retryable** `LlmError` (rate limit, - 5xx, overload) move to the next provider. -- Surface **per-attempt usage** so cost accounting stays accurate even across a - failover. +chains are **policy**, implemented by the `FallbackChain` runner (1.K) in +`@relavium/llm` — a class constructed from an ordered plan of attempts +(`{ provider, model, maxAttempts, backoff }`), plus a thin +`withFallback(plan, req, options)` façade for the common single-shot +non-streaming case. It exposes `generate(req)` and `stream(req)`, and the engine +(1.O) builds the plan from the agent's primary `model`/`provider` (+ `retry`) +followed by each authored `fallback_chain` entry: + +- Try each entry in order; on a **classified-retryable** `LlmError` (rate limit, + 5xx/overload, timeout, transport) exhaust the entry's `maxAttempts` with + backoff, then advance to the next entry. On a **fatal** `LlmError` stop + immediately. The advance/stop decision is a pure function of the classified + `kind`/`retryable` (1.I) — never content/string-sentinel inspection. +- **No blind auth retry:** an `auth` failure is never re-attempted on the same + entry; an optional out-of-band credential refresh (`onAuthError`) may grant + exactly one more attempt, otherwise it is fatal. +- **Rate-limit cooldown:** a rate-limited entry is parked in a per-provider + cooldown so an immediately-following call on the same chain skips it. +- **No failover after the first streamed content chunk:** once `stream` has + forwarded content, a mid-stream error surfaces to the node-retry layer (1.S) + rather than re-issuing on the next provider. +- **Strip-on-failover (ADR-0030):** crossing to a *different* provider drops + every `reasoning` part (and its ephemeral `signature`) from the request before + re-issuing — a signature is never replayed across a provider boundary. +- Surface **per-attempt usage** to the injected `CostTracker` (against that + attempt's model) so cost stays accurate across a failover, and report each + attempt (succeeded / failed / skipped) via an `onAttempt` observer so the + engine can emit a `cost:updated` per attempt and a warn log — **visible** + failover, never a silent provider switch. The runner imports no event bus; it + is platform-free (the host injects the `sleep` timer). This keeps per-agent fallback (e.g. Anthropic → OpenAI → DeepSeek) a config concern declared in [`agent-yaml-spec.md`](../contracts/agent-yaml-spec.md) diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 340b17e0..a6231d7e 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -2,7 +2,7 @@ > Status: Living -> Last updated: 2026-06-10 +> Last updated: 2026-06-11 - **Related**: [README.md](README.md), [phases/phase-0-foundations.md](phases/phase-0-foundations.md), [phases/phase-1-engine-and-llm.md](phases/phase-1-engine-and-llm.md), [../project-structure.md](../project-structure.md), [../tech-stack.md](../tech-stack.md) @@ -134,11 +134,14 @@ work runs two parallel lanes: > input/engine/output + surfaces) are additive and do NOT gate M2** — the seam lane proceeds straight > to **1.K**. -> **MCP client scheduled (2026-06-10).** The long-standing contract-ahead-of-implementation gap is -> closed at the decision layer: [ADR-0034](../decisions/0034-mcp-client-sdk-dependency.md) pins the -> official TypeScript MCP SDK as the client implementation and binds the inbound client to -> **workstream 2.R** at the start of [build phase 2](phases/phase-2-cli.md) (off the M3 critical -> path). No Phase-1 work changes. +> **Review-pass follow-ups landed (PR #12, merged 2026-06-11).** The 2026-06-10 engine/tooling +> review pass landed as docs/decisions only — no Phase-1 workstream changed: **MCP client scheduling** +> ([ADR-0034](../decisions/0034-mcp-client-sdk-dependency.md) pins the official TypeScript MCP SDK and +> binds the inbound client to **workstream 2.R** at the start of [build phase 2](phases/phase-2-cli.md), +> off the M3 critical path); the **`turn_limit` `ErrorCode`** (a hard session turn cap, distinct from +> the `[chat].max_messages` trim threshold); the **reserved `on_error` edge kind** +> (workflow-yaml-spec.md, not authorable in v1.0); and a CI **engine dependency-allowlist guard** + the +> pnpm install-script allowlist. No Phase-1 work changes; 1.K/1.L remain the next workstreams. Carry-over hardening is tracked in [deferred-tasks.md](deferred-tasks.md) — pick items up as Phase 1 first touches each file. diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 3d6cefd7..05e99951 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -2,7 +2,7 @@ > Status: Living -> Last updated: 2026-06-10 +> Last updated: 2026-06-11 - **Related**: [current.md](current.md), [README.md](README.md), [phases/phase-0-foundations.md](phases/phase-0-foundations.md) @@ -34,14 +34,16 @@ Severity is the review's verified rating. Check an item off in the PR that resol > multimodal forward-obligations** below (SSRF primitive, async-job ADR, media cost estimate, `partialRef` > semantics, `workspace` authz scope, retention/GC table, `vision`-alias retirement) so nothing is lost. -> **2026-06-10 engine/tooling review pass:** a review of the engine, tool, and CI surfaces against -> the current contracts produced a small set of additions, recorded in their sections below: the -> **tool-output size gate + spill-to-disk** (1.T), **conformance tool-loop / cache-hit scenarios** -> (1.F follow-up), a **token-estimate accuracy** watch item (1.AC), the **Leakwatch CI gate** -> (deferred pending a distribution path), and a **dependency-bump cooling window** (pending a pnpm -> major). The same pass settled three decisions outside this file: the MCP client dependency and -> scheduling (ADR-0034 / workstream 2.R), the reserved `on_error` edge kind -> (workflow-yaml-spec.md), and the `turn_limit` `ErrorCode` (constants.ts + sse-event-schema.md). +> **2026-06-10 engine/tooling review pass (landed in PR #12, merged 2026-06-11):** a review of the +> engine, tool, and CI surfaces against the current contracts produced a small set of additions, +> recorded in their sections below: the **tool-output size gate + spill-to-disk** (1.T), +> **conformance tool-loop / cache-hit scenarios** (1.F follow-up), a **token-estimate accuracy** watch +> item (1.AC), the **Leakwatch CI gate** (deferred pending a distribution path), and a +> **dependency-bump cooling window** (pending a pnpm major). The same pass settled three decisions +> outside this file: the MCP client dependency and scheduling (ADR-0034 / workstream 2.R), the +> reserved `on_error` edge kind (workflow-yaml-spec.md), and the `turn_limit` `ErrorCode` +> (constants.ts + sse-event-schema.md). It also landed a CI **engine dependency-allowlist guard** +> (`tools/engine-deps/check.mjs`) and the pnpm **install-script allowlist**. ## Decisions needed (maintainer call) @@ -233,10 +235,14 @@ Severity is the review's verified rating. Check an item off in the PR that resol doesn't define which keys are legal per `InputType`. Specify that matrix, then add a `WorkflowInputSchema.superRefine((type, validation) => …)`. *(minor · workflow.ts, workflow-yaml-spec.md)* -- [ ] **Verify the non-Anthropic prices in `pricing.ts` (at 1.G/1.H)** — the OpenAI / Gemini / - DeepSeek rows are best-known **placeholders** (Anthropic is confirmed via claude-api). Verify - each against the provider's pricing page when its adapter lands, and replace Gemini's flat - ≤128K-tier figures if context-tiered pricing matters. *(low → 1.G/1.H · packages/llm/src/pricing.ts)* +- [x] **Verify the non-Anthropic prices in `pricing.ts` (at 1.G/1.H)** — the OpenAI / Gemini / + DeepSeek rows were best-known **placeholders** (Anthropic confirmed via claude-api). **Done + 2026-06-11:** verified against each provider's live pricing page, which revealed five of the six + non-Anthropic models were deprecated/shut down — retired and replaced with current models + (gpt-4o→gpt-5.5, gpt-4o-mini→gpt-5.4-mini, gemini-2.0-flash→gemini-2.5-flash, + gemini-1.5-pro→gemini-2.5-pro), DeepSeek prices corrected (deepseek-chat/-reasoner now distinct, + ctx 1M / 384K out, deprecating 2026-07-24), and **Claude Fable 5** added; Opus 4.8 / Sonnet 4.6 / + Haiku 4.5 re-confirmed unchanged. *(packages/llm/src/pricing.ts)* - [x] **`model_catalog` cache-write column (at the seeder)** — `ModelPricing` carries `cacheWritePerMtokMicrocents` (Anthropic charges one), but `model_catalog` ([database-schema.md](../reference/desktop/database-schema.md)) has only diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index f53df9ec..a768d27d 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -51,7 +51,7 @@ function fakeTransport( } const REQ: LlmRequest = { - model: 'gemini-2.0-flash', + model: 'gemini-2.5-flash', messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], }; @@ -73,7 +73,7 @@ describe('Gemini adapter', () => { const transport = fakeTransport({ candidates: [] }); const adapter = createGeminiAdapter({ transport }); const req: LlmRequest = { - model: 'gemini-2.0-flash', + model: 'gemini-2.5-flash', messages: [ { role: 'user', @@ -168,7 +168,7 @@ describe('geminiErrorToLlmError — classification', () => { describe('Gemini adapter — request building (buildGeminiRequest)', () => { it('routes system → systemInstruction, tools → functionDeclarations, and tool choice modes', () => { const request = buildGeminiRequest({ - model: 'gemini-2.0-flash', + model: 'gemini-2.5-flash', system: 'be terse', temperature: 0.4, maxTokens: 64, @@ -217,7 +217,7 @@ describe('Gemini adapter — request building (buildGeminiRequest)', () => { it('round-trips tool_call → functionCall and tool_result → functionResponse by name', () => { const request = buildGeminiRequest({ - model: 'gemini-2.0-flash', + model: 'gemini-2.5-flash', messages: [ { role: 'assistant', @@ -251,7 +251,7 @@ describe('Gemini adapter — request building (buildGeminiRequest)', () => { it('wraps a non-object tool result and lets providerOptions only ADD (mapped fields win)', () => { const request = buildGeminiRequest({ - model: 'gemini-2.0-flash', + model: 'gemini-2.5-flash', providerOptions: { cachedContent: 'abc', temperature: 99 }, temperature: 0.1, messages: [ @@ -398,7 +398,7 @@ describe('Gemini adapter — remaining branches', () => { it('drops a message that lowers to zero parts', () => { const request = buildGeminiRequest({ - model: 'gemini-2.0-flash', + model: 'gemini-2.5-flash', messages: [ { role: 'user', content: [] }, { role: 'user', content: [{ type: 'text', text: 'hi' }] }, diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index ff2c1283..03c0471d 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -39,7 +39,7 @@ const completion = (message: unknown, finishReason = 'stop'): string => id: 'c', object: 'chat.completion', created: 0, - model: 'gpt-4o', + model: 'gpt-5.5', choices: [{ index: 0, message, finish_reason: finishReason, logprobs: null }], usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 }, }); @@ -62,7 +62,7 @@ const streamChunk = (choices: readonly unknown[]): Record => ({ id: 's', object: 'chat.completion.chunk', created: 0, - model: 'gpt-4o', + model: 'gpt-5.5', choices, }); @@ -71,7 +71,7 @@ const dchunk = (delta: unknown, finish: string | null = null): Record { fetch: () => Promise.reject(new Error('must fail fast before any egress')), }); const req = { - model: 'gpt-4o', + model: 'gpt-5.5', messages: [ { role: 'user' as const, @@ -227,7 +227,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () => }); await adapter.generate( { - model: 'gpt-4o', + model: 'gpt-5.5', system: 'be terse', toolChoice: 'required', tools: [{ name: 'get_weather', parameters: { type: 'object' } }], @@ -278,7 +278,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () => }); await adapter.generate( { - model: 'gpt-4o', + model: 'gpt-5.5', temperature: 0.5, stopSequences: ['STOP'], providerOptions: { seed: 42, model: 'attacker-override' }, @@ -289,7 +289,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () => expect(sent['temperature']).toBe(0.5); expect(sent['stop']).toEqual(['STOP']); expect(sent['seed']).toBe(42); // escape-hatch field reached the wire - expect(sent['model']).toBe('gpt-4o'); // mapped field wins over providerOptions + expect(sent['model']).toBe('gpt-5.5'); // mapped field wins over providerOptions }); it('maps tool_choice {name} to a named function choice', async () => { @@ -303,7 +303,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () => }); await adapter.generate( { - model: 'gpt-4o', + model: 'gpt-5.5', toolChoice: { name: 'get_weather' }, messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], }, @@ -337,7 +337,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () => let caught: unknown; try { await adapter.generate( - { model: 'gpt-4o', messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }] }, + { model: 'gpt-5.5', messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }] }, SECRET, ); } catch (err) { @@ -355,7 +355,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () => describe('OpenAI-compatible adapter — stream edge cases', () => { const REQ = { - model: 'gpt-4o', + model: 'gpt-5.5', messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'hi' }] }], }; @@ -387,7 +387,7 @@ describe('OpenAI-compatible adapter — stream edge cases', () => { id: 's', object: 'chat.completion.chunk', created: 0, - model: 'gpt-4o', + model: 'gpt-5.5', // a fragment with no preceding id+name for index 0 — can't start a tool, skipped choices: [ { @@ -401,7 +401,7 @@ describe('OpenAI-compatible adapter — stream edge cases', () => { id: 's', object: 'chat.completion.chunk', created: 0, - model: 'gpt-4o', + model: 'gpt-5.5', choices: [{ index: 0, delta: { content: 'hi' }, finish_reason: 'stop' }], }, ]), @@ -432,7 +432,7 @@ describe('OpenAI-compatible adapter — stream edge cases', () => { describe('OpenAI-compatible adapter — additional fold + generate branches', () => { const REQ = { - model: 'gpt-4o', + model: 'gpt-5.5', messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'hi' }] }], }; @@ -500,7 +500,7 @@ describe('OpenAI-compatible adapter — additional fold + generate branches', () id: 'c', object: 'chat.completion', created: 0, - model: 'gpt-4o', + model: 'gpt-5.5', choices: [], }), { status: 200, headers: { 'content-type': 'application/json' } }, @@ -568,7 +568,7 @@ describe('OpenAI-compatible adapter — reasoning + structured output (ADR-0030) }); await adapter.generate( { - model: 'gpt-4o', + model: 'gpt-5.5', responseFormat: { type: 'json', schema: { type: 'object' }, name: 'out' }, messages: REQ.messages, }, @@ -665,7 +665,7 @@ describe('OpenAI-compatible adapter — robustness (review fixes)', () => { }); await adapter.generate( { - model: 'gpt-4o', + model: 'gpt-5.5', responseFormat: { type: 'json', schema: { type: 'object' }, name: 'my schema!' }, messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], }, @@ -756,7 +756,7 @@ describe('OpenAI-compatible adapter — baseURL SSRF guard', () => { describe('OpenAI-compatible adapter — truncation + refusal normalization', () => { const REQ = { - model: 'gpt-4o', + model: 'gpt-5.5', messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'hi' }] }], }; @@ -812,7 +812,7 @@ describe('OpenAI-compatible adapter — truncation + refusal normalization', () id: 'c', object: 'chat.completion', created: 0, - model: 'gpt-4o', + model: 'gpt-5.5', choices: [ { index: 0, diff --git a/packages/llm/src/conformance/gemini.conformance.test.ts b/packages/llm/src/conformance/gemini.conformance.test.ts index 674fa5a1..0ad17d2b 100644 --- a/packages/llm/src/conformance/gemini.conformance.test.ts +++ b/packages/llm/src/conformance/gemini.conformance.test.ts @@ -58,7 +58,7 @@ describe('gemini — conformance (live, nightly)', () => { it.skipIf(liveKey === '')('generate hits the real API and returns canonical text', async () => { const result = await geminiAdapter.generate( { - model: 'gemini-2.0-flash', + model: 'gemini-2.5-flash', maxTokens: 16, messages: [{ role: 'user', content: [{ type: 'text', text: 'Reply with one word.' }] }], }, diff --git a/packages/llm/src/conformance/openai.conformance.test.ts b/packages/llm/src/conformance/openai.conformance.test.ts index 440f4c12..8ca39a39 100644 --- a/packages/llm/src/conformance/openai.conformance.test.ts +++ b/packages/llm/src/conformance/openai.conformance.test.ts @@ -18,7 +18,7 @@ describe('openai — conformance (live, nightly)', () => { it.skipIf(liveKey === '')('generate hits the real API and returns canonical text', async () => { const result = await openaiAdapter.generate( { - model: 'gpt-4o-mini', + model: 'gpt-5.4-mini', maxTokens: 16, messages: [{ role: 'user', content: [{ type: 'text', text: 'Reply with one word.' }] }], }, diff --git a/packages/llm/src/cost-tracker.test.ts b/packages/llm/src/cost-tracker.test.ts index 6cb0fb95..71ff85a1 100644 --- a/packages/llm/src/cost-tracker.test.ts +++ b/packages/llm/src/cost-tracker.test.ts @@ -45,20 +45,20 @@ describe('cost', () => { it('prices Sonnet 4.6 and DeepSeek from their own rows', () => { expect(cost('claude-sonnet-4-6', { inputTokens: 1000, outputTokens: 500 })).toBe(1_050_000); - // deepseek-chat: 1000 in @ $0.27/MTok = 27_000µ¢; 500 out @ $1.10/MTok = 55_000µ¢ - expect(cost('deepseek-chat', { inputTokens: 1000, outputTokens: 500 })).toBe(82_000); + // deepseek-chat: 1000 in @ $0.14/MTok = 14_000µ¢; 500 out @ $0.28/MTok = 14_000µ¢ + expect(cost('deepseek-chat', { inputTokens: 1000, outputTokens: 500 })).toBe(28_000); }); - it('ignores cache-write tokens for a provider with no cache-write price (gpt-4o)', () => { - // gpt-4o has no cacheWritePerMtokMicrocents → the 1000 cache-write tokens cost 0. - expect(cost('gpt-4o', { inputTokens: 1000, outputTokens: 0, cacheWriteTokens: 1000 })).toBe( - 250_000, - ); // 1000 in @ $2.50/MTok = 250_000 + it('ignores cache-write tokens for a provider with no cache-write price (gpt-5.5)', () => { + // gpt-5.5 has no cacheWritePerMtokMicrocents → the 1000 cache-write tokens cost 0. + expect(cost('gpt-5.5', { inputTokens: 1000, outputTokens: 0, cacheWriteTokens: 1000 })).toBe( + 500_000, + ); // 1000 in @ $5.00/MTok = 500_000 }); it('rounds the per-class micro-cent figure (half-up)', () => { - // gpt-4o-mini cached input $0.075/MTok = 7_500_000µ¢/MTok → 1 token = 7.5 → rounds to 8. - expect(cost('gpt-4o-mini', { inputTokens: 0, outputTokens: 0, cacheReadTokens: 1 })).toBe(8); + // gpt-5.4-mini cached input $0.075/MTok = 7_500_000µ¢/MTok → 1 token = 7.5 → rounds to 8. + expect(cost('gpt-5.4-mini', { inputTokens: 0, outputTokens: 0, cacheReadTokens: 1 })).toBe(8); }); }); @@ -72,10 +72,10 @@ describe('CostTracker', () => { costMicrocents: 1_750_000, cumulativeCostMicrocents: 1_750_000, }); - const b = tracker.record('gpt-4o', { inputTokens: 1000, outputTokens: 0 }); - expect(b.costMicrocents).toBe(250_000); - expect(b.cumulativeCostMicrocents).toBe(2_000_000); // 1_750_000 + 250_000 - expect(tracker.cumulativeCostMicrocents).toBe(2_000_000); + const b = tracker.record('gpt-5.5', { inputTokens: 1000, outputTokens: 0 }); + expect(b.costMicrocents).toBe(500_000); + expect(b.cumulativeCostMicrocents).toBe(2_250_000); // 1_750_000 + 500_000 + expect(tracker.cumulativeCostMicrocents).toBe(2_250_000); }); }); diff --git a/packages/llm/src/fallback-chain.test.ts b/packages/llm/src/fallback-chain.test.ts new file mode 100644 index 00000000..3983fa49 --- /dev/null +++ b/packages/llm/src/fallback-chain.test.ts @@ -0,0 +1,1242 @@ +import { describe, expect, it } from 'vitest'; + +import { CostTracker } from './cost-tracker.js'; +import { + FallbackChain, + stripReasoningParts, + withFallback, + type AttemptRecord, + type FallbackChainOptions, + type FallbackPlanEntry, +} from './fallback-chain.js'; +import { LlmProviderError, makeLlmError } from './llm-error.js'; +import type { + CapabilityFlags, + LlmError, + LlmErrorKind, + LlmProvider, + LlmRequest, + LlmResult, + ProviderId, + StreamChunk, + Usage, +} from './types.js'; + +// --- fakes & helpers ------------------------------------------------------------------------- + +/** A full CapabilityFlags with permissive defaults; only `tools`/`streaming` vary in these tests. */ +function caps(overrides?: { tools?: boolean; streaming?: boolean }): CapabilityFlags { + return { + tools: overrides?.tools ?? true, + streaming: overrides?.streaming ?? true, + parallelToolCalls: false, + vision: false, // === media.input.image (the refine pins them equal) + promptCache: false, + reasoning: false, + media: { + input: { image: false, audio: false, video: false, document: false }, + outputCombinations: [], + }, + }; +} + +interface FakeProvider { + readonly provider: LlmProvider; + /** The requests each generate/stream invocation received, in order. */ + readonly calls: LlmRequest[]; +} + +function makeProvider(opts: { + id: ProviderId; + supports?: { tools?: boolean; streaming?: boolean }; + generate?: (req: LlmRequest, key: string, call: number) => Promise; + stream?: (req: LlmRequest, key: string, call: number) => AsyncIterable; +}): FakeProvider { + const calls: LlmRequest[] = []; + let genCalls = 0; + let streamCalls = 0; + const provider: LlmProvider = { + id: opts.id, + supports: caps(opts.supports), + generate(req, key) { + calls.push(req); + genCalls += 1; + if (opts.generate === undefined) { + throw new Error('generate stub not provided'); + } + return opts.generate(req, key, genCalls); + }, + stream(req, key) { + calls.push(req); + streamCalls += 1; + if (opts.stream === undefined) { + throw new Error('stream stub not provided'); + } + return opts.stream(req, key, streamCalls); + }, + }; + return { provider, calls }; +} + +function entry(fake: FakeProvider, model: string, maxAttempts = 1): FallbackPlanEntry { + return { provider: fake.provider, model, maxAttempts }; +} + +const USAGE: Usage = { inputTokens: 1000, outputTokens: 500 }; + +function result(text: string, usage: Usage = USAGE): LlmResult { + return { content: [{ type: 'text', text }], stopReason: 'stop', usage, raw: undefined }; +} + +function providerError( + provider: ProviderId, + kind: LlmErrorKind, + message = 'boom', +): LlmProviderError { + return new LlmProviderError(makeLlmError({ provider, kind, message })); +} + +/** A generate stub that resolves a result (non-async to avoid an awaitless async function). */ +const resolves = (text: string, usage?: Usage) => (): Promise => + Promise.resolve(result(text, usage)); + +/** A generate stub that rejects with a classified `LlmProviderError` (the seam's generate contract). */ +const rejects = + (provider: ProviderId, kind: LlmErrorKind, message?: string) => (): Promise => + Promise.reject(providerError(provider, kind, message)); + +async function* streamFrom(chunks: StreamChunk[]): AsyncIterable { + await Promise.resolve(); // a real stream awaits I/O; this keeps the fake a true async iterable + for (const chunk of chunks) { + yield chunk; + } +} + +/** An async iterable that rejects on the first pull — models a `stream()` that fails before any chunk. */ +function streamThrowing(error: LlmProviderError): AsyncIterable { + return { + [Symbol.asyncIterator]: () => ({ + next: (): Promise> => Promise.reject(error), + }), + }; +} + +function errChunk(provider: ProviderId, kind: LlmErrorKind): StreamChunk { + return { type: 'error', error: makeLlmError({ provider, kind, message: 'boom' }) }; +} + +const STOP_CHUNK: StreamChunk = { type: 'stop', stopReason: 'stop', usage: USAGE }; + +const userReq: LlmRequest = { + model: 'incoming', + messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], +}; + +/** Options with a recording no-op sleep + a recorder for the emitted attempt trace. */ +function makeOptions(overrides?: Partial): { + options: FallbackChainOptions; + trace: AttemptRecord[]; + sleeps: number[]; +} { + const trace: AttemptRecord[] = []; + const sleeps: number[] = []; + const options: FallbackChainOptions = { + keyFor: () => 'test-key', + sleep: (ms) => { + sleeps.push(ms); + return Promise.resolve(); + }, + onAttempt: (record) => { + trace.push(record); + }, + ...overrides, + }; + return { options, trace, sleeps }; +} + +/** Run a promise that should reject with an `LlmProviderError`, returning the carried `LlmError`. */ +async function rejectedError(promise: Promise): Promise { + try { + await promise; + } catch (err) { + if (err instanceof LlmProviderError) { + return err.llmError; + } + throw err; + } + throw new Error('expected the promise to reject with an LlmProviderError'); +} + +async function collect(stream: AsyncIterable): Promise { + const out: StreamChunk[] = []; + for await (const chunk of stream) { + out.push(chunk); + } + return out; +} + +// --- construction ---------------------------------------------------------------------------- + +describe('FallbackChain construction', () => { + it('rejects an empty plan (a wiring invariant)', () => { + const { options } = makeOptions(); + expect(() => new FallbackChain([], options)).toThrowError(/at least one plan entry/); + }); + + it('rejects a plan entry with a non-positive maxAttempts (loud, not a silent skip)', () => { + const provider = makeProvider({ id: 'anthropic', generate: resolves('x') }); + const { options } = makeOptions(); + expect( + () => new FallbackChain([{ ...entry(provider, 'claude-opus-4-8'), maxAttempts: 0 }], options), + ).toThrowError(/positive integer maxAttempts/); + }); +}); + +// --- generate: failover & classification ----------------------------------------------------- + +describe('FallbackChain.generate', () => { + it('returns the primary result with no failover when the first attempt succeeds', async () => { + const primary = makeProvider({ id: 'anthropic', generate: resolves('ok') }); + const fallback = makeProvider({ id: 'openai', generate: resolves('unused') }); + const { options, trace } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const out = await chain.generate(userReq); + + expect(out.content).toEqual([{ type: 'text', text: 'ok' }]); + expect(primary.calls).toHaveLength(1); + expect(fallback.calls).toHaveLength(0); // never reached + expect(trace).toHaveLength(1); + expect(trace[0]).toMatchObject({ + attemptNumber: 1, + provider: 'anthropic', + outcome: 'succeeded', + }); + }); + + it('fails over to the next provider on a retryable error, then succeeds', async () => { + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'overloaded') }); + const fallback = makeProvider({ id: 'openai', generate: resolves('recovered') }); + const { options, trace } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const out = await chain.generate(userReq); + + expect(out.content).toEqual([{ type: 'text', text: 'recovered' }]); + expect(primary.calls).toHaveLength(1); + expect(fallback.calls).toHaveLength(1); + expect(trace.map((r) => [r.provider, r.outcome])).toEqual([ + ['anthropic', 'failed'], + ['openai', 'succeeded'], + ]); + expect(trace.map((r) => r.attemptNumber)).toEqual([1, 2]); + }); + + it('stops immediately on a fatal error and never reaches the fallback', async () => { + const primary = makeProvider({ + id: 'anthropic', + generate: rejects('anthropic', 'content_filter'), + }); + const fallback = makeProvider({ id: 'openai', generate: resolves('unused') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('content_filter'); + expect(err.retryable).toBe(false); + expect(fallback.calls).toHaveLength(0); // fatal does not fall through + }); + + it('normalizes a non-LlmProviderError throw to a fatal `unknown` and stops', async () => { + const primary = makeProvider({ + id: 'anthropic', + generate: () => Promise.reject(new TypeError('a bare programming error')), + }); + const fallback = makeProvider({ id: 'openai', generate: resolves('unused') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('unknown'); + expect(fallback.calls).toHaveLength(0); + }); + + it('normalizes a non-Error thrown value to a fatal `unknown` with a generic message', async () => { + const provider = makeProvider({ + id: 'anthropic', + generate: () => { + // A provider that throws a non-Error value (not the seam contract, but the runner must + // normalize it rather than leak it) — exercises the `#errorOf` fallback message. + // eslint-disable-next-line @typescript-eslint/only-throw-error -- deliberate non-Error throw + throw 'a bare string failure'; // NOSONAR — the non-Error throw is exactly what this test exercises + }, + }); + const { options } = makeOptions(); + const chain = new FallbackChain([entry(provider, 'claude-opus-4-8')], options); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('unknown'); + expect(err.message).toBe('unknown provider failure'); + }); + + it('exhausts the chain and throws the last error when every entry fails retryably', async () => { + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'overloaded') }); + const fallback = makeProvider({ id: 'openai', generate: rejects('openai', 'timeout') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('timeout'); // the last attempt's classified error + expect(err.provider).toBe('openai'); + }); +}); + +// --- the failover decision is pure-on-discriminant ------------------------------------------- + +describe('FallbackChain.generate — pure-on-discriminant decision', () => { + it('does not fail over on a successful result whose CONTENT looks like an error', async () => { + const primary = makeProvider({ + id: 'anthropic', + generate: resolves('Error: this is a normal answer that mentions Error:'), + }); + const fallback = makeProvider({ id: 'openai', generate: resolves('unused') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const out = await chain.generate(userReq); + + expect(out.content).toEqual([ + { type: 'text', text: 'Error: this is a normal answer that mentions Error:' }, + ]); + expect(fallback.calls).toHaveLength(0); // content is never inspected + }); + + it('does not fail over on a fatal error whose MESSAGE looks transient', async () => { + const primary = makeProvider({ + id: 'anthropic', + generate: rejects('anthropic', 'bad_request', 'Error: temporarily unavailable, retry'), + }); + const fallback = makeProvider({ id: 'openai', generate: resolves('unused') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('bad_request'); // the discriminant, not the message, decides + expect(fallback.calls).toHaveLength(0); + }); + + it('does not fail over on an empty/malformed result body (it is a success)', async () => { + const primary = makeProvider({ + id: 'anthropic', + generate: () => + Promise.resolve({ content: [], stopReason: 'stop', usage: USAGE, raw: undefined }), + }); + const fallback = makeProvider({ id: 'openai', generate: resolves('unused') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const out = await chain.generate(userReq); + + expect(out.content).toEqual([]); // an empty body is a result, not a failure + expect(fallback.calls).toHaveLength(0); // body shape never triggers failover + }); +}); + +// --- cost accounting across a failover ------------------------------------------------------- + +describe('FallbackChain.generate — per-attempt cost across a failover', () => { + it('records the winning attempt against ITS model and accumulates in the tracker', async () => { + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'overloaded') }); + const fallback = makeProvider({ + id: 'openai', + generate: resolves('recovered', { inputTokens: 1000, outputTokens: 500 }), + }); + const tracker = new CostTracker(); + tracker.record('claude-opus-4-8', { inputTokens: 1000, outputTokens: 500 }); // a prior turn → 1_750_000µ¢ + const { options, trace } = makeOptions({ costTracker: tracker }); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.4-mini')], + options, + ); + + await chain.generate(userReq); + + // gpt-5.4-mini: 1000 in @ $0.75/MTok = 75_000µ¢; 500 out @ $4.50/MTok = 225_000µ¢ → 300_000. + expect(trace[0]).toMatchObject({ outcome: 'failed' }); + expect(trace[0]?.cost).toBeUndefined(); // a failed attempt with no usage records no cost + expect(trace[1]).toMatchObject({ outcome: 'succeeded' }); + expect(trace[1]?.cost?.costMicrocents).toBe(300_000); // THIS attempt, priced against gpt-5.4-mini + // the running total threads the prior turn forward (not reset, not the single-attempt figure): + expect(trace[1]?.cost?.cumulativeCostMicrocents).toBe(2_050_000); // 1_750_000 + 300_000 + expect(tracker.cumulativeCostMicrocents).toBe(2_050_000); + }); +}); + +// --- ADR-0030 reasoning strip on cross-provider failover ------------------------------------- + +describe('FallbackChain — ADR-0030 strip-on-failover', () => { + const reqWithReasoning: LlmRequest = { + model: 'incoming', + messages: [ + { role: 'user', content: [{ type: 'text', text: 'solve it' }] }, + { + role: 'assistant', + content: [ + { type: 'reasoning', text: 'thinking...', signature: 'sig-from-anthropic' }, + { type: 'text', text: 'partial' }, + ], + }, + { role: 'user', content: [{ type: 'text', text: 'continue' }] }, + ], + }; + + it('strips reasoning parts (and signature) from the request sent to a DIFFERENT provider', async () => { + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'overloaded') }); + const fallback = makeProvider({ id: 'openai', generate: resolves('ok') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + await chain.generate(reqWithReasoning); + + const sentToFallback = fallback.calls[0]; + const parts = sentToFallback?.messages.flatMap((m) => m.content) ?? []; + expect(parts.some((p) => p.type === 'reasoning')).toBe(false); + // the surrounding non-reasoning content survives + expect(parts.filter((p) => p.type === 'text')).toHaveLength(3); + // the primary (originating provider) received the reasoning unchanged + const sentToPrimary = primary.calls[0]; + const primaryParts = sentToPrimary?.messages.flatMap((m) => m.content) ?? []; + expect(primaryParts.some((p) => p.type === 'reasoning')).toBe(true); + // the caller's request is not mutated + expect(reqWithReasoning.messages[1]?.content.some((p) => p.type === 'reasoning')).toBe(true); + }); + + it('keeps reasoning on a SAME-provider retry (only a provider boundary strips)', async () => { + const provider = makeProvider({ + id: 'anthropic', + generate: (_req, _key, call) => + call === 1 + ? Promise.reject(providerError('anthropic', 'overloaded')) + : Promise.resolve(result('ok')), + }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [{ ...entry(provider, 'claude-opus-4-8'), maxAttempts: 2 }], + options, + ); + + await chain.generate(reqWithReasoning); + + expect(provider.calls).toHaveLength(2); + const secondCallParts = provider.calls[1]?.messages.flatMap((m) => m.content) ?? []; + expect(secondCallParts.some((p) => p.type === 'reasoning')).toBe(true); // not stripped + }); + + it('keeps reasoning for a same-provider entry after an intervening provider is skipped', async () => { + // [anthropic, openai(skipped, lacks tools), anthropic] — the skipped openai entry must NOT count + // as a provider boundary, or the third (same-provider) entry would wrongly lose its reasoning. + const toolReq: LlmRequest = { + ...reqWithReasoning, + tools: [{ name: 'read_file', parameters: { type: 'object' } }], + }; + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'overloaded') }); + const skipped = makeProvider({ + id: 'openai', + supports: { tools: false }, + generate: resolves('x'), + }); + const sameProvider = makeProvider({ id: 'anthropic', generate: resolves('ok') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [ + entry(primary, 'claude-opus-4-8'), + entry(skipped, 'gpt-5.5'), + entry(sameProvider, 'claude-haiku-4-5'), + ], + options, + ); + + await chain.generate(toolReq); + + expect(skipped.calls).toHaveLength(0); // openai skipped (no tools) + const parts = sameProvider.calls[0]?.messages.flatMap((m) => m.content) ?? []; + expect(parts.some((p) => p.type === 'reasoning')).toBe(true); // same provider → reasoning kept + }); +}); + +describe('stripReasoningParts', () => { + it('drops reasoning parts, removes emptied messages, and does not mutate the input', () => { + const req: LlmRequest = { + model: 'm', + messages: [ + { role: 'assistant', content: [{ type: 'reasoning', text: 't', signature: 's' }] }, + { + role: 'assistant', + content: [ + { type: 'reasoning', text: 't2' }, + { type: 'text', text: 'kept' }, + ], + }, + ], + }; + + const out = stripReasoningParts(req); + + expect(out.messages).toHaveLength(1); // the reasoning-only message is dropped + expect(out.messages[0]?.content).toEqual([{ type: 'text', text: 'kept' }]); + expect(req.messages).toHaveLength(2); // input untouched + }); + + it('merges messages left adjacent and same-role after a reasoning-only drop (alternation stays valid)', () => { + const req: LlmRequest = { + model: 'm', + messages: [ + { role: 'user', content: [{ type: 'text', text: 'a' }] }, + { role: 'assistant', content: [{ type: 'reasoning', text: 'think', signature: 's' }] }, // reasoning-only + { role: 'user', content: [{ type: 'text', text: 'b' }] }, + ], + }; + + const out = stripReasoningParts(req); + + // the assistant turn is dropped and the two now-adjacent user turns merge — no `[user, user]` + expect(out.messages).toHaveLength(1); + expect(out.messages[0]).toEqual({ + role: 'user', + content: [ + { type: 'text', text: 'a' }, + { type: 'text', text: 'b' }, + ], + }); + }); +}); + +// --- auth nuance: no blind retry + optional one-shot refresh ---------------------------------- + +describe('FallbackChain.generate — auth handling', () => { + it('never re-attempts an auth failure on the same entry (no blind retry loop)', async () => { + const provider = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'auth') }); + const { options } = makeOptions(); + // A generous budget — auth must still attempt exactly once. + const chain = new FallbackChain( + [{ ...entry(provider, 'claude-opus-4-8'), maxAttempts: 5 }], + options, + ); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('auth'); + expect(provider.calls).toHaveLength(1); // one attempt, not five + }); + + it('grants exactly one extra attempt after a successful out-of-band credential refresh', async () => { + const provider = makeProvider({ + id: 'anthropic', + generate: (_req, _key, call) => + call === 1 + ? Promise.reject(providerError('anthropic', 'auth')) + : Promise.resolve(result('ok-after-refresh')), + }); + let refreshCalls = 0; + const { options } = makeOptions({ + onAuthError: () => { + refreshCalls += 1; + return true; + }, + }); + const chain = new FallbackChain([entry(provider, 'claude-opus-4-8')], options); // maxAttempts 1 + + const out = await chain.generate(userReq); + + expect(out.content).toEqual([{ type: 'text', text: 'ok-after-refresh' }]); + expect(refreshCalls).toBe(1); + expect(provider.calls).toHaveLength(2); // the refresh bought one more attempt despite budget 1 + }); + + it('refreshes at most once even when the granted retry also fails auth (no auth loop)', async () => { + const provider = makeProvider({ + id: 'anthropic', + // auth on the original AND on the granted retry — the second must be fatal, not re-armed. + generate: (_req, _key, call) => + call <= 2 + ? Promise.reject(providerError('anthropic', 'auth')) + : Promise.resolve(result('never')), + }); + let refreshCalls = 0; + const { options } = makeOptions({ + onAuthError: () => { + refreshCalls += 1; + return true; + }, + }); + const chain = new FallbackChain([entry(provider, 'claude-opus-4-8')], options); // maxAttempts 1 + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('auth'); + expect(refreshCalls).toBe(1); // the one-shot guard is not re-armed by the granted retry + expect(provider.calls).toHaveLength(2); // one original + exactly one granted, then fatal (no third) + }); + + it('grants the auth retry ON TOP of the configured budget, not capping it', async () => { + const provider = makeProvider({ + id: 'anthropic', + // attempt 1 auth (refresh → +1), attempt 2 overloaded; the remaining configured budget must survive. + generate: (_req, _key, call) => { + if (call === 1) return Promise.reject(providerError('anthropic', 'auth')); + return Promise.reject(providerError('anthropic', 'overloaded')); + }, + }); + const { options } = makeOptions({ onAuthError: () => true }); + // budget 3 + one auth bonus = 4 same-provider attempts before the chain exhausts. + const chain = new FallbackChain( + [{ ...entry(provider, 'claude-opus-4-8'), maxAttempts: 3 }], + options, + ); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('overloaded'); + expect(provider.calls).toHaveLength(4); // 3 configured + 1 auth bonus — the budget was not truncated + }); + + it('is fatal when the credential refresh declines, and refreshes at most once per provider', async () => { + const provider = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'auth') }); + let refreshCalls = 0; + const { options } = makeOptions({ + onAuthError: () => { + refreshCalls += 1; + return false; + }, + }); + const chain = new FallbackChain( + [{ ...entry(provider, 'claude-opus-4-8'), maxAttempts: 3 }], + options, + ); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('auth'); + expect(refreshCalls).toBe(1); + expect(provider.calls).toHaveLength(1); + }); + + it('treats a throwing credential-refresh hook as a declined refresh (fatal, contract intact)', async () => { + const provider = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'auth') }); + const { options } = makeOptions({ + onAuthError: () => { + throw new Error('the host hook blew up'); + }, + }); + const chain = new FallbackChain([entry(provider, 'claude-opus-4-8')], options); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('auth'); // the ORIGINAL auth error surfaces, not the hook's throw + expect(provider.calls).toHaveLength(1); // a failed refresh grants no extra attempt + }); +}); + +// --- backoff + rate-limit cooldown ----------------------------------------------------------- + +describe('FallbackChain — backoff and cooldown', () => { + it('exhausts the entry budget with exponential backoff between attempts, then advances', async () => { + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'overloaded') }); + const fallback = makeProvider({ id: 'openai', generate: resolves('ok') }); + const { options, sleeps } = makeOptions({ backoffBaseMs: 100, backoffMaxMs: 1000 }); + const chain = new FallbackChain( + [{ ...entry(primary, 'claude-opus-4-8'), maxAttempts: 3 }, entry(fallback, 'gpt-5.5')], + options, + ); + + await chain.generate(userReq); + + expect(primary.calls).toHaveLength(3); // exhausted the budget + expect(sleeps).toEqual([100, 200]); // backoff before attempts 2 and 3, exponential, no inter-entry delay + }); + + it('respects the linear backoff curve and the ceiling', async () => { + const provider = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'timeout') }); + const { options, sleeps } = makeOptions({ backoffBaseMs: 100, backoffMaxMs: 250 }); + const chain = new FallbackChain( + [ + { + provider: provider.provider, + model: 'claude-opus-4-8', + maxAttempts: 4, + backoff: 'linear', + }, + ], + options, + ); + + await rejectedError(chain.generate(userReq)); + + expect(sleeps).toEqual([100, 200, 250]); // linear 100/200/300 capped at 250 + }); + + it('parks a rate-limited provider in cooldown so the next call skips it', async () => { + let clock = 0; + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'rate_limit') }); + const fallback = makeProvider({ id: 'openai', generate: resolves('ok') }); + const { options, trace } = makeOptions({ now: () => clock, cooldownMs: 1000 }); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + await chain.generate(userReq); // call 1: primary rate-limits → cooldown → fallback wins + expect(primary.calls).toHaveLength(1); + + trace.length = 0; + await chain.generate(userReq); // call 2: still within cooldown → primary skipped + expect(primary.calls).toHaveLength(1); // NOT hammered again + expect(fallback.calls).toHaveLength(2); + expect(trace[0]).toMatchObject({ provider: 'anthropic', outcome: 'skipped' }); + expect(trace[0]?.skipReason).toMatch(/cooldown/); + + trace.length = 0; + clock = 2000; // cooldown elapsed + await chain.generate(userReq); // call 3: primary retried again + expect(primary.calls).toHaveLength(2); + }); +}); + +// --- capability skip ------------------------------------------------------------------------- + +describe('FallbackChain — capability skip', () => { + it('skips a provider that cannot satisfy a required capability without consuming an attempt', async () => { + const toolReq: LlmRequest = { + ...userReq, + tools: [{ name: 'read_file', parameters: { type: 'object' } }], + }; + const primary = makeProvider({ + id: 'gemini', + supports: { tools: false }, + generate: resolves('unused'), + }); + const fallback = makeProvider({ id: 'openai', generate: resolves('ok') }); + const { options, trace } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'gemini-2.5-pro'), entry(fallback, 'gpt-5.5')], + options, + ); + + const out = await chain.generate(toolReq); + + expect(out.content).toEqual([{ type: 'text', text: 'ok' }]); + expect(primary.calls).toHaveLength(0); // skipped, never attempted + expect(trace[0]).toMatchObject({ provider: 'gemini', outcome: 'skipped' }); + expect(trace[0]?.skipReason).toMatch(/capability/); + }); + + it('throws a synthesized error when every entry is skipped', async () => { + const toolReq: LlmRequest = { + ...userReq, + tools: [{ name: 'read_file', parameters: { type: 'object' } }], + }; + const a = makeProvider({ id: 'gemini', supports: { tools: false }, generate: resolves('x') }); + const b = makeProvider({ id: 'deepseek', supports: { tools: false }, generate: resolves('x') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(a, 'gemini-2.5-pro'), entry(b, 'deepseek-chat')], + options, + ); + + const err = await rejectedError(chain.generate(toolReq)); + + expect(err.kind).toBe('unknown'); + expect(err.message).toMatch(/exhausted/); + }); +}); + +// --- abort ----------------------------------------------------------------------------------- + +describe('FallbackChain — cancellation', () => { + it('treats an already-aborted signal as fatal and never attempts a provider', async () => { + const primary = makeProvider({ id: 'anthropic', generate: resolves('unused') }); + const { options } = makeOptions(); + const chain = new FallbackChain([entry(primary, 'claude-opus-4-8')], options); + const aborted: LlmRequest = { + ...userReq, + signal: { aborted: true, addEventListener() {}, removeEventListener() {} }, + }; + + const err = await rejectedError(chain.generate(aborted)); + + expect(err.kind).toBe('cancelled'); + expect(primary.calls).toHaveLength(0); + }); + + it('does not fail over when a provider reports cancellation', async () => { + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'cancelled') }); + const fallback = makeProvider({ id: 'openai', generate: resolves('unused') }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const err = await rejectedError(chain.generate(userReq)); + + expect(err.kind).toBe('cancelled'); + expect(fallback.calls).toHaveLength(0); + }); +}); + +// --- streaming ------------------------------------------------------------------------------- + +describe('FallbackChain.stream', () => { + it('forwards a successful stream and records usage from the stop chunk', async () => { + const provider = makeProvider({ + id: 'anthropic', + stream: () => streamFrom([{ type: 'text_delta', text: 'hello' }, STOP_CHUNK]), + }); + const tracker = new CostTracker(); + const { options, trace } = makeOptions({ costTracker: tracker }); + const chain = new FallbackChain([entry(provider, 'claude-haiku-4-5')], options); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toEqual([{ type: 'text_delta', text: 'hello' }, STOP_CHUNK]); + expect(trace[0]).toMatchObject({ outcome: 'succeeded' }); + // claude-haiku-4-5: 1000 in @ $1/MTok = 100_000µ¢; 500 out @ $5/MTok = 250_000µ¢ → 350_000. + expect(trace[0]?.cost?.costMicrocents).toBe(350_000); + }); + + it('records a content-free success with no usage when the stream omits a stop chunk', async () => { + const provider = makeProvider({ + id: 'anthropic', + stream: () => streamFrom([{ type: 'text_delta', text: 'partial' }]), + }); + const { options, trace } = makeOptions(); + const chain = new FallbackChain([entry(provider, 'claude-haiku-4-5')], options); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toEqual([{ type: 'text_delta', text: 'partial' }]); + expect(trace[0]).toMatchObject({ outcome: 'succeeded' }); + expect(trace[0]?.usage).toBeUndefined(); + expect(trace[0]?.cost).toBeUndefined(); + }); + + it('treats a content-free stop-only stream as a success (committed stays false)', async () => { + const provider = makeProvider({ id: 'anthropic', stream: () => streamFrom([STOP_CHUNK]) }); + const { options, trace } = makeOptions({ costTracker: new CostTracker() }); + const chain = new FallbackChain([entry(provider, 'claude-haiku-4-5')], options); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toEqual([STOP_CHUNK]); + expect(trace).toHaveLength(1); + expect(trace[0]).toMatchObject({ outcome: 'succeeded' }); + expect(trace[0]?.cost?.costMicrocents).toBe(350_000); // usage still came from the stop chunk + }); + + it('grants one extra stream attempt after an out-of-band credential refresh', async () => { + const provider = makeProvider({ + id: 'anthropic', + stream: (_req, _key, call) => + call === 1 + ? streamFrom([errChunk('anthropic', 'auth')]) + : streamFrom([{ type: 'text_delta', text: 'ok-after-refresh' }, STOP_CHUNK]), + }); + const { options } = makeOptions({ onAuthError: () => true }); + const chain = new FallbackChain([entry(provider, 'claude-opus-4-8')], options); // maxAttempts 1 + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toEqual([{ type: 'text_delta', text: 'ok-after-refresh' }, STOP_CHUNK]); + expect(provider.calls).toHaveLength(2); // refresh bought one more attempt despite budget 1 + }); + + it('refreshes at most once on the stream path even when the granted retry fails auth', async () => { + const provider = makeProvider({ + id: 'anthropic', + stream: (_req, _key, call) => + call <= 2 + ? streamFrom([errChunk('anthropic', 'auth')]) + : streamFrom([{ type: 'text_delta', text: 'never' }, STOP_CHUNK]), + }); + let refreshCalls = 0; + const { options } = makeOptions({ + onAuthError: () => { + refreshCalls += 1; + return true; + }, + }); + const chain = new FallbackChain([entry(provider, 'claude-opus-4-8')], options); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toHaveLength(1); + if (chunks[0]?.type === 'error') { + expect(chunks[0].error.kind).toBe('auth'); + } + expect(refreshCalls).toBe(1); // one-shot guard holds on the stream path too + expect(provider.calls).toHaveLength(2); // one original + exactly one granted, then fatal + }); + + it('strips reasoning on a cross-provider stream failover (ADR-0030)', async () => { + const reqWithReasoning: LlmRequest = { + model: 'incoming', + messages: [ + { role: 'user', content: [{ type: 'text', text: 'solve it' }] }, + { + role: 'assistant', + content: [ + { type: 'reasoning', text: 'thinking...', signature: 'sig-from-anthropic' }, + { type: 'text', text: 'partial' }, + ], + }, + { role: 'user', content: [{ type: 'text', text: 'continue' }] }, + ], + }; + const primary = makeProvider({ + id: 'anthropic', + stream: () => streamFrom([errChunk('anthropic', 'overloaded')]), + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([{ type: 'text_delta', text: 'recovered' }, STOP_CHUNK]), + }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + await collect(chain.stream(reqWithReasoning)); + + const fallbackParts = fallback.calls[0]?.messages.flatMap((m) => m.content) ?? []; + expect(fallbackParts.some((p) => p.type === 'reasoning')).toBe(false); // stripped on the boundary + const primaryParts = primary.calls[0]?.messages.flatMap((m) => m.content) ?? []; + expect(primaryParts.some((p) => p.type === 'reasoning')).toBe(true); // originator kept it + }); + + it('fails over transparently on a pre-content error chunk', async () => { + const primary = makeProvider({ + id: 'anthropic', + stream: () => streamFrom([errChunk('anthropic', 'overloaded')]), + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([{ type: 'text_delta', text: 'recovered' }, STOP_CHUNK]), + }); + const { options, trace } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + + // the consumer never sees the primary's error chunk — the failover is transparent + expect(chunks).toEqual([{ type: 'text_delta', text: 'recovered' }, STOP_CHUNK]); + expect(trace.map((r) => [r.provider, r.outcome])).toEqual([ + ['anthropic', 'failed'], + ['openai', 'succeeded'], + ]); + }); + + it('does NOT fail over after the first content chunk — it surfaces the error', async () => { + const primary = makeProvider({ + id: 'anthropic', + stream: () => + streamFrom([ + { type: 'text_delta', text: 'partial output' }, + errChunk('anthropic', 'overloaded'), + ]), + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([{ type: 'text_delta', text: 'never' }, STOP_CHUNK]), + }); + const { options, trace } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toEqual([ + { type: 'text_delta', text: 'partial output' }, + errChunk('anthropic', 'overloaded'), + ]); + expect(fallback.calls).toHaveLength(0); // committed → no failover + expect(trace).toHaveLength(1); + expect(trace[0]).toMatchObject({ provider: 'anthropic', outcome: 'failed' }); + }); + + it('commits the stream on a non-text content chunk (tool_call_start), preventing failover', async () => { + const primary = makeProvider({ + id: 'anthropic', + stream: () => + streamFrom([ + { type: 'tool_call_start', id: 'tc1', name: 'read_file' }, + errChunk('anthropic', 'overloaded'), + ]), + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([{ type: 'text_delta', text: 'never' }, STOP_CHUNK]), + }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toEqual([ + { type: 'tool_call_start', id: 'tc1', name: 'read_file' }, + errChunk('anthropic', 'overloaded'), + ]); + expect(fallback.calls).toHaveLength(0); // a non-text content chunk commits → no failover + }); + + it('surfaces a post-content throw without failing over', async () => { + const primary = makeProvider({ + id: 'anthropic', + stream: async function* () { + yield { type: 'text_delta', text: 'partial' } satisfies StreamChunk; + await Promise.resolve(); + throw providerError('anthropic', 'transport'); + }, + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([{ type: 'text_delta', text: 'never' }, STOP_CHUNK]), + }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks[0]).toEqual({ type: 'text_delta', text: 'partial' }); + expect(chunks[1]?.type).toBe('error'); + expect(fallback.calls).toHaveLength(0); + }); + + it('fails over when the provider throws before any content (iterator creation)', async () => { + const primary = makeProvider({ + id: 'anthropic', + stream: () => streamThrowing(providerError('anthropic', 'transport')), + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([{ type: 'text_delta', text: 'recovered' }, STOP_CHUNK]), + }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toEqual([{ type: 'text_delta', text: 'recovered' }, STOP_CHUNK]); + }); + + it('stops on a fatal pre-content error and surfaces it as an error chunk', async () => { + const primary = makeProvider({ + id: 'anthropic', + stream: () => streamFrom([errChunk('anthropic', 'content_filter')]), + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([{ type: 'text_delta', text: 'never' }, STOP_CHUNK]), + }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toHaveLength(1); + expect(chunks[0]).toMatchObject({ type: 'error' }); + if (chunks[0]?.type === 'error') { + expect(chunks[0].error.kind).toBe('content_filter'); + } + expect(fallback.calls).toHaveLength(0); + }); + + it('yields a synthesized error chunk when the chain is exhausted', async () => { + const primary = makeProvider({ + id: 'anthropic', + stream: () => streamFrom([errChunk('anthropic', 'overloaded')]), + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([errChunk('openai', 'rate_limit')]), + }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.5')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toHaveLength(1); + if (chunks[0]?.type === 'error') { + expect(chunks[0].error.kind).toBe('rate_limit'); // the last attempt's error + } + }); + + it('skips a non-streaming provider and yields the cancelled error when aborted', async () => { + const nonStreaming = makeProvider({ + id: 'gemini', + supports: { streaming: false }, + stream: () => streamFrom([STOP_CHUNK]), + }); + const fallback = makeProvider({ + id: 'openai', + stream: () => streamFrom([{ type: 'text_delta', text: 'ok' }, STOP_CHUNK]), + }); + const { options, trace } = makeOptions(); + const chain = new FallbackChain( + [entry(nonStreaming, 'gemini-2.5-pro'), entry(fallback, 'gpt-5.5')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + expect(chunks).toEqual([{ type: 'text_delta', text: 'ok' }, STOP_CHUNK]); + expect(nonStreaming.calls).toHaveLength(0); + expect(trace[0]).toMatchObject({ provider: 'gemini', outcome: 'skipped' }); + + // and an already-aborted stream stops before any provider + const aborted: LlmRequest = { + ...userReq, + signal: { aborted: true, addEventListener() {}, removeEventListener() {} }, + }; + const abortedChunks = await collect(chain.stream(aborted)); + expect(abortedChunks).toHaveLength(1); + if (abortedChunks[0]?.type === 'error') { + expect(abortedChunks[0].error.kind).toBe('cancelled'); + } + }); + + it('yields a synthesized exhausted error when every stream entry is skipped', async () => { + const a = makeProvider({ + id: 'gemini', + supports: { streaming: false }, + stream: () => streamFrom([STOP_CHUNK]), + }); + const b = makeProvider({ + id: 'deepseek', + supports: { streaming: false }, + stream: () => streamFrom([STOP_CHUNK]), + }); + const { options } = makeOptions(); + const chain = new FallbackChain( + [entry(a, 'gemini-2.5-pro'), entry(b, 'deepseek-chat')], + options, + ); + + const chunks = await collect(chain.stream(userReq)); + + expect(chunks).toHaveLength(1); + if (chunks[0]?.type === 'error') { + expect(chunks[0].error.kind).toBe('unknown'); + expect(chunks[0].error.message).toMatch(/exhausted/); + } + expect(a.calls).toHaveLength(0); + expect(b.calls).toHaveLength(0); + }); + + it('stops with a cancelled chunk when aborted between stream attempts', async () => { + const signal = { aborted: false, addEventListener() {}, removeEventListener() {} }; + const provider = makeProvider({ + id: 'anthropic', + stream: () => streamFrom([errChunk('anthropic', 'overloaded')]), + }); + const { options } = makeOptions({ + sleep: () => { + signal.aborted = true; // a caller abort lands during backoff, before the next attempt + return Promise.resolve(); + }, + }); + const chain = new FallbackChain( + [{ ...entry(provider, 'claude-opus-4-8'), maxAttempts: 3 }], + options, + ); + + const chunks = await collect(chain.stream({ ...userReq, signal })); + + expect(provider.calls).toHaveLength(1); // the second attempt is cancelled before it runs + expect(chunks).toHaveLength(1); + if (chunks[0]?.type === 'error') { + expect(chunks[0].error.kind).toBe('cancelled'); + } + }); + + it('applies backoff and exhausts the budget on the streaming path too', async () => { + const provider = makeProvider({ + id: 'anthropic', + stream: () => streamFrom([errChunk('anthropic', 'overloaded')]), + }); + const { options, sleeps } = makeOptions({ backoffBaseMs: 100 }); + const chain = new FallbackChain( + [{ ...entry(provider, 'claude-opus-4-8'), maxAttempts: 2 }], + options, + ); + + await collect(chain.stream(userReq)); + + expect(provider.calls).toHaveLength(2); + expect(sleeps).toEqual([100]); // one backoff between the two attempts + }); +}); + +// --- withFallback façade --------------------------------------------------------------------- + +describe('withFallback façade', () => { + it('runs a single-shot generate and returns the result', async () => { + const provider = makeProvider({ id: 'anthropic', generate: resolves('facade') }); + const { options } = makeOptions(); + + const out = await withFallback([entry(provider, 'claude-opus-4-8')], userReq, options); + + expect(out.content).toEqual([{ type: 'text', text: 'facade' }]); + }); +}); diff --git a/packages/llm/src/fallback-chain.ts b/packages/llm/src/fallback-chain.ts new file mode 100644 index 00000000..ab27d306 --- /dev/null +++ b/packages/llm/src/fallback-chain.ts @@ -0,0 +1,641 @@ +import type { BackoffStrategy } from '@relavium/shared'; + +import type { CostTracker, CostUpdate } from './cost-tracker.js'; +import { isRetryable, LlmProviderError, makeLlmError } from './llm-error.js'; +import type { + LlmError, + LlmMessage, + LlmProvider, + LlmRequest, + LlmResult, + ProviderId, + StreamChunk, + Usage, +} from './types.js'; +import { supportsRequest } from './capabilities.js'; + +export type { BackoffStrategy }; + +/** + * The `FallbackChain` runner (1.K) — the seam's last Phase-1 policy layer. It walks an ordered plan + * of provider attempts: within an entry it retries the **same** provider up to that entry's budget on + * a classified-**retryable** `LlmError` (with backoff), then advances to the next entry; on a **fatal** + * `LlmError` it stops immediately rather than masking a real bug by falling through. Adapters stay + * dumb — fallback is **policy outside the adapter** ([ADR-0011](../../../docs/decisions/0011-internal-llm-abstraction.md), + * [llm-provider-seam.md](../../../docs/reference/shared-core/llm-provider-seam.md)). + * + * Every decision is a pure function of the classified `LlmError` discriminant (`kind`/`retryable`, + * 1.I) — **never** content/string-sentinel inspection: a body containing `'Error:'` or an empty + * body does not by itself trigger failover. The runner reuses {@link isRetryable} rather than + * re-deriving the partition, so the single source of truth holds. + * + * Four behavioural nuances the chain honours (per phase-1 §1.K): + * - **No blind auth retry.** An `auth`-class failure is never re-attempted on the same entry (it + * repeats deterministically); at most ONE out-of-band credential refresh + * ({@link FallbackChainOptions.onAuthError}) buys exactly one more attempt, never the attempt loop. + * Auth is otherwise fatal — it stops the chain. + * - **Rate-limit cooldown.** A rate-limited entry is parked in a per-provider cooldown so an + * immediately-following call on the same chain instance skips the saturated provider rather than + * hammering it again. + * - **No failover after the first content chunk.** Once a stream has emitted any content downstream, a + * mid-stream error surfaces to the node-retry layer (1.S) instead of silently re-issuing on the next + * provider — re-issuing would duplicate tokens and tool side effects. + * - **Visible failover.** Each attempt (succeeded / failed / skipped) is reported via + * {@link FallbackChainOptions.onAttempt} so the engine (1.O) can emit a `cost:updated` per attempt + * and a structured warn log — never a silent provider switch. + * + * **ADR-0030 strip-on-failover:** when advancing to a *different* provider, the runner drops every + * `reasoning` content part (and its ephemeral provider `signature`) from the request before + * re-issuing — a provider-signed signature is a same-provider, same-turn continuity token and is + * never replayed across a provider boundary. + */ + +/** + * One resolved entry in the fallback plan: a provider **instance** paired with the canonical model id + * to send and the attempt budget at that entry. The engine (1.O) builds the ordered plan from the + * agent's primary `model`/`provider` (+ `retry`) followed by each authored `fallback_chain` entry + * (`{ model, provider, max_attempts }`); the runner consumes the normalized plan and never imports + * the agent schema. + */ +export interface FallbackPlanEntry { + /** The resolved adapter for this entry (its `id` selects the credential and drives cooldown state). */ + readonly provider: LlmProvider; + /** The canonical model id sent for this entry (each entry may name a different model). */ + readonly model: string; + /** Attempt budget at this entry — a positive integer (the primary uses `retry.max`, each fallback its `max_attempts`). */ + readonly maxAttempts: number; + /** Same-model retry backoff curve for this entry; defaults to `'exponential'`. */ + readonly backoff?: BackoffStrategy; +} + +/** How one attempt in the visible trace ended. */ +export type AttemptOutcome = 'succeeded' | 'failed' | 'skipped'; + +/** + * One entry in the attempt trace — the visibility payload the engine (1.O) turns into a `cost:updated` + * per attempt and a warn log; the runner itself emits no events and imports no event bus. `error` is + * already secret-free (the adapter redacted `message`); as everywhere, a sink must never serialize + * `LlmError.cause` (the run-event error shape is only `{ code, message, retryable }`). + */ +export interface AttemptRecord { + /** + * 1-based **positional** index of this record in the current `generate`/`stream` call's trace — + * it counts skipped entries too, so it is NOT the run-event spec's per-real-call "retry attempt" + * number ([sse-event-schema.md](../../../docs/reference/contracts/sse-event-schema.md)). The + * engine (1.O) derives the `cost:updated.attemptNumber` it emits (e.g. by counting only the + * non-skipped records), rather than forwarding this field verbatim. + */ + readonly attemptNumber: number; + /** The provider this attempt targeted. */ + readonly provider: ProviderId; + /** The canonical model id this attempt used. */ + readonly model: string; + /** Whether the attempt succeeded, failed (provider error), or was skipped (capability/cooldown). */ + readonly outcome: AttemptOutcome; + /** The usage the attempt produced, when it produced any (a successful call). */ + readonly usage?: Usage; + /** The cost folded into the tracker for this attempt (present iff a `costTracker` is wired and usage existed). */ + readonly cost?: CostUpdate; + /** The classified failure on `outcome === 'failed'` (secret-free; never serialize `.cause`). */ + readonly error?: LlmError; + /** Why a `'skipped'` entry was skipped (provider in cooldown, or it can't satisfy the request). */ + readonly skipReason?: string; +} + +/** Dependencies injected into a {@link FallbackChain} — all timing is injectable so tests are deterministic. */ +export interface FallbackChainOptions { + /** + * Resolve the credential for a provider at attempt time. Host-aware in value (a resolved key on + * Node hosts, a keychain reference on the desktop, a managed token in managed mode), `string` in + * type. The runner threads it through unchanged and never logs, stores, or inspects it. + */ + readonly keyFor: (provider: ProviderId) => string | Promise; + /** + * The per-node/session cost sink. `record(model, usage)` is called once per attempt that produced + * usage, against **that attempt's** canonical model id, so cost stays accurate across a failover. + */ + readonly costTracker?: CostTracker; + /** Visibility hook fired once per attempt (succeeded / failed / skipped). */ + readonly onAttempt?: (record: AttemptRecord) => void; + /** + * The delay primitive used for backoff between same-entry retries. **Required and host-injected**: + * the seam is platform-free (no ambient `setTimeout`), so the host supplies the timer — a + * `setTimeout`-based delay on every real surface, a controllable fake in tests. + */ + readonly sleep: (ms: number) => Promise; + /** Injectable clock for cooldown bookkeeping (default: `Date.now`, an ECMAScript primitive). */ + readonly now?: () => number; + /** Base backoff delay in ms before the first retry of an entry (default 250). */ + readonly backoffBaseMs?: number; + /** Backoff delay ceiling in ms (default 8000). */ + readonly backoffMaxMs?: number; + /** How long a rate-limited provider is parked before a later call retries it (default 30000 ms). */ + readonly cooldownMs?: number; + /** + * Optional single out-of-band credential refresh on an `auth` failure. Called at most once per + * provider per chain instance; returning `true` grants exactly one more attempt at that entry, + * `false`/absent makes the auth failure fatal. Never becomes a retry loop. + */ + readonly onAuthError?: (provider: ProviderId) => boolean | Promise; +} + +const DEFAULT_BACKOFF_BASE_MS = 250; +const DEFAULT_BACKOFF_MAX_MS = 8_000; +const DEFAULT_COOLDOWN_MS = 30_000; + +/** What a classified failure means for the chain (a pure function of `LlmError.kind`). */ +type Verdict = 'fatal' | 'retryable' | 'auth-refreshed'; + +/** + * Strip every `reasoning` content part from a request's messages — a pure transform producing a new + * request (the input is never mutated). Dropping the whole part removes its ephemeral `signature` + * along with `text`/`redacted`; a message left with no content is dropped. Runs on a cross-provider + * advance so a provider-signed reasoning block never crosses a provider boundary (ADR-0030). + */ +export function stripReasoningParts(req: LlmRequest): LlmRequest { + const kept = req.messages + .map((message) => ({ + ...message, + content: message.content.filter((part) => part.type !== 'reasoning'), + })) + .filter((message) => message.content.length > 0); + // Dropping a reasoning-only message can leave two adjacent same-role messages, which strict + // providers (e.g. Anthropic) reject as a non-alternating sequence — so a failover meant to RESCUE + // the turn would instead 400. Merge adjacent same-role messages to keep the request well-formed. + // (A provider that additionally collapses distinct seam roles — Anthropic maps `tool`→user — owns + // that provider-specific normalization in its adapter; this only guarantees the seam-level shape.) + const messages: LlmMessage[] = []; + for (const message of kept) { + const previous = messages.at(-1); + if (previous !== undefined && previous.role === message.role) { + messages[messages.length - 1] = { + ...previous, + content: [...previous.content, ...message.content], + }; + } else { + messages.push(message); + } + } + return { ...req, messages }; +} + +/** The backoff delay before the `retryIndex`-th retry of an entry (0 = before the 2nd attempt). */ +function backoffDelayMs( + strategy: BackoffStrategy, + retryIndex: number, + baseMs: number, + maxMs: number, +): number { + const raw = strategy === 'exponential' ? baseMs * 2 ** retryIndex : baseMs * (retryIndex + 1); + return Math.min(raw, maxMs); +} + +/** A content chunk commits a stream — anything other than the terminal `stop`/`error` arms. */ +function isContentChunk(chunk: StreamChunk): boolean { + return chunk.type !== 'stop' && chunk.type !== 'error'; +} + +/** What one `generate` attempt produced. */ +type GenerateAttempt = + | { readonly status: 'success'; readonly result: LlmResult } + | { readonly status: 'error'; readonly error: LlmError }; + +export class FallbackChain { + readonly #plan: readonly FallbackPlanEntry[]; + readonly #options: FallbackChainOptions; + readonly #sleep: (ms: number) => Promise; + readonly #now: () => number; + readonly #backoffBaseMs: number; + readonly #backoffMaxMs: number; + readonly #cooldownMs: number; + /** Per-provider cooldown expiry (ms), persisted across calls on this instance (rate-limit nuance). */ + readonly #cooldownUntil = new Map(); + /** Providers whose one-shot auth refresh has already been spent on this instance. */ + readonly #authRefreshed = new Set(); + /** A provider id to attribute an all-skipped synthetic error to (the last plan entry's). */ + readonly #exhaustedProvider: ProviderId; + + constructor(plan: readonly FallbackPlanEntry[], options: FallbackChainOptions) { + const lastEntry = plan.at(-1); + if (lastEntry === undefined) { + // A wiring invariant, not a provider failure: the engine always supplies at least the primary. + throw new Error('FallbackChain requires at least one plan entry'); + } + for (const planEntry of plan) { + // The engine derives each budget from a schema-validated `retry.max` / `max_attempts`; guard + // here too so a miswired plan fails loudly rather than silently skipping an entry with no + // emitted attempt (which would violate the "visible, never a silent provider switch" rule). + if (!Number.isInteger(planEntry.maxAttempts) || planEntry.maxAttempts < 1) { + throw new Error('FallbackChain plan entry requires a positive integer maxAttempts'); + } + } + this.#exhaustedProvider = lastEntry.provider.id; + this.#plan = plan; + this.#options = options; + this.#sleep = options.sleep; + this.#now = options.now ?? Date.now; + this.#backoffBaseMs = options.backoffBaseMs ?? DEFAULT_BACKOFF_BASE_MS; + this.#backoffMaxMs = options.backoffMaxMs ?? DEFAULT_BACKOFF_MAX_MS; + this.#cooldownMs = options.cooldownMs ?? DEFAULT_COOLDOWN_MS; + } + + /** + * Run the chain for a non-streaming request. Returns the first entry's `LlmResult`; throws an + * `LlmProviderError` carrying the classified `LlmError` once the chain is exhausted or a fatal + * error stops it. Each attempt is reported via `onAttempt`; per-attempt usage is recorded against + * that attempt's model so cost stays accurate across a failover. + */ + async generate(req: LlmRequest): Promise { + const run = new ChainRun(req); + for (const entry of this.#plan) { + this.#throwIfAborted(req, entry.provider.id); + const skip = this.#skipReason(entry, run.previewRequest(entry)); + if (skip !== undefined) { + this.#emit(run.next(entry, { outcome: 'skipped', skipReason: skip })); + continue; + } + const entryReq = run.beginEntry(entry); // strips on a provider boundary — only for attempted entries + const result = await this.#runEntryGenerate(entry, entryReq, req, run); + if (result !== undefined) { + return result; + } + } + throw new LlmProviderError(run.lastError ?? this.#exhaustedError()); + } + + /** + * Run one entry's attempt budget (+ any auth bonus) on the non-streaming path. Returns the result + * on success, or `undefined` to advance to the next entry; throws `LlmProviderError` on a fatal + * classification (which stops the whole chain). + */ + async #runEntryGenerate( + entry: FallbackPlanEntry, + entryReq: LlmRequest, + req: LlmRequest, + run: ChainRun, + ): Promise { + const budget = entry.maxAttempts; + let bonus = 0; // one extra attempt granted by a successful auth refresh (never the retry loop) + for (let attempt = 1; attempt <= budget + bonus; attempt += 1) { + this.#throwIfAborted(req, entry.provider.id); + const outcome = await this.#runGenerateAttempt(entry, entryReq, run); + if (outcome.status === 'success') { + return outcome.result; + } + run.lastError = outcome.error; + const verdict = await this.#afterFailure(entry, outcome.error); + if (verdict === 'fatal') { + throw new LlmProviderError(outcome.error); + } + if (verdict === 'auth-refreshed') { + bonus += 1; // +1 attempt ON TOP of the configured budget; retry now (a fresh credential) + continue; + } + if (attempt < budget + bonus) { + await this.#backoff(entry, attempt - 1); + } + } + return undefined; // budget exhausted → advance to the next entry + } + + /** + * Run the chain for a streaming request. Yields the surviving provider's chunks. Failover is only + * attempted **before** the first content chunk is forwarded; once content has been emitted, a + * later error chunk is surfaced to the consumer (1.S node-retry) rather than re-issued. Like the + * seam's `stream`, a terminal failure is surfaced as an `error` chunk, not a throw. + */ + async *stream(req: LlmRequest): AsyncIterable { + const run = new ChainRun(req); + for (const entry of this.#plan) { + if (this.#aborted(req)) { + yield { type: 'error', error: this.#cancelledError(entry.provider.id) }; + return; + } + const skip = this.#skipReason(entry, run.previewRequest(entry), { streaming: true }); + if (skip !== undefined) { + this.#emit(run.next(entry, { outcome: 'skipped', skipReason: skip })); + continue; + } + const entryReq = run.beginEntry(entry); // strips on a provider boundary — only for attempted entries + const action = yield* this.#runEntryStream(entry, entryReq, req, run); + if (action === 'done') { + return; + } + } + yield { type: 'error', error: run.lastError ?? this.#exhaustedError() }; + } + + /** + * Run one entry's attempt budget (+ any auth bonus) on the streaming path. Yields the surviving + * provider's chunks; returns `'done'` when the stream is complete (a success, a committed/surfaced + * failure, a fatal pre-content stop, or a cancellation) or `'advance'` to try the next entry. + */ + async *#runEntryStream( + entry: FallbackPlanEntry, + entryReq: LlmRequest, + req: LlmRequest, + run: ChainRun, + ): AsyncGenerator { + const budget = entry.maxAttempts; + let bonus = 0; // one extra attempt granted by a successful auth refresh (never the retry loop) + for (let attempt = 1; attempt <= budget + bonus; attempt += 1) { + if (this.#aborted(req)) { + yield { type: 'error', error: this.#cancelledError(entry.provider.id) }; + return 'done'; + } + const record = run.next(entry); + const attemptState: StreamAttemptState = { committed: false }; + const failure = yield* this.#runStreamAttempt(entry, entryReq, record, attemptState); + if (attemptState.committed) { + return 'done'; // content forwarded — any later failure was surfaced inside the attempt + } + if (failure === undefined) { + return 'done'; // a clean, content-free completion (e.g. a bare `stop`) — success emitted + } + run.lastError = failure; + const verdict = await this.#afterFailure(entry, failure); + if (verdict === 'fatal') { + yield { type: 'error', error: failure }; + return 'done'; + } + if (verdict === 'auth-refreshed') { + bonus += 1; // +1 attempt ON TOP of the configured budget; retry now (a fresh credential) + continue; + } + if (attempt < budget + bonus) { + await this.#backoff(entry, attempt - 1); + } + } + return 'advance'; // budget exhausted → try the next entry + } + + /** Execute one `generate` attempt and report it; returns the success/error outcome. */ + async #runGenerateAttempt( + entry: FallbackPlanEntry, + entryReq: LlmRequest, + run: ChainRun, + ): Promise { + const record = run.next(entry); + try { + const key = await this.#resolveKey(entry.provider.id); + const result = await entry.provider.generate(entryReq, key); + this.#emitSuccess(record, entry.model, result.usage); + return { status: 'success', result }; + } catch (err) { + const error = this.#errorOf(err, entry.provider.id); + this.#emit({ ...record, outcome: 'failed', error }); + return { status: 'error', error }; + } + } + + /** + * Execute one `stream` attempt: forward every chunk, flip `state.committed` on the first content + * chunk, and surface a post-content failure inline (no failover). Returns the pre-content failure + * to fail over on, or `undefined` if the stream completed (a success was already emitted, or a + * post-content failure was surfaced). + */ + async *#runStreamAttempt( + entry: FallbackPlanEntry, + entryReq: LlmRequest, + record: AttemptRecord, + state: StreamAttemptState, + ): AsyncGenerator { + let usage: Usage | undefined; + try { + const key = await this.#resolveKey(entry.provider.id); + for await (const chunk of entry.provider.stream(entryReq, key)) { + if (chunk.type === 'error') { + this.#emit({ ...record, outcome: 'failed', error: chunk.error }); + if (state.committed) { + yield chunk; // surface a mid-stream failure; the node-retry layer (1.S) owns it + return undefined; + } + return chunk.error; // pre-content failure → caller decides failover + } + if (chunk.type === 'stop') { + usage = chunk.usage; + } + state.committed = state.committed || isContentChunk(chunk); + yield chunk; + } + } catch (err) { + const error = this.#errorOf(err, entry.provider.id); + this.#emit({ ...record, outcome: 'failed', error }); + if (state.committed) { + yield { type: 'error', error }; + return undefined; + } + return error; // pre-content throw → caller decides failover + } + this.#emitSuccess(record, entry.model, usage); + return undefined; + } + + /** + * Decide what a classified failure means for the chain and apply its side effects. Pure function of + * `error.kind`/`retryable` (1.I) — never the message. Handles the auth nuance (no blind retry; one + * optional refresh) and the rate-limit cooldown. + */ + async #afterFailure(entry: FallbackPlanEntry, error: LlmError): Promise { + if (error.kind === 'auth') { + // Never a blind retry loop — at most ONE out-of-band credential refresh, then fatal. + const hook = this.#options.onAuthError; + if (hook !== undefined && !this.#authRefreshed.has(entry.provider.id)) { + this.#authRefreshed.add(entry.provider.id); + if (await this.#refreshCredential(hook, entry.provider.id)) { + return 'auth-refreshed'; + } + } + return 'fatal'; + } + if (error.kind === 'rate_limit') { + // Park the saturated provider so a later call on this chain skips it (does not hammer it). + this.#cooldownUntil.set(entry.provider.id, this.#now() + this.#cooldownMs); + return 'retryable'; + } + return isRetryable(error.kind) ? 'retryable' : 'fatal'; + } + + /** + * Invoke the optional credential-refresh hook, treating any throw/rejection as a declined refresh. + * A misbehaving host hook must not break the runner's error contract (generate rejects with an + * `LlmProviderError`; stream yields an `error` chunk) — on a hook failure the original auth error + * stays fatal and the engine surfaces it to the run-event/log, so the throw is deliberately not + * re-raised here (the runner has no log sink of its own). + */ + async #refreshCredential( + hook: (provider: ProviderId) => boolean | Promise, + provider: ProviderId, + ): Promise { + try { + return await hook(provider); + } catch { + return false; + } + } + + /** Whether to skip an entry without consuming an attempt (cooldown or unmet capability). */ + #skipReason( + entry: FallbackPlanEntry, + req: LlmRequest, + opts?: { readonly streaming?: boolean }, + ): string | undefined { + const cooldownUntil = this.#cooldownUntil.get(entry.provider.id); + if (cooldownUntil !== undefined && this.#now() < cooldownUntil) { + return 'provider in rate-limit cooldown'; + } + if (opts?.streaming === true && !entry.provider.supports.streaming) { + return 'provider does not support streaming'; + } + if (!supportsRequest(entry.provider.supports, req)) { + return 'provider does not support a required capability'; + } + return undefined; + } + + #throwIfAborted(req: LlmRequest, provider: ProviderId): void { + if (this.#aborted(req)) { + throw new LlmProviderError(this.#cancelledError(provider)); + } + } + + #aborted(req: LlmRequest): boolean { + return req.signal?.aborted === true; + } + + #cancelledError(provider: ProviderId): LlmError { + return makeLlmError({ provider, kind: 'cancelled', message: 'request aborted' }); + } + + #exhaustedError(): LlmError { + // No real provider error to surface (every entry was skipped): synthesize a fatal one. + return makeLlmError({ + provider: this.#exhaustedProvider, + kind: 'unknown', + message: 'fallback chain exhausted: no provider could serve the request', + }); + } + + /** Normalize a thrown value into an `LlmError` — `LlmProviderError` carries one; anything else is `unknown`. */ + #errorOf(caught: unknown, provider: ProviderId): LlmError { + if (caught instanceof LlmProviderError) { + return caught.llmError; + } + return makeLlmError({ + provider, + kind: 'unknown', + message: caught instanceof Error ? caught.message : 'unknown provider failure', + cause: caught, + }); + } + + async #backoff(entry: FallbackPlanEntry, retryIndex: number): Promise { + const delay = backoffDelayMs( + entry.backoff ?? 'exponential', + retryIndex, + this.#backoffBaseMs, + this.#backoffMaxMs, + ); + await this.#sleep(delay); + } + + async #resolveKey(provider: ProviderId): Promise { + return this.#options.keyFor(provider); + } + + /** + * Emit a success record, folding usage into the cost tracker. `usage`/`cost` are included only when + * present (a stream that ended without a `stop` chunk has no usage) — `exactOptionalPropertyTypes` + * forbids an explicit `undefined` on an optional field. + */ + #emitSuccess(record: AttemptRecord, model: string, usage: Usage | undefined): void { + if (usage === undefined) { + this.#emit({ ...record, outcome: 'succeeded' }); + return; + } + const cost = this.#options.costTracker?.record(model, usage); + this.#emit({ + ...record, + outcome: 'succeeded', + usage, + ...(cost === undefined ? {} : { cost }), + }); + } + + #emit(record: AttemptRecord): void { + this.#options.onAttempt?.(record); + } +} + +/** Mutable per-attempt flag: whether any content chunk has been forwarded (commits the stream). */ +interface StreamAttemptState { + committed: boolean; +} + +/** + * Per-call mutable state: the running (possibly reasoning-stripped) request, the per-provider strip + * latch, the attempt counter, and the most recent failure for exhaustion surfacing. + */ +class ChainRun { + #req: LlmRequest; + #lastProvider: ProviderId | undefined; + #attemptNumber = 0; + lastError: LlmError | undefined; + + constructor(req: LlmRequest) { + this.#req = req; + } + + /** + * The request a skip check sees — the running (already-stripped) request with this entry's model, + * **without** advancing the strip latch. A skipped entry is not a provider boundary, so it must not + * pollute `#lastProvider` (which would wrongly strip reasoning for a later same-provider entry). + */ + previewRequest(entry: FallbackPlanEntry): LlmRequest { + return { ...this.#req, model: entry.model }; + } + + /** + * Begin an entry that will actually be attempted: return the request to send (its model), stripping + * reasoning parts once a provider boundary is crossed (ADR-0030). The strip mutates the running + * request permanently for the rest of the call (idempotent), so reasoning never reaches any provider + * past the originating one. Called once per attempted entry — after the skip check, never for a + * skipped entry — so the latch tracks only providers that were actually invoked. + */ + beginEntry(entry: FallbackPlanEntry): LlmRequest { + const providerId = entry.provider.id; + if (this.#lastProvider !== undefined && this.#lastProvider !== providerId) { + this.#req = stripReasoningParts(this.#req); + } + this.#lastProvider = providerId; + return { ...this.#req, model: entry.model }; + } + + /** Allocate the next 1-based attempt record skeleton for this entry. */ + next( + entry: FallbackPlanEntry, + extra?: Pick, + ): AttemptRecord { + this.#attemptNumber += 1; + return { + attemptNumber: this.#attemptNumber, + provider: entry.provider.id, + model: entry.model, + outcome: extra?.outcome ?? 'failed', + ...(extra?.skipReason === undefined ? {} : { skipReason: extra.skipReason }), + }; + } +} + +/** + * Single-shot façade over {@link FallbackChain} for the common non-streaming case — constructs a + * transient chain and runs `generate`. Use the class directly when you need streaming or want the + * per-provider rate-limit cooldown to persist across calls. + */ +export function withFallback( + plan: readonly FallbackPlanEntry[], + req: LlmRequest, + options: FallbackChainOptions, +): Promise { + return new FallbackChain(plan, options).generate(req); +} diff --git a/packages/llm/src/index.ts b/packages/llm/src/index.ts index b315d091..4a5feeef 100644 --- a/packages/llm/src/index.ts +++ b/packages/llm/src/index.ts @@ -124,6 +124,16 @@ export type { ModelPricing, CanonicalModelId } from './pricing.js'; export { priceModel, cost, CostTracker } from './cost-tracker.js'; export type { CostUpdate } from './cost-tracker.js'; +// FallbackChain runner — fallback policy outside the adapters (1.K). +export { FallbackChain, withFallback, stripReasoningParts } from './fallback-chain.js'; +export type { + FallbackPlanEntry, + FallbackChainOptions, + AttemptRecord, + AttemptOutcome, + BackoffStrategy, +} from './fallback-chain.js'; + // ToolNormalizer (1.E). export { toWire, diff --git a/packages/llm/src/pricing.ts b/packages/llm/src/pricing.ts index 64adda9b..65bdac46 100644 --- a/packages/llm/src/pricing.ts +++ b/packages/llm/src/pricing.ts @@ -11,10 +11,15 @@ import type { ProviderId } from './types.js'; * * USD/MTok → micro-cents/MTok is `usd × 1e8` (e.g. $5.00 → 500_000_000). * - * **Verification.** Anthropic figures are confirmed against the model catalog (claude-api, - * 2026-05-26). The OpenAI / Gemini / DeepSeek figures are best-known **placeholders — VERIFY - * against each provider's pricing page before relying on a cost figure**; updating one is a - * one-line edit. (Gemini's real pricing is context-tiered; the ≤128K tier is used here.) + * **Verification (2026-06-11).** Every row verified against the provider's live pricing page: + * Anthropic via the claude-api pricing page (Opus 4.8 / Sonnet 4.6 / Haiku 4.5 unchanged; the + * flagship **Claude Fable 5** added); OpenAI (developers.openai.com), Gemini (ai.google.dev), and + * DeepSeek (api-docs.deepseek.com) re-fetched the same day. The prior + * `gpt-4o` / `gpt-4o-mini` / `gemini-2.0-flash` / `gemini-1.5-pro` rows were retired — shut down or + * removed from the provider catalogs — and replaced with the current flagship/mini and Pro/Flash + * models. Gemini Pro/Flash are context-tiered: the ≤200K (Pro) and text/image/video (Flash) tier is + * used here. DeepSeek's `deepseek-chat`/`-reasoner` ids alias `deepseek-v4-flash` (non-thinking / + * thinking) and are scheduled for deprecation on 2026-07-24 — re-verify then. */ export interface ModelPricing { @@ -39,7 +44,18 @@ const usd = (perMtok: number): number => Math.round(perMtok * USD_PER_MTOK_TO_MI /** Canonical model id → pricing. The canonical id is what an authored agent/workflow names. */ export const MODEL_PRICING = { - // --- Anthropic (confirmed: claude-api model catalog, 2026-05-26) ---------------------------- + // --- Anthropic (verified 2026-06-11: platform.claude.com pricing page) ---------------------- + 'claude-fable-5': { + provider: 'anthropic', + nativeId: 'claude-fable-5', + displayName: 'Claude Fable 5', + contextWindowTokens: 1_000_000, + maxOutputTokens: 128_000, + inputPerMtokMicrocents: usd(10), + outputPerMtokMicrocents: usd(50), + cachedInputPerMtokMicrocents: usd(1), // cache read = 0.1× input + cacheWritePerMtokMicrocents: usd(12.5), // cache write (5-min TTL) = 1.25× input + }, 'claude-opus-4-8': { provider: 'anthropic', nativeId: 'claude-opus-4-8', @@ -74,70 +90,71 @@ export const MODEL_PRICING = { cacheWritePerMtokMicrocents: usd(1.25), }, - // --- OpenAI (VERIFY against platform.openai.com/pricing) ------------------------------------ - 'gpt-4o': { + // --- OpenAI (verified 2026-06-11: developers.openai.com/api/docs/pricing) ------------------- + 'gpt-5.5': { provider: 'openai', - nativeId: 'gpt-4o', - displayName: 'GPT-4o', - contextWindowTokens: 128_000, - maxOutputTokens: 16_384, - inputPerMtokMicrocents: usd(2.5), - outputPerMtokMicrocents: usd(10), - cachedInputPerMtokMicrocents: usd(1.25), // OpenAI auto-caches; no separate write charge + nativeId: 'gpt-5.5', + displayName: 'GPT-5.5', + contextWindowTokens: 1_000_000, + maxOutputTokens: 128_000, + inputPerMtokMicrocents: usd(5), + outputPerMtokMicrocents: usd(30), + cachedInputPerMtokMicrocents: usd(0.5), // OpenAI auto-caches; no separate write charge }, - 'gpt-4o-mini': { + 'gpt-5.4-mini': { provider: 'openai', - nativeId: 'gpt-4o-mini', - displayName: 'GPT-4o mini', - contextWindowTokens: 128_000, - maxOutputTokens: 16_384, - inputPerMtokMicrocents: usd(0.15), - outputPerMtokMicrocents: usd(0.6), + nativeId: 'gpt-5.4-mini', + displayName: 'GPT-5.4 mini', + contextWindowTokens: 400_000, + maxOutputTokens: 128_000, + inputPerMtokMicrocents: usd(0.75), + outputPerMtokMicrocents: usd(4.5), cachedInputPerMtokMicrocents: usd(0.075), }, - // --- Gemini (VERIFY against ai.google.dev/pricing; real pricing is context-tiered) ---------- - 'gemini-2.0-flash': { + // --- Gemini (verified 2026-06-11: ai.google.dev/gemini-api/docs/pricing; context-tiered) ---- + 'gemini-2.5-flash': { provider: 'gemini', - nativeId: 'gemini-2.0-flash', - displayName: 'Gemini 2.0 Flash', + nativeId: 'gemini-2.5-flash', + displayName: 'Gemini 2.5 Flash', contextWindowTokens: 1_048_576, - maxOutputTokens: 8_192, - inputPerMtokMicrocents: usd(0.1), - outputPerMtokMicrocents: usd(0.4), - cachedInputPerMtokMicrocents: usd(0.025), + maxOutputTokens: 65_536, + inputPerMtokMicrocents: usd(0.3), // text/image/video tier (audio: $1.00/MTok) + outputPerMtokMicrocents: usd(2.5), + cachedInputPerMtokMicrocents: usd(0.03), }, - 'gemini-1.5-pro': { + 'gemini-2.5-pro': { provider: 'gemini', - nativeId: 'gemini-1.5-pro', - displayName: 'Gemini 1.5 Pro', - contextWindowTokens: 2_097_152, - maxOutputTokens: 8_192, - inputPerMtokMicrocents: usd(1.25), - outputPerMtokMicrocents: usd(5), - cachedInputPerMtokMicrocents: usd(0.3125), + nativeId: 'gemini-2.5-pro', + displayName: 'Gemini 2.5 Pro', + contextWindowTokens: 1_048_576, + maxOutputTokens: 65_536, + inputPerMtokMicrocents: usd(1.25), // prompts ≤200K tier (>200K: $2.50 in / $15 out) + outputPerMtokMicrocents: usd(10), + cachedInputPerMtokMicrocents: usd(0.125), }, - // --- DeepSeek (VERIFY against api-docs.deepseek.com; served via the OpenAI-compatible adapter) - + // --- DeepSeek (verified 2026-06-11: api-docs.deepseek.com; via the OpenAI-compatible adapter) - + // deepseek-chat/-reasoner alias deepseek-v4-flash (non-thinking / thinking); deprecating 2026-07-24. 'deepseek-chat': { provider: 'deepseek', nativeId: 'deepseek-chat', - displayName: 'DeepSeek-V3 (chat)', - contextWindowTokens: 64_000, - maxOutputTokens: 8_192, - inputPerMtokMicrocents: usd(0.27), - outputPerMtokMicrocents: usd(1.1), - cachedInputPerMtokMicrocents: usd(0.07), // cache-hit input + displayName: 'DeepSeek-V4-Flash (chat)', + contextWindowTokens: 1_000_000, + maxOutputTokens: 384_000, + inputPerMtokMicrocents: usd(0.14), + outputPerMtokMicrocents: usd(0.28), + cachedInputPerMtokMicrocents: usd(0.0028), // cache-hit input }, 'deepseek-reasoner': { provider: 'deepseek', nativeId: 'deepseek-reasoner', - displayName: 'DeepSeek-R1 (reasoner)', - contextWindowTokens: 64_000, - maxOutputTokens: 8_192, - inputPerMtokMicrocents: usd(0.55), - outputPerMtokMicrocents: usd(2.19), - cachedInputPerMtokMicrocents: usd(0.14), + displayName: 'DeepSeek-V4-Flash (reasoner)', + contextWindowTokens: 1_000_000, + maxOutputTokens: 384_000, + inputPerMtokMicrocents: usd(0.435), + outputPerMtokMicrocents: usd(0.87), + cachedInputPerMtokMicrocents: usd(0.003625), }, } as const satisfies Readonly>; diff --git a/packages/llm/src/types.test.ts b/packages/llm/src/types.test.ts index 2b0f49da..59d54d47 100644 --- a/packages/llm/src/types.test.ts +++ b/packages/llm/src/types.test.ts @@ -79,7 +79,7 @@ describe('seam request/message/tool schemas', () => { it('accepts a request with tools, toolChoice, and the providerOptions escape hatch', () => { expect( LlmRequestSchema.safeParse({ - model: 'gpt-4o', + model: 'gpt-5.5', system: 'be terse', messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], tools: [{ name: 'read_file', parameters: { type: 'object' } }], diff --git a/packages/shared/src/agent.ts b/packages/shared/src/agent.ts index 2ef2c418..52eafd7a 100644 --- a/packages/shared/src/agent.ts +++ b/packages/shared/src/agent.ts @@ -28,6 +28,7 @@ export const ProviderSchema = z.enum(LLM_PROVIDERS); /** Backoff curve for `retry` and for engine-side retry config. */ export const BackoffStrategySchema = z.enum(['linear', 'exponential']); +export type BackoffStrategy = z.infer; /** Transient-error retry on the *same* model. */ export const RetrySchema = z