Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 30 additions & 7 deletions docs/reference/shared-core/llm-provider-seam.md
Original file line number Diff line number Diff line change
Expand Up @@ -561,13 +561,36 @@ the seam types are unchanged. The full metering design is in
## Fallback lives outside the adapter

Adapters stay **dumb**: they normalize one provider and nothing more. Fallback
chains are **policy**, implemented in a small `withFallback(providers:
LlmProvider[])` runner (in `packages/core`/`packages/llm`):

- Try `providers[0]`; on a **classified-retryable** `LlmError` (rate limit,
5xx, overload) move to the next provider.
- Surface **per-attempt usage** so cost accounting stays accurate even across a
failover.
chains are **policy**, implemented by the `FallbackChain` runner (1.K) in
`@relavium/llm` — a class constructed from an ordered plan of attempts
(`{ provider, model, maxAttempts, backoff }`), plus a thin
`withFallback(plan, req, options)` façade for the common single-shot
non-streaming case. It exposes `generate(req)` and `stream(req)`, and the engine
(1.O) builds the plan from the agent's primary `model`/`provider` (+ `retry`)
followed by each authored `fallback_chain` entry:

- Try each entry in order; on a **classified-retryable** `LlmError` (rate limit,
5xx/overload, timeout, transport) exhaust the entry's `maxAttempts` with
backoff, then advance to the next entry. On a **fatal** `LlmError` stop
immediately. The advance/stop decision is a pure function of the classified
`kind`/`retryable` (1.I) — never content/string-sentinel inspection.
- **No blind auth retry:** an `auth` failure is never re-attempted on the same
entry; an optional out-of-band credential refresh (`onAuthError`) may grant
exactly one more attempt, otherwise it is fatal.
- **Rate-limit cooldown:** a rate-limited entry is parked in a per-provider
cooldown so an immediately-following call on the same chain skips it.
- **No failover after the first streamed content chunk:** once `stream` has
forwarded content, a mid-stream error surfaces to the node-retry layer (1.S)
rather than re-issuing on the next provider.
- **Strip-on-failover (ADR-0030):** crossing to a *different* provider drops
every `reasoning` part (and its ephemeral `signature`) from the request before
re-issuing — a signature is never replayed across a provider boundary.
- Surface **per-attempt usage** to the injected `CostTracker` (against that
attempt's model) so cost stays accurate across a failover, and report each
attempt (succeeded / failed / skipped) via an `onAttempt` observer so the
engine can emit a `cost:updated` per attempt and a warn log — **visible**
failover, never a silent provider switch. The runner imports no event bus; it
is platform-free (the host injects the `sleep` timer).

This keeps per-agent fallback (e.g. Anthropic → OpenAI → DeepSeek) a config
concern declared in [`agent-yaml-spec.md`](../contracts/agent-yaml-spec.md)
Expand Down
15 changes: 9 additions & 6 deletions docs/roadmap/current.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> Status: Living

> Last updated: 2026-06-10
> Last updated: 2026-06-11

- **Related**: [README.md](README.md), [phases/phase-0-foundations.md](phases/phase-0-foundations.md), [phases/phase-1-engine-and-llm.md](phases/phase-1-engine-and-llm.md), [../project-structure.md](../project-structure.md), [../tech-stack.md](../tech-stack.md)

Expand Down Expand Up @@ -134,11 +134,14 @@ work runs two parallel lanes:
> input/engine/output + surfaces) are additive and do NOT gate M2** — the seam lane proceeds straight
> to **1.K**.

> **MCP client scheduled (2026-06-10).** The long-standing contract-ahead-of-implementation gap is
> closed at the decision layer: [ADR-0034](../decisions/0034-mcp-client-sdk-dependency.md) pins the
> official TypeScript MCP SDK as the client implementation and binds the inbound client to
> **workstream 2.R** at the start of [build phase 2](phases/phase-2-cli.md) (off the M3 critical
> path). No Phase-1 work changes.
> **Review-pass follow-ups landed (PR #12, merged 2026-06-11).** The 2026-06-10 engine/tooling
> review pass landed as docs/decisions only — no Phase-1 workstream changed: **MCP client scheduling**
> ([ADR-0034](../decisions/0034-mcp-client-sdk-dependency.md) pins the official TypeScript MCP SDK and
> binds the inbound client to **workstream 2.R** at the start of [build phase 2](phases/phase-2-cli.md),
> off the M3 critical path); the **`turn_limit` `ErrorCode`** (a hard session turn cap, distinct from
> the `[chat].max_messages` trim threshold); the **reserved `on_error` edge kind**
> (workflow-yaml-spec.md, not authorable in v1.0); and a CI **engine dependency-allowlist guard** + the
> pnpm install-script allowlist. No Phase-1 work changes; 1.K/1.L remain the next workstreams.

Carry-over hardening is tracked in [deferred-tasks.md](deferred-tasks.md) — pick items up as Phase 1
first touches each file.
Expand Down
32 changes: 19 additions & 13 deletions docs/roadmap/deferred-tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> Status: Living

> Last updated: 2026-06-10
> Last updated: 2026-06-11

- **Related**: [current.md](current.md), [README.md](README.md), [phases/phase-0-foundations.md](phases/phase-0-foundations.md)

Expand Down Expand Up @@ -34,14 +34,16 @@ Severity is the review's verified rating. Check an item off in the PR that resol
> multimodal forward-obligations** below (SSRF primitive, async-job ADR, media cost estimate, `partialRef`
> semantics, `workspace` authz scope, retention/GC table, `vision`-alias retirement) so nothing is lost.

> **2026-06-10 engine/tooling review pass:** a review of the engine, tool, and CI surfaces against
> the current contracts produced a small set of additions, recorded in their sections below: the
> **tool-output size gate + spill-to-disk** (1.T), **conformance tool-loop / cache-hit scenarios**
> (1.F follow-up), a **token-estimate accuracy** watch item (1.AC), the **Leakwatch CI gate**
> (deferred pending a distribution path), and a **dependency-bump cooling window** (pending a pnpm
> major). The same pass settled three decisions outside this file: the MCP client dependency and
> scheduling (ADR-0034 / workstream 2.R), the reserved `on_error` edge kind
> (workflow-yaml-spec.md), and the `turn_limit` `ErrorCode` (constants.ts + sse-event-schema.md).
> **2026-06-10 engine/tooling review pass (landed in PR #12, merged 2026-06-11):** a review of the
> engine, tool, and CI surfaces against the current contracts produced a small set of additions,
> recorded in their sections below: the **tool-output size gate + spill-to-disk** (1.T),
> **conformance tool-loop / cache-hit scenarios** (1.F follow-up), a **token-estimate accuracy** watch
> item (1.AC), the **Leakwatch CI gate** (deferred pending a distribution path), and a
> **dependency-bump cooling window** (pending a pnpm major). The same pass settled three decisions
> outside this file: the MCP client dependency and scheduling (ADR-0034 / workstream 2.R), the
> reserved `on_error` edge kind (workflow-yaml-spec.md), and the `turn_limit` `ErrorCode`
> (constants.ts + sse-event-schema.md). It also landed a CI **engine dependency-allowlist guard**
> (`tools/engine-deps/check.mjs`) and the pnpm **install-script allowlist**.

## Decisions needed (maintainer call)

Expand Down Expand Up @@ -233,10 +235,14 @@ Severity is the review's verified rating. Check an item off in the PR that resol
doesn't define which keys are legal per `InputType`. Specify that matrix, then add a
`WorkflowInputSchema.superRefine((type, validation) => …)`. *(minor · workflow.ts,
workflow-yaml-spec.md)*
- [ ] **Verify the non-Anthropic prices in `pricing.ts` (at 1.G/1.H)** — the OpenAI / Gemini /
DeepSeek rows are best-known **placeholders** (Anthropic is confirmed via claude-api). Verify
each against the provider's pricing page when its adapter lands, and replace Gemini's flat
≤128K-tier figures if context-tiered pricing matters. *(low → 1.G/1.H · packages/llm/src/pricing.ts)*
- [x] **Verify the non-Anthropic prices in `pricing.ts` (at 1.G/1.H)** — the OpenAI / Gemini /
DeepSeek rows were best-known **placeholders** (Anthropic confirmed via claude-api). **Done
2026-06-11:** verified against each provider's live pricing page, which revealed five of the six
non-Anthropic models were deprecated/shut down — retired and replaced with current models
(gpt-4o→gpt-5.5, gpt-4o-mini→gpt-5.4-mini, gemini-2.0-flash→gemini-2.5-flash,
gemini-1.5-pro→gemini-2.5-pro), DeepSeek prices corrected (deepseek-chat/-reasoner now distinct,
ctx 1M / 384K out, deprecating 2026-07-24), and **Claude Fable 5** added; Opus 4.8 / Sonnet 4.6 /
Haiku 4.5 re-confirmed unchanged. *(packages/llm/src/pricing.ts)*
- [x] **`model_catalog` cache-write column (at the seeder)** — `ModelPricing` carries
`cacheWritePerMtokMicrocents` (Anthropic charges one), but `model_catalog`
([database-schema.md](../reference/desktop/database-schema.md)) has only
Expand Down
12 changes: 6 additions & 6 deletions packages/llm/src/adapters/gemini.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ function fakeTransport(
}

const REQ: LlmRequest = {
model: 'gemini-2.0-flash',
model: 'gemini-2.5-flash',
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
};

Expand All @@ -73,7 +73,7 @@ describe('Gemini adapter', () => {
const transport = fakeTransport({ candidates: [] });
const adapter = createGeminiAdapter({ transport });
const req: LlmRequest = {
model: 'gemini-2.0-flash',
model: 'gemini-2.5-flash',
messages: [
{
role: 'user',
Expand Down Expand Up @@ -168,7 +168,7 @@ describe('geminiErrorToLlmError — classification', () => {
describe('Gemini adapter — request building (buildGeminiRequest)', () => {
it('routes system → systemInstruction, tools → functionDeclarations, and tool choice modes', () => {
const request = buildGeminiRequest({
model: 'gemini-2.0-flash',
model: 'gemini-2.5-flash',
system: 'be terse',
temperature: 0.4,
maxTokens: 64,
Expand Down Expand Up @@ -217,7 +217,7 @@ describe('Gemini adapter — request building (buildGeminiRequest)', () => {

it('round-trips tool_call → functionCall and tool_result → functionResponse by name', () => {
const request = buildGeminiRequest({
model: 'gemini-2.0-flash',
model: 'gemini-2.5-flash',
messages: [
{
role: 'assistant',
Expand Down Expand Up @@ -251,7 +251,7 @@ describe('Gemini adapter — request building (buildGeminiRequest)', () => {

it('wraps a non-object tool result and lets providerOptions only ADD (mapped fields win)', () => {
const request = buildGeminiRequest({
model: 'gemini-2.0-flash',
model: 'gemini-2.5-flash',
providerOptions: { cachedContent: 'abc', temperature: 99 },
temperature: 0.1,
messages: [
Expand Down Expand Up @@ -398,7 +398,7 @@ describe('Gemini adapter — remaining branches', () => {

it('drops a message that lowers to zero parts', () => {
const request = buildGeminiRequest({
model: 'gemini-2.0-flash',
model: 'gemini-2.5-flash',
messages: [
{ role: 'user', content: [] },
{ role: 'user', content: [{ type: 'text', text: 'hi' }] },
Expand Down
36 changes: 18 additions & 18 deletions packages/llm/src/adapters/openai.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ const completion = (message: unknown, finishReason = 'stop'): string =>
id: 'c',
object: 'chat.completion',
created: 0,
model: 'gpt-4o',
model: 'gpt-5.5',
choices: [{ index: 0, message, finish_reason: finishReason, logprobs: null }],
usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
});
Expand All @@ -62,7 +62,7 @@ const streamChunk = (choices: readonly unknown[]): Record<string, unknown> => ({
id: 's',
object: 'chat.completion.chunk',
created: 0,
model: 'gpt-4o',
model: 'gpt-5.5',
choices,
});

Expand All @@ -71,7 +71,7 @@ const dchunk = (delta: unknown, finish: string | null = null): Record<string, un
id: 's',
object: 'chat.completion.chunk',
created: 0,
model: 'gpt-4o',
model: 'gpt-5.5',
choices: [{ index: 0, delta, finish_reason: finish }],
});

Expand All @@ -98,7 +98,7 @@ describe('OpenAI-compatible adapter', () => {
fetch: () => Promise.reject(new Error('must fail fast before any egress')),
});
const req = {
model: 'gpt-4o',
model: 'gpt-5.5',
messages: [
{
role: 'user' as const,
Expand Down Expand Up @@ -227,7 +227,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () =>
});
await adapter.generate(
{
model: 'gpt-4o',
model: 'gpt-5.5',
system: 'be terse',
toolChoice: 'required',
tools: [{ name: 'get_weather', parameters: { type: 'object' } }],
Expand Down Expand Up @@ -278,7 +278,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () =>
});
await adapter.generate(
{
model: 'gpt-4o',
model: 'gpt-5.5',
temperature: 0.5,
stopSequences: ['STOP'],
providerOptions: { seed: 42, model: 'attacker-override' },
Expand All @@ -289,7 +289,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () =>
expect(sent['temperature']).toBe(0.5);
expect(sent['stop']).toEqual(['STOP']);
expect(sent['seed']).toBe(42); // escape-hatch field reached the wire
expect(sent['model']).toBe('gpt-4o'); // mapped field wins over providerOptions
expect(sent['model']).toBe('gpt-5.5'); // mapped field wins over providerOptions
});

it('maps tool_choice {name} to a named function choice', async () => {
Expand All @@ -303,7 +303,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () =>
});
await adapter.generate(
{
model: 'gpt-4o',
model: 'gpt-5.5',
toolChoice: { name: 'get_weather' },
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
},
Expand Down Expand Up @@ -337,7 +337,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () =>
let caught: unknown;
try {
await adapter.generate(
{ model: 'gpt-4o', messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }] },
{ model: 'gpt-5.5', messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }] },
SECRET,
);
} catch (err) {
Expand All @@ -355,7 +355,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () =>

describe('OpenAI-compatible adapter — stream edge cases', () => {
const REQ = {
model: 'gpt-4o',
model: 'gpt-5.5',
messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'hi' }] }],
};

Expand Down Expand Up @@ -387,7 +387,7 @@ describe('OpenAI-compatible adapter — stream edge cases', () => {
id: 's',
object: 'chat.completion.chunk',
created: 0,
model: 'gpt-4o',
model: 'gpt-5.5',
// a fragment with no preceding id+name for index 0 — can't start a tool, skipped
choices: [
{
Expand All @@ -401,7 +401,7 @@ describe('OpenAI-compatible adapter — stream edge cases', () => {
id: 's',
object: 'chat.completion.chunk',
created: 0,
model: 'gpt-4o',
model: 'gpt-5.5',
choices: [{ index: 0, delta: { content: 'hi' }, finish_reason: 'stop' }],
},
]),
Expand Down Expand Up @@ -432,7 +432,7 @@ describe('OpenAI-compatible adapter — stream edge cases', () => {

describe('OpenAI-compatible adapter — additional fold + generate branches', () => {
const REQ = {
model: 'gpt-4o',
model: 'gpt-5.5',
messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'hi' }] }],
};

Expand Down Expand Up @@ -500,7 +500,7 @@ describe('OpenAI-compatible adapter — additional fold + generate branches', ()
id: 'c',
object: 'chat.completion',
created: 0,
model: 'gpt-4o',
model: 'gpt-5.5',
choices: [],
}),
{ status: 200, headers: { 'content-type': 'application/json' } },
Expand Down Expand Up @@ -568,7 +568,7 @@ describe('OpenAI-compatible adapter — reasoning + structured output (ADR-0030)
});
await adapter.generate(
{
model: 'gpt-4o',
model: 'gpt-5.5',
responseFormat: { type: 'json', schema: { type: 'object' }, name: 'out' },
messages: REQ.messages,
},
Expand Down Expand Up @@ -665,7 +665,7 @@ describe('OpenAI-compatible adapter — robustness (review fixes)', () => {
});
await adapter.generate(
{
model: 'gpt-4o',
model: 'gpt-5.5',
responseFormat: { type: 'json', schema: { type: 'object' }, name: 'my schema!' },
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
},
Expand Down Expand Up @@ -756,7 +756,7 @@ describe('OpenAI-compatible adapter — baseURL SSRF guard', () => {

describe('OpenAI-compatible adapter — truncation + refusal normalization', () => {
const REQ = {
model: 'gpt-4o',
model: 'gpt-5.5',
messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'hi' }] }],
};

Expand Down Expand Up @@ -812,7 +812,7 @@ describe('OpenAI-compatible adapter — truncation + refusal normalization', ()
id: 'c',
object: 'chat.completion',
created: 0,
model: 'gpt-4o',
model: 'gpt-5.5',
choices: [
{
index: 0,
Expand Down
2 changes: 1 addition & 1 deletion packages/llm/src/conformance/gemini.conformance.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ describe('gemini — conformance (live, nightly)', () => {
it.skipIf(liveKey === '')('generate hits the real API and returns canonical text', async () => {
const result = await geminiAdapter.generate(
{
model: 'gemini-2.0-flash',
model: 'gemini-2.5-flash',
maxTokens: 16,
messages: [{ role: 'user', content: [{ type: 'text', text: 'Reply with one word.' }] }],
},
Expand Down
2 changes: 1 addition & 1 deletion packages/llm/src/conformance/openai.conformance.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ describe('openai — conformance (live, nightly)', () => {
it.skipIf(liveKey === '')('generate hits the real API and returns canonical text', async () => {
const result = await openaiAdapter.generate(
{
model: 'gpt-4o-mini',
model: 'gpt-5.4-mini',
maxTokens: 16,
messages: [{ role: 'user', content: [{ type: 'text', text: 'Reply with one word.' }] }],
},
Expand Down
Loading
Loading