diff --git a/README.md b/README.md index 2dfcd49..c2319b0 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ graph TD ## 1b. Container Health Checks & Auto-Restart -All core containers are configured with health checks in the Quadlet templates under [`quadlets/`](quadlets/). The legacy [`pod.yaml`](pod.yaml) is retained as a compatibility template. Quadlet `HealthOnFailure=kill` together with systemd `Restart=always` enables automatic recovery from unhealthy containers. +All core containers are configured with health checks in the Quadlet templates under [`quadlets/`](https://github.com/sheepdestroyer/LLM-Routing/tree/main/quadlets). The legacy [`pod.yaml`](https://github.com/sheepdestroyer/LLM-Routing/blob/main/pod.yaml) is retained as a compatibility template. Quadlet `HealthOnFailure=kill` together with systemd `Restart=always` enables automatic recovery from unhealthy containers. | Container | Liveness Probe | Readiness Probe | |:---|---:|---:| @@ -770,7 +770,7 @@ To maximize throughput under concurrent queries, `llama-server` is configured wi #### 3. Custom Memory Endpoint Proxy & MCP Server To allow Goose (and other agents) to store, list, and delete persistent preference/factual memories, we implemented a custom memory stack: * **Triage Router Memory Proxy**: Exposes a catch-all route `@app.api_route("/v1/memory{path:path}", methods=["GET", "POST", "DELETE", "PUT"])` in `router/main.py` that intercepts memory calls and proxies them to the LiteLLM gateway (port 4000) using the securely-loaded `LITELLM_MASTER_KEY` authorization. -* **Memory MCP Bridge Server**: Created a custom stdio MCP server in [memory_mcp.py](router/memory_mcp.py) that exposes the `rememberMemory`, `retrieveMemories`, and `removeSpecificMemory` tools. The script proxies these commands directly to `http://localhost:5000/v1/memory`. +* **Memory MCP Bridge Server**: Created a custom stdio MCP server in [memory_mcp.py](https://github.com/sheepdestroyer/LLM-Routing/blob/main/router/memory_mcp.py) that exposes the `rememberMemory`, `retrieveMemories`, and `removeSpecificMemory` tools. The script proxies these commands directly to `http://localhost:5000/v1/memory`. * **Goose Integration**: The built-in memory extension is disabled in `~/.config/goose/config.yaml` and replaced with the `litellm-memory` custom command-line extension running our bridge server. ## 9c. Ollama Proxy Integration (via LiteLLM ollama_chat) @@ -816,10 +816,31 @@ For auto-routing modes, the Triage Router handles failures by silently falling b ## 9d. Live Stack Tier Testing & Verification The repository includes an automated integration script to test the 5-tier intent routing pipeline on the live gateway stack: -* **Location**: [verify_reasoning_tiers.py](scripts/verification/verify_reasoning_tiers.py) +* **Location**: [verify_reasoning_tiers.py](https://github.com/sheepdestroyer/LLM-Routing/blob/main/scripts/verification/verify_reasoning_tiers.py) This script acts as an end-to-end routing smoke test by sending five sequential chat completion requests (from simple to advanced prompt complexities) to the gateway's `llm-routing-auto-free` auto-triage route, verifying that: 1. The gateway successfully routes the prompt to the expected LiteLLM model group or provider. + +## 9e. Home Assistant Integration & Responses API Support + +LLM-Routing provides full compatibility with Home Assistant's `openai_conversation` official integration, supporting both legacy Chat Completions (`/v1/chat/completions`) and the OpenAI Responses API (`/v1/responses` and `/responses`). + +### Configuration in Home Assistant +* **Base URL**: `https://litellm.x570.vendeuvre.lan/v1` (Production) or `https://dev.vendeuvre.lan/v1` (Dev) +* **API Key**: Any valid LiteLLM key or master key +* **Supported Models for Home Assistant**: + * `local-qwen-3.6-hass` (Recommended: local Qwen model with thinking disabled for fast Assist action responses) + * `local-qwen-3.6` (Local Qwen model with preserve_thinking enabled) + * `gpt-4o-mini` (Model alias provided for Home Assistant configuration flows & AI task options) + * `gpt-4o` (Model alias provided for high-capability task options) + * `llm-routing-auto-free` (Automatic complexity classification across free tiers) + +### Responses API & Tools Compatibility +* **Endpoints**: Exposed on both `POST /v1/responses` and `POST /responses`. +* **Home Location Lookup**: `client.responses.create` calls issued during HA configuration using `gpt-4o-mini` succeed without `Invalid model name` errors. +* **Assist Tools / Function Calling**: Fully supported. Home Assistant tool definitions (`HassTurnOn`, `HassTurnOff`, custom intent schemas) are correctly formatted and returned as `function_call` output items. +* **Code Interpreter**: Supported; request/response events pass through to compatible backends without schema rejection. +* **Web Search**: Passed through gracefully to backends supporting web search; local models return informative responses if search capability is not enabled on the underlying model. 2. The responses are returned successfully with acceptable latency. ### How to Run diff --git a/litellm/config.yaml b/litellm/config.yaml index f13f481..9ee311b 100644 --- a/litellm/config.yaml +++ b/litellm/config.yaml @@ -35,6 +35,10 @@ litellm_settings: - gpt-4o-mini-transcribe - gpt-4o-mini-tts - tts-1 + - local-qwen-3.6 + - local-qwen-3.6-hass + - gpt-4o-mini + - gpt-4o fallbacks: - agent-simple-core: - agent-medium-core @@ -100,6 +104,14 @@ model_list: preserve_thinking: true request_timeout: 600 model_name: local-qwen-3.6 + model_info: + supports_vision: true + supports_reasoning: true + supports_function_calling: true + mode: chat + max_tokens: 524288 + max_input_tokens: 524288 + is_public_model_group: true - litellm_params: api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER api_key: local-token @@ -109,6 +121,48 @@ model_list: enable_thinking: false request_timeout: 600 model_name: local-qwen-3.6-hass + model_info: + supports_vision: true + supports_reasoning: true + supports_function_calling: true + mode: chat + max_tokens: 524288 + max_input_tokens: 524288 + is_public_model_group: true +- litellm_params: + api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER + api_key: local-token + model: openai/local-qwen-3.6-hass + extra_body: + chat_template_kwargs: + enable_thinking: false + request_timeout: 600 + model_name: gpt-4o-mini + model_info: + supports_vision: true + supports_reasoning: true + supports_function_calling: true + mode: chat + max_tokens: 524288 + max_input_tokens: 524288 + is_public_model_group: true +- litellm_params: + api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER + api_key: local-token + model: openai/local-qwen-3.6-hass + extra_body: + chat_template_kwargs: + enable_thinking: false + request_timeout: 600 + model_name: gpt-4o + model_info: + supports_vision: true + supports_reasoning: true + supports_function_calling: true + mode: chat + max_tokens: 524288 + max_input_tokens: 524288 + is_public_model_group: true - litellm_params: api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER api_key: local-token diff --git a/router/main.py b/router/main.py index 2487b4b..ed6baa7 100644 --- a/router/main.py +++ b/router/main.py @@ -1971,6 +1971,131 @@ async def proxy_models(): raise HTTPException(status_code=502, detail="Model proxy failed") +@app.api_route("/v1/responses", methods=["POST"]) +@app.api_route("/responses", methods=["POST"]) +async def responses_api(request: Request): + """Handle incoming OpenAI Responses API requests (/v1/responses and /responses). + + Proxies requests to LiteLLM's /v1/responses endpoint, performing triage classification + when an auto model (e.g. llm-routing-auto-free) is requested, while supporting model aliases + (such as gpt-4o-mini, local-qwen-3.6-hass) and tool/streaming executions. + """ + try: + body = await request.json() + except Exception: + raise HTTPException(status_code=400, detail="Invalid JSON payload") + + await sync_cooldowns_from_valkey() + + client_model = body.get("model", "llm-routing-auto-free") + + AUTO_MODELS = { + "llm-routing-auto-free", + "llm-routing-auto-agy", + "llm-routing-auto-ollama", + "llm-routing-auto-agy-ollama", + } + + last_user_message = "" + input_field = body.get("input") + if isinstance(input_field, str): + last_user_message = input_field + elif isinstance(input_field, list): + for item in input_field: + if isinstance(item, str): + last_user_message += item + elif isinstance(item, dict): + if item.get("type") == "text": + last_user_message += item.get("text", "") + elif item.get("role") == "user": + content = item.get("content") or "" + if isinstance(content, str): + last_user_message += content + elif isinstance(content, list): + last_user_message += "".join( + b.get("text", "") + for b in content + if isinstance(b, dict) and b.get("type") == "text" + ) + + if not last_user_message: + instructions = body.get("instructions") + if isinstance(instructions, str): + last_user_message = instructions + + target_model = client_model + if client_model in AUTO_MODELS or client_model == "llm-routing-ollama": + bypass_cache = request.headers.get("x-bypass-cache") == "true" + ( + target_model, + triage_latency, + was_cache_hit, + raw_classification, + ) = await classify_request(last_user_message, bypass_cache=bypass_cache) + logger.info(f"Responses API Triage decision: Routing to -> '{target_model}'") + + body_to_send = body.copy() + body_to_send["model"] = target_model + + litellm_key = os.getenv("LITELLM_MASTER_KEY") + auth_header = request.headers.get("Authorization") + if not auth_header or not auth_header.startswith("Bearer "): + auth_header = f"Bearer {litellm_key}" + + headers = { + "Authorization": auth_header, + "Content-Type": request.headers.get("content-type", "application/json"), + } + + is_streaming = body_to_send.get("stream", False) + client = get_http_client() + url = f"{LITELLM_URL}/v1/responses" + + logger.info(f"Proxying Responses API request for model={target_model} to {url}") + + if is_streaming: + async def response_streamer(): + try: + async with client.stream( + "POST", + url, + json=body_to_send, + headers=headers, + timeout=600.0, + ) as resp: + async for chunk in resp.aiter_bytes(): + yield chunk + except Exception as stream_err: + logger.error(f"Error during Responses API streaming proxy: {stream_err}") + raise + + return StreamingResponse(response_streamer(), media_type="text/event-stream") + else: + try: + r = await client.post( + url, + json=body_to_send, + headers=headers, + timeout=600.0, + ) + response_headers = dict(r.headers) + for h in [ + "content-encoding", + "content-length", + "transfer-encoding", + "connection", + ]: + response_headers.pop(h, None) + return Response( + content=r.content, + status_code=r.status_code, + headers=response_headers, + ) + except Exception as e: + logger.error(f"Failed to proxy Responses API request: {e}") + raise HTTPException(status_code=502, detail="Responses proxy failed") + + @app.post("/v1/chat/completions") async def chat_completions(request: Request): """Handle incoming OpenAI-compatible chat completions requests. @@ -2025,6 +2150,10 @@ async def chat_completions(request: Request): "agent-reasoning-core", "agent-advanced-core", "llm-routing-agy", + "local-qwen-3.6", + "local-qwen-3.6-hass", + "gpt-4o-mini", + "gpt-4o", } AUTO_MODELS = { diff --git a/router/tests/test_responses_api.py b/router/tests/test_responses_api.py new file mode 100644 index 0000000..0f6c35b --- /dev/null +++ b/router/tests/test_responses_api.py @@ -0,0 +1,177 @@ +import json +import pytest +from unittest.mock import AsyncMock, MagicMock, patch +from fastapi import Response, HTTPException +from fastapi.responses import JSONResponse, StreamingResponse + +from router.main import responses_api + + +@pytest.mark.anyio +async def test_responses_api_direct_model_non_streaming(): + """Test POST /v1/responses with direct model (gpt-4o-mini) non-streaming.""" + mock_request = MagicMock() + mock_request.json = AsyncMock(return_value={ + "model": "gpt-4o-mini", + "input": "Hello from Home Assistant config flow!" + }) + mock_request.headers = {"content-type": "application/json", "Authorization": "Bearer test-token"} + + mock_lite_resp = MagicMock() + mock_lite_resp.status_code = 200 + mock_lite_resp.content = json.dumps({ + "id": "resp_test123", + "object": "response", + "model": "gpt-4o-mini", + "status": "completed", + "output": [ + { + "id": "msg_123", + "type": "message", + "role": "assistant", + "content": [{"type": "output_text", "text": "Hello there!"}], + "status": "completed" + } + ] + }).encode("utf-8") + mock_lite_resp.headers = {"content-type": "application/json"} + + mock_client = AsyncMock() + mock_client.post.return_value = mock_lite_resp + + with patch("router.main.get_http_client", return_value=mock_client), \ + patch("router.main.sync_cooldowns_from_valkey", new=AsyncMock()): + response = await responses_api(mock_request) + assert isinstance(response, Response) + assert response.status_code == 200 + data = json.loads(response.body) + assert data["id"] == "resp_test123" + assert data["model"] == "gpt-4o-mini" + assert data["output"][0]["content"][0]["text"] == "Hello there!" + + +@pytest.mark.anyio +async def test_responses_api_auto_model_triage(): + """Test POST /v1/responses with auto model (llm-routing-auto-free) triggers classifier.""" + mock_request = MagicMock() + mock_request.json = AsyncMock(return_value={ + "model": "llm-routing-auto-free", + "input": "Fix typo in variable name x = 1" + }) + mock_request.headers = {"content-type": "application/json"} + + mock_lite_resp = MagicMock() + mock_lite_resp.status_code = 200 + mock_lite_resp.content = json.dumps({ + "id": "resp_triage123", + "object": "response", + "model": "agent-simple-core", + "status": "completed", + "output": [] + }).encode("utf-8") + mock_lite_resp.headers = {"content-type": "application/json"} + + mock_client = AsyncMock() + mock_client.post.return_value = mock_lite_resp + + with patch("router.main.classify_request", new=AsyncMock(return_value=("agent-simple-core", 10.0, False, "simple"))), \ + patch("router.main.get_http_client", return_value=mock_client), \ + patch("router.main.sync_cooldowns_from_valkey", new=AsyncMock()): + response = await responses_api(mock_request) + assert isinstance(response, Response) + assert response.status_code == 200 + # Verify body sent to LiteLLM was updated with target_model + called_args, called_kwargs = mock_client.post.call_args + assert called_kwargs["json"]["model"] == "agent-simple-core" + + +@pytest.mark.anyio +async def test_responses_api_with_tools(): + """Test POST /v1/responses with Home Assistant style tool definitions.""" + mock_request = MagicMock() + mock_request.json = AsyncMock(return_value={ + "model": "local-qwen-3.6-hass", + "input": "Turn on living room light", + "tools": [ + { + "type": "function", + "name": "HassTurnOn", + "parameters": {"type": "object", "properties": {"domain": {"type": "string"}}} + }, + {"type": "code_interpreter"}, + {"type": "web_search"} + ] + }) + mock_request.headers = {"content-type": "application/json"} + + mock_lite_resp = MagicMock() + mock_lite_resp.status_code = 200 + mock_lite_resp.content = json.dumps({ + "id": "resp_tool123", + "object": "response", + "model": "local-qwen-3.6-hass", + "status": "completed", + "output": [ + { + "id": "fc_123", + "type": "function_call", + "name": "HassTurnOn", + "arguments": '{"domain": "light"}', + "status": "completed" + } + ] + }).encode("utf-8") + mock_lite_resp.headers = {"content-type": "application/json"} + + mock_client = AsyncMock() + mock_client.post.return_value = mock_lite_resp + + with patch("router.main.get_http_client", return_value=mock_client), \ + patch("router.main.sync_cooldowns_from_valkey", new=AsyncMock()): + response = await responses_api(mock_request) + assert isinstance(response, Response) + assert response.status_code == 200 + data = json.loads(response.body) + assert data["output"][0]["type"] == "function_call" + assert data["output"][0]["name"] == "HassTurnOn" + + +@pytest.mark.anyio +async def test_responses_api_invalid_json(): + """Test POST /v1/responses with malformed JSON body.""" + mock_request = MagicMock() + mock_request.json = AsyncMock(side_effect=ValueError("Invalid JSON")) + + with pytest.raises(HTTPException) as exc_info: + await responses_api(mock_request) + assert exc_info.value.status_code == 400 + assert "Invalid JSON payload" in exc_info.value.detail + + +@pytest.mark.anyio +async def test_responses_api_streaming(): + """Test POST /v1/responses with stream=True.""" + mock_request = MagicMock() + mock_request.json = AsyncMock(return_value={ + "model": "gpt-4o-mini", + "input": "Stream test", + "stream": True + }) + mock_request.headers = {"content-type": "application/json"} + + async def mock_aiter_bytes(): + yield b'data: {"type":"response.created"}\n\n' + yield b'data: [DONE]\n\n' + + mock_stream_ctx = MagicMock() + mock_stream_ctx.__aenter__ = AsyncMock(return_value=MagicMock(aiter_bytes=mock_aiter_bytes)) + mock_stream_ctx.__aexit__ = AsyncMock() + + mock_client = AsyncMock() + mock_client.stream.return_value = mock_stream_ctx + + with patch("router.main.get_http_client", return_value=mock_client), \ + patch("router.main.sync_cooldowns_from_valkey", new=AsyncMock()): + response = await responses_api(mock_request) + assert isinstance(response, StreamingResponse) + assert response.media_type == "text/event-stream"