Ollama routing broken: classifier llama-server (port 8080) not deployed — all requests fall through to ollama-deepseek-v4-pro

[sheepdestroyer]

Description

verify_ollama_routing.py fails on the first test case in the dev environment: a simple prompt ("Write a hello world in Python") sent to llm-routing-auto-ollama returns model ollama-deepseek-v4-pro instead of the expected agent-simple-core.

Run: python scripts/verification/verify_ollama_routing.py

Actual output:

--- 1. Testing llm-routing-auto-ollama ---
Request: model=llm-routing-auto-ollama, prompt='Write a hello world in Python...'
Response: model=ollama-deepseek-v4-pro, text='...'
 FAILURE: Expected model 'agent-simple-core', but got 'ollama-deepseek-v4-pro'

Root Cause

The local classifier llama-server (port 8080) is not deployed. No container, pod, or systemd unit provides the http://127.0.0.1:8080/v1 endpoint that the triage router's classifier depends on.

Failure cascade

Step	Code	What happens
1	classify_request()	Sends prompt to http://127.0.0.1:8080/v1/chat/completions
2	Connection refused	httpx.ConnectError — port 8080 has no listener
3	router/main.py:1088-1091	except Exception → returns "agent-advanced-core" with reason "advanced (exception)"
4	router/main.py:1910-1917	agent-advanced-core is in the Ollama gating list → should_try_ollama = True
5	router/main.py:2205-2207	agent-advanced-core → maps to "ollama-deepseek-v4-pro"
6	execute_proxy()	Proxies to LiteLLM as ollama-deepseek-v4-pro

Result: ALL requests, regardless of prompt complexity, get routed to the most expensive Ollama model.

Current state

$ curl --max-time 3 http://127.0.0.1:8080/health
curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused

$ podman ps --filter "name=llama" --format "{{.Names}}"
(no output — no classifier container exists)

$ podman pod inspect dev-router-pod --format '{{.Containers}}'
map[dev-router-pod-valkey-cache ... dev-router-pod-minio-s3 ...]
(no classifier/llama-server container in the pod)

Configuration reference

router/config.yaml:

router:
  router_model:
    api_base: "http://127.0.0.1:8080/v1"
    model: "gemma4-26a4b-routing"

data/router-rendered/config.yaml confirms the rendered config matches.

Affected components

1. verify_ollama_routing.py — all 6 test cases broken

All test cases fail because every prompt gets classified as agent-advanced-core and routed to ollama-deepseek-v4-pro:

Test	Prompt	Expected model	Actual (broken)
1	"Write a hello world in Python" (simple)	agent-simple-core	ollama-deepseek-v4-pro
2	"Implement a custom memory-efficient Trie in C++" (complex)	ollama-deepseek-v4-flash	ollama-deepseek-v4-pro
3	"Design a distributed pub/sub system..." (reasoning)	ollama-deepseek-v4-pro	ollama-deepseek-v4-pro
4	"Write a hello world in Python" (simple, ollama)	ollama-deepseek-v4-flash	ollama-deepseek-v4-pro
5	"Implement a custom memory-efficient Trie in C++" (complex, ollama)	ollama-deepseek-v4-flash	ollama-deepseek-v4-pro
6	"Design a distributed pub/sub system..." (reasoning, ollama)	ollama-deepseek-v4-pro	ollama-deepseek-v4-pro

Tests 3 and 6 coincidentally pass (correct model for wrong reason).

2. All llm-routing-auto-* traffic affected

Any client using llm-routing-auto-ollama or llm-routing-auto gets degraded routing:

• Simple/medium/complex prompts → the fallback agent-advanced-core classification
• Results in all traffic going to the most expensive/highest-latency model
• Token costs inflated for simple queries that should use agent-simple-core or agent-medium-core

3. Metrics pollution

The metrics endpoint shows 80 advanced_requests_total vs 45 simple_requests_total — the advanced count is inflated by classification failures counted as legitimate advanced-tier requests.

Secondary concern: test expectation validity

Even when the classifier is working, verify_ollama_routing.py test #1 expects the response model to be "agent-simple-core". However, the routing logic at router/main.py:1910-1917 explicitly gates Ollama to only agent-complex-core, agent-reasoning-core, and agent-advanced-core. Simple prompts classified as agent-simple-core skip the Ollama path and go through OpenRouter — the returned model name would be whatever OpenRouter/LiteLLM reports, not "agent-simple-core".

This test expectation may need to be re-examined once the classifier is operational.

Environment

• Dev: dev.vendeuvre.lan, pod dev-router-pod
• Router version: latest dev image
• Classifier config: gemma4-26a4b-routing on http://127.0.0.1:8080/v1

Requirements

1. Deploy a llama-server classifier container serving gemma4-26a4b-routing on port 8080
2. Add it to pod.yaml as part of the dev-router-pod (or as a separate service)
3. Ensure it auto-starts on boot (systemd unit or pod auto-start)
4. Update SERVICES.md if the deployment topology changes from the documented qwen-2b-routing
5. Re-run verify_ollama_routing.py after deployment and validate all test expectations
6. Consider health-check monitoring for the classifier endpoint to alert on future outages

[sheepdestroyer]

There were lots of changes since,
@jules Investigate & Report

[google-labs-jules]

Jules is on it. When finished, you will see another comment and be able to review a PR.

[google-labs-jules]

Ready for a review! A PR has been created.