PR 204 Review Fixes

.agents/AGENTS.md

@@ -18,3 +18,11 @@ When working on this project, always refer to the dedicated **NotebookLM Compani
 Use the `notebooklm` MCP tools to search or ask questions about this codebase and stack:
 - Run `notebook_ask` with `notebook_id: "llm-triage-gateway"` to ground your reasoning or implementation plans.
 - If you need session continuation, remember to reuse the `session_id` returned by previous queries.
+
+## Git Rebase & Conflict Resolution Policy
+To prevent directory reorganization regressions, outdated file restorations, or security credential overrides during merge conflict resolution, all automated agents must strictly follow these rules:
+
+1. **Rebase Over Merge**: Always fetch and rebase the topic/feature branch onto the latest `master` base branch (using `git rebase origin/master`) instead of performing `git merge`.
+2. **Directory Rename Safety**: If Git reports conflicts related to moved directories or files, do not manually stage deletions of tracked files from moved directories (e.g., under the old `tests/` or `scripts/` paths) or re-create files at the root level. Resolve conflicts by directing all changes and file operations to the newly refactored paths.
+3. **Verify Security Credentials**: Never accept resolutions that overwrite configuration files (`pod.yaml`, `start-stack.sh`) with hardcoded default passwords. Ensure placeholder-based configurations are preserved.
+4. **Enforce Test Suite Count**: Run the full unit test suite (`pytest`) after conflict resolution. Verify that the total number of passing tests is equal to or greater than before the resolution.

.agents/branch_audit_plan.md

@@ -0,0 +1,56 @@
+# Git Merges & Regressions Audit Report
+
+## Problem Overview
+During the merging of concurrent pull requests, automated conflict-resolution bots (`jules[bot]`) silently discarded or reverted several code blocks and logic optimizations. 
+
+To ensure complete stability, we performed a systematic, automated three-dot diffing audit (`git diff CURRENT_BRANCH...TARGET_BRANCH`) comparing our current workspace against every active and merged remote branch in the repository.
+
+---
+
+## Branch-by-Branch Audit Checklist
+
+| Branch Name / PR | Status | Notes / Discrepancies Checked |
+| :--- | :---: | :--- |
+| `origin/cleanup/remove-unused-get-live-gemini-oauth-token` (PR #195) | **CLEAN** | All changes present in current codebase. |
+| `origin/perf-optimize-gemini-oauth-token-7488735575783948165` (PR #158) | **CLEAN** | All changes present in current codebase. |
+| `origin/refactor-record-tool-usage-17779903941173107649` (PR #157) | **CLEAN** | All changes present in current codebase. |
+| `origin/chore/get-pr-status-tests-18330387033569416831` (PR #193) | **CLEAN** | All changes present in current codebase. |
+| `origin/chore/perf-async-annotations-4499495305587811104` (PR #172) | **CLEAN** | All changes present in current codebase. |
+| `origin/chore/add-gemini-oauth-tests-17524029845400706758` (PR #163) | **CLEAN** | All changes present in current codebase. |
+| `origin/dependabot/docker/berriai/litellm-v1.90.2` (PR #192) | **CLEAN** | Diff showed `v1.90.2` LiteLLM image version bump, which is already applied. |
+| `origin/security/fix-memory-mcp-category-injection-14795006529015952965` (PR #173) | **CLEAN** | All changes present in current codebase. |
+| `origin/test/add-read-annotations-sync-tests-12043640228306609857` (PR #168) | **CLEAN** | All changes present in current codebase. |
+| `origin/chore/test-parse-key-parameterized-14651890987031951450` (PR #169) | **CLEAN** | All changes present in current codebase. |
+| `origin/perf/async-quota-exhausted-4722057800691922723` (PR #156) | **CLEAN** | All changes present in current codebase. |
+| `origin/perf-benchmark-classifier-14678555128983342452` (PR #155) | **CLEAN** | All changes present in current codebase. |
+| `origin/chore/add-test-redis-from-url-12746122485451637611` (PR #174) | **CLEAN** | All changes present in current codebase. |
+| `origin/chore/test-is-memory-key-14668590644102848561` (PR #164) | **CLEAN** | All changes present in current codebase. |
+| `origin/⚡/optimize-agy-log-read-1519383298987985129` (PR #170) | **CLEAN** | All changes present in current codebase. |
+| `origin/chore/add-oauth-status-tests-15591368961301252072` (PR #166) | **CLEAN** | All changes present in current codebase. |
+| `origin/chore/test-get-goose-sessions-16558526271462662412` (PR #165) | **CLEAN** | All changes present in current codebase. |
+| `origin/fix-http-client-proliferation-11317864999360875839` | **CLEAN** | HTTP client unification is fully present in our workspace. |
+| `origin/fix/test-record-tool-usage-3580129709200353766` | **CLEAN** | Root-level test files were reorganized to `/tests`, fully preserved. |
+| `origin/fix/secrets-relocation` (PR #184) | **CLEAN** | Checked out placeholders and updated `start-stack.sh`/`pod.yaml` securely. |
+| `origin/perf/offload-aa-scores-sync-16551127323385849872` (PR #98) | **RESTORED** | **Regression Found**: The asynchronous scores loading optimization was lost on master. Re-implemented and verified. |
+
+---
+
+## Detailed Regression Analysis & Fix: Offload AA Scores Loading
+
+### The Regression
+* **File:** `router/main.py`
+* **Issue:** `_load_aa_scores()` (which does a blocking synchronous JSON disk read of `aa_scores.json`) was called directly at free-model/roster call sites, specifically `sync_adaptive_router_roster()` and `get_best_free_model()`, rather than inside `compute_free_model_score()`. The fix was to move `_load_aa_scores()` behind `await asyncio.to_thread(...)` and remove the direct synchronous call from the scoring path.
+* **Merged Fix Lost:** The merged PR #98 had offloaded this operation to a background thread pool using `await asyncio.to_thread(_load_aa_scores)` outside the loops in `sync_adaptive_router_roster` and `get_best_free_model`, but this change was completely discarded during automated conflict resolution.
+
+### The Resolution
+1. **`router/main.py`**:
+   * Restored `await asyncio.to_thread(_load_aa_scores)` inside `sync_adaptive_router_roster()` and `get_best_free_model()` if the cache is not yet loaded.
+   * Removed the blocking `_load_aa_scores()` call from inside `compute_free_model_score()`.
+2. **`tests/test_compute_free_model_score.py`**:
+   * Updated the test cases to explicitly call `router_main._load_aa_scores()` within the test setups to align with the asynchronous offloading.
+
+---
+
+## Verification Status
+* **Test Suite:** Ran `PYTHONPATH=router:. pytest`
+* **Result:** **181 tests passed** successfully (100% success rate).

.agents/jules_regression_analysis.md

@@ -0,0 +1,81 @@
+# Analysis of Bot Merge Conflict Regressions & Guardrails
+
+This document breaks down the root causes of the regressions introduced by the automated agent (`jules[bot]`) during recent merges, compares Git resolution strategies, and outlines prompt modifications and guardrails to prevent future occurrences.
+
+---
+
+## 1. What Went Wrong?
+
+The primary failure arose from a combination of **reorganization conflict logic** and **blind resolution choices**:
+
+### A. The Directory Rename / Delete Conflict Mismatch
+* **Context**: PR #181 reorganized the repository by moving files at the root (like `test_a2_verify.py` and `host_agy_daemon.py`) into nested directories (`tests/` and `scripts/`).
+* **The Bot's Branch State**: The bot was working on older branches (`perf-optimize-gemini-oauth-token`, `refactor-record-tool-usage`) whose base commits predated the reorganization.
+* **The Failure**: When merging `master` into these older feature branches, Git detected conflict types like `rename/delete` or `directory rename detection`. Because the bot resolved conflicts locally inside the old workspace structure, it staged the **deletion** of the new files inside `tests/` and `scripts/` and re-introduced older, obsolete root-level files.
+* **Result**: Merges from these branches back to `master` cleanly deleted the subfolder-bound tests and re-added old copies at the root.
+
+### B. Lack of Cross-Check Verification
+* The bot resolved conflicts file-by-file without compiling the whole project or validating the full test suite (`pytest`) on the combined codebase.
+* It accepted obsolete file versions wholesale (e.g., reverting the dynamic passwords in `pod.yaml` and `start-stack.sh` to hardcoded credentials) because it assumed conflict markers in one block did not impact security/logic blocks elsewhere.
+
+---
+
+## 2. Which Git Strategy Should Have Been Used?
+
+Instead of merging `master` directly into older feature branches (which creates complex, multi-directional merge graphs), the following strategies are far safer for LLM agents:
+
+### Option A: `git rebase master` (Recommended for Feature Branches)
+Rather than a merge commit, the agent should rebase the feature branch onto the latest `master` commit:
+```bash
+git checkout feature-branch
+git fetch origin
+git rebase origin/master
+```
+* **Why it works**: Rebase replays each feature branch commit one-by-one on top of the reorganized `master` commit. 
+* **Verify Relocated Files**: While Git's rename detection can assist with directory/file moves during rebase, it is not foolproof. The agent must manually verify relocated files (such as renamed test paths) after rebasing to ensure all changes landed in their correct new locations rather than being lost or orphaned.
+
+### Option B: Merge with explicit Merge Drivers
+If rebasing is not used, the agent must inspect renames before committing:
+```bash
+git diff --name-status origin/master...HEAD
+```
+This lists any deleted/added files to quickly verify that no folder moves were silently discarded.
+
+---
+
+## 3. Recommended Prompt Guardrails and System Rules
+
+To prevent automated agents from causing directory and code regressions, the following rules should be appended to the agent's instructions (e.g., in `.agents/AGENTS.md` or system guidelines):
+
+### Rule 1: Git Conflict Rebase Mandate
+> [!IMPORTANT]
+> When updating a feature branch with `master`/`main` changes, always prefer `git rebase` over `git merge`. If conflicts arise due to renamed directories, do not manually delete folders or re-add root counterparts. Ensure files are modified in their new paths.
+
+### Rule 2: Complete Test Suite Verification
+> [!WARNING]
+> Never push conflict resolutions without running the full test suite.
+> * If the workspace previously had $N$ passing tests, the resolved branch must have at least $N$ passing tests.
+> * Confirm all files staged for deletion or addition are intentional using `git diff --stat origin/master`.
+
+### Rule 3: File Integrity & Verification Checklist
+Add a post-conflict verification script step to the agent workflow:
+```bash
+# Verify no files were moved to root unexpectedly
+git status --porcelain
+```
+
+---
+
+## 4. Proposed Instructions / Prompts for Bot Agents
+
+When tasking an agent with conflict resolution or merging, use a structured prompt like this:
+
+```markdown
+You are resolving merge conflicts for the branch [branch_name].
+
+1. Run `git fetch origin` and `git rebase origin/master`.
+2. If conflicts arise, inspect whether any files were renamed/moved in master. Apply your changes to the renamed files in their new locations, rather than re-creating them at their old locations.
+3. Once rebased, run the entire test suite: `pytest`.
+4. Run `git diff --stat origin/master` and review every file addition/deletion. Verify that no tests or scripts are missing compared to master.
+5. If any test files or core logic sections are missing, checkout the missing files from master and apply changes cleanly.
+```

.github/workflows/test.yml

@@ -23,13 +23,13 @@ jobs:
           python-version: '3.11'
 
       - name: Install dependencies
-        run: pip install httpx==0.28.1 pytest pytest-asyncio anyio pyyaml fastapi "pydantic>=2.0,<3.0" uvicorn python-multipart asyncpg langfuse redis
+        run: pip install httpx==0.28.1 pytest pytest-asyncio anyio pyyaml fastapi "pydantic>=2.0,<3.0" uvicorn python-multipart asyncpg langfuse redis aiofiles
 
       - name: Run Unit Tests
-        run: CONFIG_PATH=router/config.yaml PYTHONPATH=.:router pytest --ignore=test_agy_behavior.py --ignore=test_agy_tiers.py
+        run: CONFIG_PATH=router/config.yaml PYTHONPATH=.:router pytest --ignore=tests/test_agy_behavior.py --ignore=tests/test_agy_tiers.py --ignore=tests/test_antigravity.py
 
       - name: Run Breaker Verification
-        run: python3 verify_breaker.py
+        run: PYTHONPATH=. python3 scripts/verification/verify_breaker.py
 
       - name: Run Integration Verification
-        run: python3 test_a2_verify.py
+        run: PYTHONPATH=.:router python3 tests/test_a2_verify.py

README.md

@@ -66,7 +66,7 @@ graph TD
     style QwenLocal fill:#f0f0f0,stroke:#999,stroke-width:1px;
 ```
 
-> **Version Pin**: LiteLLM Gateway runs `ghcr.io/berriai/litellm:v1.88.0`. See §3B for pinning policy.
+> **Version Pin**: LiteLLM Gateway runs `ghcr.io/berriai/litellm:v1.90.2`. See §3B for pinning policy.
 
 ---
 
@@ -269,7 +269,7 @@ Exposes the entry endpoint (`http://localhost:5000/v1`) and evaluates prompt com
 > Model capabilities, token limits, and costs are visible in LiteLLM's Model Hub Table at `http://localhost:4000/ui/?page=model-hub-table` (or port 4000 on the gateway host).
 
 ### B. LiteLLM Proxy Gateway (`litellm/config.yaml`)
-- **Version Pinning**: The LiteLLM gateway runs `ghcr.io/berriai/litellm:v1.88.0` (latest stable as of June 2026). The tag is explicitly pinned in `pod.yaml` — never use `:latest`. Check available tags with `skopeo list-tags docker://ghcr.io/berriai/litellm` before upgrading. ClickHouse runs `docker.io/clickhouse/clickhouse-server:26.5.1` (upgraded from 24.8, June 2026). Valkey Cache runs `docker.io/valkey/valkey:9.1.0-alpine` (upgraded from 8, June 2026).
+- **Version Pinning**: The LiteLLM gateway, ClickHouse, and Valkey Cache image tags are explicitly pinned in `pod.yaml` — never use `:latest`. Check available tags with `skopeo list-tags` or registry hubs before upgrading.
 Orchestrates routing fallback chains, Redis caching, and telemetry callbacks:
 - **`drop_params: true`**: Automatically strips unsupported arguments when transitioning to models that don't support them.
 - **Request Timeouts (`300s`)**: Provides ample padding to prevent connection aborts during dynamic RAM swapping operations on the local GPU `llama-server`.
@@ -790,7 +790,31 @@ The cooldown mechanism works as follows:
 
 For auto-routing modes, the Triage Router handles failures by silently falling back to the classified free tier cascade. For direct requests to `llm-routing-ollama`, the router returns `429` immediately during cooldown, allowing LiteLLM to skip this model group and cascade to `openrouter-auto`. The cooldown status is visible via the `/metrics` endpoint (`ollama_cooldown_active` and `ollama_cooldown_remaining_seconds` gauges).
 
-## 10. Performance Benchmarks\n\nThrough our local benchmarks, the following performance characteristics have been achieved:
+## 9d. Live Stack Tier Testing & Verification
+
+The repository includes an automated integration script to test the 5-tier intent routing pipeline on the live gateway stack:
+* **Location**: [test_reasoning_tiers.py](scripts/verification/test_reasoning_tiers.py)
+
+This script sends five sequential chat completion requests (from simple to advanced prompt complexities) to the gateway's `llm-routing-auto-free` auto-triage route, verifying that:
+1. The local classifier correctly categorizes and labels the prompt intent.
+2. The gateway successfully routes the prompt to the mapped LiteLLM fallback model group or provider.
+3. The responses are returned successfully with acceptable latency.
+
+### How to Run
+
+Ensure the container stack is deployed and healthy:
+```bash
+./start-stack.sh
+```
+
+Execute the verification script:
+```bash
+./scripts/verification/test_reasoning_tiers.py
+```
+
+## 10. Performance Benchmarks
+
+Through our local benchmarks, the following performance characteristics have been achieved:
 
 | Triage Evaluation Layer | Latency Footprint | Hardware Offload | Efficiency Ratio |
 | :--- | :---: | :---: | :---: |

litellm/entrypoint.py

@@ -98,6 +98,30 @@ def strptime(cls, date_str: str, fmt: str) -> original_datetime:
 datetime.datetime = RobustDatetime
 sys.stdout.flush()
 
+# Register both RobustDatetime AND the original datetime with Prisma's
+# singledispatch serializer. When entrypoint.py replaces datetime.datetime
+# with RobustDatetime before Prisma loads, Prisma's own
+# @serializer.register(datetime.datetime) ends up registering RobustDatetime.
+# But database drivers (psycopg2) return the *original* C-level datetime
+# instances, which no longer match. We must register both classes.
+try:
+    from prisma.builder import serializer
+except ImportError:
+    serializer = None
+
+if serializer is not None:
+    def _serialize_dt(dt):
+        """Serialize datetime to ISO8601 with timezone (UTC if naive)."""
+        if dt.tzinfo is None:
+            dt = dt.replace(tzinfo=timezone.utc)
+        else:
+            dt = dt.astimezone(timezone.utc)
+        return dt.isoformat().replace("+00:00", "Z")
+    serializer.register(original_datetime, _serialize_dt)
+    serializer.register(RobustDatetime, _serialize_dt)
+    print("🩹 Registered original_datetime + RobustDatetime with Prisma serializer")
+sys.stdout.flush()
+
 # Start LiteLLM Proxy
 import litellm
 from litellm.proxy.proxy_cli import run_server