[LHF-04] Naive character-based token estimation logic

[sheepdestroyer]

Overview

The estimate_prompt_tokens function uses a simplistic character-count heuristic (len(content) // 4) to estimate token counts. While this provides a rough approximation for English prose, it fails significantly for code (heavy punctuation), multi-byte Unicode strings (CJK, emojis), and whitespace-padded content, compromising context-window clamping accuracy and routing metrics.

Location

File: router/main.py — estimate_prompt_tokens(), Lines 68–85

Current Code

def estimate_prompt_tokens(body: dict) -> int:
    """Estimate prompt tokens by counting characters in message contents (1 token ~= 4 chars)
    to avoid inflating metrics with large tool/schema declarations.
    """
    tokens = 0
    for msg in body.get("messages", []):
        if not isinstance(msg, dict):
            continue
        content = msg.get("content") or ""
        if isinstance(content, str):
            tokens += len(content) // 4
        elif isinstance(content, list):
            for block in content:
                if isinstance(block, dict) and block.get("type") == "text":
                    tokens += len(block.get("text") or "") // 4
    tokens += 50
    return max(1, tokens)

Problems

Content Type	Actual Tokens	len//4 Estimate	Error
English prose	~100	~100	✅ ~0%
Python code	~100	~60	❌ -40% (punctuation is individually tokenized)
CJK text (Chinese/Japanese)	~50	~75	❌ +50% (multi-byte chars inflate len())
Whitespace-padded JSON	~100	~200	❌ +100% (spaces count as chars but not tokens)
System emojis	~10	~40	❌ +300% (4-byte chars)

Impact

• Severity: 🟡 Medium (Accuracy)
• Under-estimation for code prompts → context window overflows not caught by the router
• Over-estimation for CJK → unnecessarily routing to larger models
• Metrics dashboard shows inaccurate token counts, misleading cost analysis

Recommendations

Option A: Word-count baseline (Zero dependency, better accuracy)

A word-count heuristic provides tighter tracking than character division for mixed content:

def estimate_prompt_tokens(body: dict) -> int:
    """Estimate prompt tokens using word-count heuristic.
    Average English: ~1.3 tokens/word. Code: ~1.5 tokens/word.
    We use 1.3 as a conservative multiplier.
    """
    tokens = 0
    for msg in body.get("messages", []):
        if not isinstance(msg, dict):
            continue
        content = msg.get("content") or ""
        if isinstance(content, str):
            words = len(content.split())
            tokens += int(words * 1.3)
        elif isinstance(content, list):
            for block in content:
                if isinstance(block, dict) and block.get("type") == "text":
                    text = block.get("text") or ""
                    tokens += int(len(text.split()) * 1.3)
    tokens += 50  # metadata overhead
    return max(1, tokens)

Option B: Hybrid heuristic (Best accuracy without dependencies)

Combine character and word counting for a more robust estimate:

def estimate_prompt_tokens(body: dict) -> int:
    tokens = 0
    for msg in body.get("messages", []):
        if not isinstance(msg, dict):
            continue
        content = msg.get("content") or ""
        if isinstance(content, str):
            # Average of char-based and word-based estimates
            char_est = len(content) // 4
            word_est = int(len(content.split()) * 1.3)
            tokens += (char_est + word_est) // 2
        elif isinstance(content, list):
            for block in content:
                if isinstance(block, dict) and block.get("type") == "text":
                    text = block.get("text") or ""
                    char_est = len(text) // 4
                    word_est = int(len(text.split()) * 1.3)
                    tokens += (char_est + word_est) // 2
    tokens += 50
    return max(1, tokens)

Option C: tiktoken (Most accurate, adds dependency)

For production-grade accuracy, use OpenAI's tiktoken library:

import tiktoken
_enc = tiktoken.get_encoding("cl100k_base")  # GPT-4 / Claude compatible

def estimate_prompt_tokens(body: dict) -> int:
    tokens = 0
    for msg in body.get("messages", []):
        content = msg.get("content") or ""
        if isinstance(content, str):
            tokens += len(_enc.encode(content))
    tokens += 50
    return max(1, tokens)

Note: tiktoken adds a dependency but provides exact tokenizer-aligned counts. Consider the tradeoff vs. zero-dependency heuristics.

Acceptance Criteria

• Token estimation accuracy is within ±20% for code, prose, and mixed content
• Multi-byte Unicode inputs do not produce wildly inflated estimates
• The function remains fast (sub-millisecond for typical message arrays)
• No new hard dependencies unless explicitly approved
• Existing tests pass and dashboard metrics remain consistent

[google-labs-jules]

Jules is on it. When finished, you will see another comment and be able to review a PR.

[google-labs-jules]

Ready for a review! A PR has been created.