Add bounded cache-aware routing for Hugging Face

dev-notes/hugging-face-sticky-routing.md

@@ -1,15 +1,17 @@
-# Hugging Face session-scoped provider routing
+# Hugging Face session routing and cache adaptation
 
-Issue: <https://github.com/huggingface/tau/issues/559>
+Issues: <https://github.com/huggingface/tau/issues/559>,
+<https://github.com/huggingface/tau/issues/572>
 
 ## What changed
 
 Tau pins a logical model from its built-in `huggingface` provider to one
 explicit Hugging Face Inference Provider. Users can configure a per-model
 `inference_providers` map in `~/.tau/providers.json`. Without a preference, the
 first request uses automatic routing; after it succeeds, Tau reads the
-`x-inference-provider` response header and rebuilds the runtime with that explicit
-suffix. The session index records the route for resume.
+normalized `response_provider` value and rebuilds the runtime with that explicit
+suffix. The session index records the route and whether it came from an explicit
+choice or automatic routing.
 
 The OpenAI-compatible runtime now supports an internal logical-to-wire model alias.
 For example, the harness and persisted assistant messages continue to use
@@ -18,11 +20,22 @@ For example, the harness and persisted assistant messages continue to use
 catalog and keeps context windows, capabilities, pricing, thinking controls, and
 model selection keyed by the logical model.
 
-`/session` reports the pin. Route changes belong to the external Hugging Face
-extension, which uses the public extension API. Switching logical models selects the new
-model's configured pin or returns to automatic Hugging Face routing when none
-exists. Existing records without the optional field remain compatible and pin
-after their next successful automatic response.
+Automatic routes now receive a bounded cache check. Requests below 4,096 prompt
+tokens are ignored. The first eligible request on a route is a cold warm-up; two
+later append-only requests with absent cache telemetry or an explicit zero cache
+read trigger candidate discovery. A positive cache read retains the route. Tau
+tries at most three routes across at most nine eligible requests, and keeps the
+current route if discovery or all remaining candidates fail.
+Candidate discovery filters the model API mapping to validated `status: live`,
+`task: conversational` suffixes, sorts them lexicographically, and skips routes
+already attempted in the session.
+
+`/session` reports the pin and evaluation phase. Every automatic route change is
+a typed coding-session event consumed by print and TUI frontends. Manual route
+selection remains available through the external Hugging Face extension. An
+explicit route locks evaluation; `/route automatic` resets it. Historical
+records with a pin but no source field are treated as explicit, so an upgrade
+cannot silently take over a user's existing route.
 
 ## Why it exists
 
@@ -32,30 +45,38 @@ full misses only seconds after large cache reads, followed immediately by anothe
 large hit. That pattern is consistent with requests moving between provider or
 worker cache domains rather than normal TTL expiry.
 
+In the motivating GLM-5.2 comparison, `deepinfra` reported roughly 99% reuse
+after its cold request. Five append-only requests through `scaleway` returned
+null cache details for an approximately 12.5k-token prefix and incurred about
+$0.13 in Hugging Face billing, consistent with processing the prompt as fresh.
+That is evidence about reported reuse and billing, not proof that a backend has
+no internal cache.
+
 An explicit `:<provider>` suffix narrows one source of routing changes. It does
 not guarantee a cache hit: the selected provider can still evict entries,
 load-balance across workers, or expire them.
 
 ## Architecture
 
-Provider preferences, session metadata, and model-route selection remain in
-`tau_coding`. The reusable `tau_agent` harness receives the logical model and has
-no Hugging Face-specific behavior. `tau_ai` only gains a provider-neutral
-`model_aliases` transport option, used to put a different model ID in the wire
-payload while preserving the logical model in normalized events.
+Provider preferences, adaptive policy, session metadata, and model-route
+selection remain in `tau_coding`. State snapshots use the existing session
+`CustomEntry` mechanism, so evidence, unavailable candidates, and transitions
+resume with the active branch. The reusable `tau_agent` harness receives the
+logical model and has no Hugging Face-specific behavior. `tau_ai` and the
+provider-neutral `Usage` model expose only whether a cache-read counter was
+reported, which distinguishes absent telemetry from a reported zero.
 
 [Hugging Face's own Chat UI](https://github.com/huggingface/chat-ui/blob/main/src/lib/server/endpoints/openai/endpointOai.ts)
 consumes `x-inference-provider` from OpenAI-compatible responses, so Tau uses
 that header rather than guessing which mapping is fastest.
 The pin is committed only after a successful stream. Existing OpenAI-compatible
 retries keep the same wire model and stop retrying after streamed model output.
 
-The first version deliberately does not automatically fail over a stale explicit
-pin or send provider-specific cache-affinity fields. Safe failover also needs a
-user-visible reroute event plus durable retry/reroute telemetry; silently falling
-back would hide temporary cache-locality loss. Users can explicitly reset with
-the Hugging Face extension. Backing providers differ in accepted affinity fields, so no
-unknown field is enabled for the entire gateway.
+Tau deliberately does not fail over an explicit pin or send provider-specific
+cache-affinity fields. Errors, aborts, model changes, compaction boundaries,
+route mismatches, and non-append-only contexts do not add cache-failure evidence.
+Backing providers differ in accepted affinity fields, so no unknown field is
+enabled for the entire gateway.
 
 ## Configure and validate
 

src/tau_agent/messages.py

@@ -48,6 +48,7 @@ class Usage(WireModel):
     input: int = 0
     output: int = 0
     cache_read: int = 0
+    cache_read_reported: bool | None = None
     cache_write: int = 0
     cache_write_1h: int | None = None
     reasoning: int | None = None

src/tau_ai/openai_compatible.py

@@ -1286,6 +1286,7 @@ def _parse_chunk_usage(raw: Mapping[str, Any]) -> Usage:
     # 0 does not fall through.
     if cached_tokens is None:
         cached_tokens = _int_or_none(raw.get("prompt_cache_hit_tokens"))
+    cache_read_reported = cached_tokens is not None
     cache_read = cached_tokens or 0
     fresh_input = max(0, prompt_tokens - cache_read - cache_write)
     output = _int_or_zero(raw.get("completion_tokens"))
@@ -1297,6 +1298,7 @@ def _parse_chunk_usage(raw: Mapping[str, Any]) -> Usage:
         input=fresh_input,
         output=output,
         cache_read=cache_read,
+        cache_read_reported=cache_read_reported,
         cache_write=cache_write,
         reasoning=reasoning,
         total_tokens=fresh_input + output + cache_read + cache_write,
@@ -1322,6 +1324,11 @@ def _usage_from_responses_event(chunk: Mapping[str, Any]) -> Usage | None:
         if isinstance(input_details, Mapping)
         else 0
     )
+    cache_read_reported = (
+        _int_or_none(input_details.get("cached_tokens")) is not None
+        if isinstance(input_details, Mapping)
+        else False
+    )
     cache_write = (
         _int_or_zero(input_details.get("cache_write_tokens"))
         if isinstance(input_details, Mapping)
@@ -1339,6 +1346,7 @@ def _usage_from_responses_event(chunk: Mapping[str, Any]) -> Usage | None:
         input=max(0, _int_or_zero(raw.get("input_tokens")) - cache_read - cache_write),
         output=_int_or_zero(raw.get("output_tokens")),
         cache_read=cache_read,
+        cache_read_reported=cache_read_reported,
         cache_write=cache_write,
         reasoning=reasoning,
         total_tokens=_int_or_zero(raw.get("total_tokens")),

src/tau_coding/cli.py

@@ -54,7 +54,12 @@
     export_session_artifact,
     normalize_export_format,
 )
-from tau_coding.session_manager import CodingSessionRecord, SessionManager, validate_session_id
+from tau_coding.session_manager import (
+    CodingSessionRecord,
+    HuggingFaceRouteMode,
+    SessionManager,
+    validate_session_id,
+)
 from tau_coding.shell_config import load_shell_settings
 from tau_coding.tui import run_tui_app
 from tau_coding.update_check import (
@@ -912,12 +917,15 @@ async def run_openai_print_mode(
         provider_name=provider_name if explicit_selection else record.provider_name,
         model=model if explicit_selection else record.model,
     )
-    inference_provider = (
-        record.inference_provider
-        if resume_session_id is not None
+    uses_resumed_route = (
+        resume_session_id is not None
         and record.provider_name == "huggingface"
         and selection.provider.name == "huggingface"
         and record.model == selection.model
+    )
+    inference_provider = (
+        record.inference_provider
+        if uses_resumed_route
         else selection.provider.inference_providers.get(selection.model)
         if isinstance(selection.provider, OpenAICompatibleProviderConfig)
         and selection.provider.name == "huggingface"
@@ -941,6 +949,9 @@ async def run_openai_print_mode(
             session_manager=manager,
             provider_name=selection.provider.name,
             inference_provider=inference_provider,
+            inference_provider_mode=(
+                record.inference_provider_mode if uses_resumed_route else None
+            ),
             provider_settings=settings,
             runtime_provider_config=selection.provider,
             shell_command_prefix=shell_settings.shell_command_prefix,
@@ -1023,6 +1034,7 @@ async def run_print_mode(
     session_manager: SessionManager | None = None,
     provider_name: str = DEFAULT_PROVIDER_NAME,
     inference_provider: str | None = None,
+    inference_provider_mode: HuggingFaceRouteMode | None = None,
     provider_settings: ProviderSettings | None = None,
     runtime_provider_config: ProviderConfig | None = None,
     shell_command_prefix: str | None = None,
@@ -1051,6 +1063,7 @@ async def run_print_mode(
             session_manager=session_manager,
             provider_name=provider_name,
             inference_provider=inference_provider,
+            inference_provider_mode=inference_provider_mode,
             provider_settings=provider_settings,
             runtime_provider_config=runtime_provider_config,
             shell_command_prefix=shell_command_prefix,

src/tau_coding/commands.py

@@ -436,6 +436,9 @@ def _status_command(context: CommandContext) -> CommandResult:
     if session.provider_name == "huggingface":
         route = getattr(session, "inference_provider", None) or "automatic"
         lines.append(f"Hugging Face inference provider: {route}")
+        routing_status = getattr(session, "huggingface_routing_status", None)
+        if routing_status:
+            lines.append(f"Hugging Face cache routing: {routing_status}")
     context_window_source = getattr(session, "context_window_source", None)
     if context_window_source:
         lines.append(f"Context window source: {context_window_source}")

src/tau_coding/events.py

@@ -74,6 +74,22 @@ class AutoRetryEndEvent(WireModel):
     final_error: str | None = Field(None)
 
 
+class HuggingFaceRouteEvent(WireModel):
+    type: Literal["huggingface_route"] = "huggingface_route"
+    status: Literal["changed", "exhausted"]
+    previous_route: str | None = None
+    route: str
+    reason: str
+
+    @property
+    def display_text(self) -> str:
+        """Return the concise route notice shown by human frontends."""
+        if self.status == "changed":
+            previous = self.previous_route or "automatic"
+            return f"Hugging Face route changed: {previous} -> {self.route} ({self.reason})"
+        return f"Hugging Face route evaluation stopped on {self.route}: {self.reason}"
+
+
 type SessionOwnEvent = Annotated[
     SessionAgentEndEvent
     | AgentSettledEvent
@@ -84,7 +100,8 @@ class AutoRetryEndEvent(WireModel):
     | SessionInfoChangedEvent
     | ThinkingLevelChangedEvent
     | AutoRetryStartEvent
-    | AutoRetryEndEvent,
+    | AutoRetryEndEvent
+    | HuggingFaceRouteEvent,
     Field(discriminator="type"),
 ]
 type CodingSessionEvent = AgentEvent | SessionOwnEvent