Skip to content

Add bounded cache-aware routing for Hugging Face - #599

Open
KirschBluteX wants to merge 3 commits into
huggingface:mainfrom
KirschBluteX:codex/issue-572-hf-cache-routing
Open

Add bounded cache-aware routing for Hugging Face#599
KirschBluteX wants to merge 3 commits into
huggingface:mainfrom
KirschBluteX:codex/issue-572-hf-cache-routing

Conversation

@KirschBluteX

@KirschBluteX KirschBluteX commented Aug 17, 2026

Copy link
Copy Markdown

Summary

The built-in Hugging Face provider can keep sending large append-only prompts through a route that reports no cache reuse. This adds bounded, session-scoped adaptation while keeping tau_agent provider-neutral.

  • Distinguish absent cache telemetry, reported zero, and positive reuse in the existing usage path.
  • Ignore cold, short, and non-comparable requests; warm an automatic route once, then evaluate two comparable misses. Candidate discovery is limited to live conversational providers, at most three routes, and at most nine eligible requests.
  • Persist routing evidence, attempted routes, evaluation mode, and every automatic transition through the existing session index and CustomEntry mechanisms. Explicit configured or extension-selected routes remain locked; /route automatic resets evaluation.
  • Surface every automatic route change through typed print, transcript, JSON, and TUI events.
  • Retain the current usable route when candidate discovery, route switching, or persistence cannot complete.

The policy observes provider-reported cache telemetry only and remains deliberately bounded. It does not change explicit user route precedence or add Hugging Face behavior to the reusable agent harness.

Closes #572

Validation

  • uv run pytest -q: 1547 passed, 3 platform-specific tests skipped
  • Focused cache-routing suite: 18 passed
  • uv run ruff check .
  • uv run ruff format --check .
  • uv run mypy
  • Hugo documentation build and Pagefind indexing
  • git diff --check origin/main...HEAD

No dependencies, versions, or generated artifacts changed.

@KirschBluteX
KirschBluteX force-pushed the codex/issue-572-hf-cache-routing branch from d23253f to 5415e02 Compare August 18, 2026 00:10
@KirschBluteX KirschBluteX changed the title Add bounded cache-aware Hugging Face routing Add bounded cache-aware routing for Hugging Face Aug 18, 2026
@KirschBluteX
KirschBluteX marked this pull request as ready for review August 18, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adapt Hugging Face routing when a backend lacks effective prompt caching

1 participant