Type: Feature proposal (design / discussion — no code)
What
Replace the OpenRouter-style /models catalog with the Router Dashboard, whose centerpiece is the router itself, framed as an eval of routing decisions.
Why
The current /models page is a catalog. It advertises breadth (shared with OpenRouter) and conveys nothing about the router — the actual product. It also uses $/token, the wrong unit for agentic cost-optimization (cost-per-run, cache economics, quality floor, reliability dominate).
Thesis
Artificial Analysis is a map; BitRouter is the driver and the road. Differentiation follows from sitting inside the data path:
- Unit is the route, not the model (aliases like
code/balanced).
- Numbers are yours — profile + baseline drive every figure.
- Numbers are live — p50, uptime, cheapest-healthy provider, cache-hit.
- AA's cache-routing disclaimer is our headline — we optimize the variance, we don't disclaim it.
- We don't fight AA on neutrality; we win on checkability (auditable vs. the user's own bill) and cite AA for the quality axis.
Core mechanic — the regret triangle
Evaluate the router, not models: naive baseline → BitRouter route → oracle.
- baseline → router = savings (headline)
- router → oracle = honesty (showing where we lose makes the wins believable)
Visualized: AA's cost-vs-quality scatter becomes a vector with an arrow per task-class. Paired with a live decision trace (reusing the mono terminal) as proof-of-life.
Data sources (decided)
- Synthetic task suite → the eval (needs known-good ground truth; reproducibility beats the neutrality problem).
- Live telemetry → the trace + personalized savings (authentic, but no per-task quality truth).
- Catalog demoted to an SEO/coverage substrate, re-columned to cost-per-task.
Open questions for the implementation plan
- Does the synthetic suite exist? If not, v1 leads with the live trace; regret triangle becomes roadmap.
- Are aliases a mature product? The route-as-unit spine depends on it (else fall back to route≈model for v1).
- AA licensing/attribution; telemetry privacy/aggregation story.
Full design spec drafted at docs/superpowers/specs/2026-06-22-router-eval-dashboard-design.md (held locally; not merged).
🤖 Generated with Claude Code
Type: Feature proposal (design / discussion — no code)
What
Replace the OpenRouter-style
/modelscatalog with the Router Dashboard, whose centerpiece is the router itself, framed as an eval of routing decisions.Why
The current
/modelspage is a catalog. It advertises breadth (shared with OpenRouter) and conveys nothing about the router — the actual product. It also uses$/token, the wrong unit for agentic cost-optimization (cost-per-run, cache economics, quality floor, reliability dominate).Thesis
Artificial Analysis is a map; BitRouter is the driver and the road. Differentiation follows from sitting inside the data path:
code/balanced).Core mechanic — the regret triangle
Evaluate the router, not models:
naive baseline → BitRouter route → oracle.Visualized: AA's cost-vs-quality scatter becomes a vector with an arrow per task-class. Paired with a live decision trace (reusing the mono terminal) as proof-of-life.
Data sources (decided)
Open questions for the implementation plan
Full design spec drafted at
docs/superpowers/specs/2026-06-22-router-eval-dashboard-design.md(held locally; not merged).🤖 Generated with Claude Code