Pick one in ~10 seconds, understand why, and check every number yourself. A decision tree over 160+ gateways, a handbook that explains how they actually work (and fail), and a reproducible cost benchmark plus dated evidence behind every claim — including what we exclude and why. CC0, no vendor money, no affiliate links.
💸 Built the hard way: I burned $788 on AI coding in a single day — one flagship model ate 78% of it, just because I'd defaulted everything to the priciest option. So I mapped the whole gateway landscape. → full story in Why this exists below · the full comparison
Languages: English · 简体中文
🧭 Pick one 🌍 State of the space 📖 Learn how they work 📊 Verify the numbers 🚀 Live site
- Which gateway should I use
- 🌍 State of the landscape (2026)
- ⚡ 10-second answers
- 📖 Learn: how AI gateways work
- 🔥 Top gateways (by stars)
- 💰 Cost-first: cheapest multi-model access
- 🔓 Self-hosted open source
- 🏢 Enterprise & compliance
- 🌐 First-party gateways (cloud & model vendors)
- 🇨🇳 China ecosystem
- 🤖 MCP & agent gateways
- 🔧 More by capability (cross-cutting)
- Quick comparison
- The requirements map
- How to choose safely
- 📊 Latest evaluations
- 📰 What's new
- 🚀 Recent releases (auto-updated)
- 📚 Essential reading
- Guides & comparisons
- FAQ
- Glossary
- Why this exists
- 🔌 Use the data — it's an API
- 🔗 Related lists
- Star history
⚡ Fast answer — one sane default per need (alternatives in each linked section):
| I need… | Start with | Drill into |
|---|---|---|
| Cheapest access to many models, zero ops | Vercel / Cloudflare (0% markup, your keys) · OpenRouter (~5.5% fee, one balance) | Cost-first |
| Zero markup on my own keys | Vercel / Cloudflare | Cost-first |
| Self-host, broadest features | LiteLLM | Self-hosted |
| Self-host, lowest overhead | Bifrost (Go) | Self-hosted |
| Route Claude Code / Codex to another model | LiteLLM / Bifrost (both measured 3/3) | Smart routing |
| Cut my Claude Code / Cursor bill — without a ban | claude-code-router (BYO-key, zero ban reports) | Coding-agent routers compared |
| Cut token spend without changing model or provider | headroom (compression proxy — attacks context size, not price) | Self-hosted |
| China models + team key billing | new-api | China ecosystem |
| Enterprise K8s + audit | Kong / Higress — audit logs are paid/hosted-tier on both (tiers) | Enterprise |
| Strongest compliance (HIPAA/FedRAMP) | Azure / Bedrock | First-party |
| Sensitive prompts — who sees / logs / trains on them? | Self-host or ZDR-default (Vercel / Requesty) | Who sees your prompts |
| Govern agents / MCP traffic | agentgateway | MCP & agents |
📋 The full decision tree — every branch, copy-pasteable
Do you want to self-host?
│
├─ NO — hosted, minimal ops
│ ├─ Cheapest access to many models ──────────▶ OpenRouter · Vercel AI Gateway (0% markup)
│ ├─ Free control plane over your own keys ───▶ Cloudflare AI Gateway
│ ├─ EU data residency matters ───────────────▶ Requesty · Eden AI · nexos.ai
│ └─ Already on one cloud ────────────────────▶ AWS Bedrock · Azure APIM · Vertex AI
│
└─ YES — self-hosted / open source
├─ Python stack, broadest features ─────────▶ LiteLLM
├─ Raw performance (Go/Rust/TS) ────────────▶ Bifrost (0.62 ms measured) · Portkey Gateway (2.65 ms; fidelity 1/3)
├─ Built-in evals + observability ──────────▶ LiteLLM · Bifrost · Helicone (maintenance mode)
├─ Multi-user keys + budgets + admin UI ────▶ LiteLLM · new-api (family / small team)
├─ CN models + CNY billing ─────────────────▶ new-api · one-api · GPT-Load
├─ Enterprise K8s, audit, guardrails ───────▶ Kong · Higress · APISIX · Envoy AI Gateway
└─ Governing AI agents & MCP traffic ───────▶ agentgateway · Lunar.dev
- ✅ Independent — no vendor money, no affiliate links, CC0. Unlike affiliate-driven relay "rankings," nobody pays to appear here.
- ✅ Reproducible, not asserted. Every cost cell is computed from open pricing data by a unit-tested script; stars refresh daily via CI.
- ✅ Honest about risk. We disclose CVEs, label archived/stale projects, and exclude gray-market relays — with the research to back it.
Why this matters: the same task can cost 100× more depending on the model behind your gateway. An AI gateway sits between your code and LLM providers — one endpoint, one key, many models — handling routing, failover, caching, rate limits, cost tracking and guardrails, so you change a
base_urlinstead of rewriting your app. Pick the gateway here, then the evaluation set shows which model to route to.
⭐ Found this useful? Star it — that's how the next engineer choosing a gateway finds it. CC0, no signup, no tracking, no vendor money.
The standalone AI gateway is being squeezed from both ends. Underneath, hyperscalers now ship routing and spend caps natively; above, the coding-agent wave produced a faster-growing class of gateway that most round-ups miss. The part that consolidated is the independent middle — a company whose entire product was the gateway.
| 2026 | What happened | State |
|---|---|---|
| TensorZero ⭐ 11.7k | VC-backed OSS gateway archived its repo (Jun 11) | Read-only; Apache-2.0 code and forks live on |
| Pydantic AI Gateway ⭐ 192 | Archived (Mar 30), folded into Logfire | The gateway became a platform feature |
| Helicone ⭐ 6.1k | Acquired by Mintlify (Mar) | Still committing — roadmap risk, not liveness |
| Portkey | Acquired by Palo Alto Networks (closed May 29) | Now a security platform's control plane |
| OpenRouter | Stripe reportedly in talks, ~$10B (Jul 23) | Unconfirmed — a signal, not a fact |
| BricksLLM, Glide | No commits since 2025-01 / 2024-08 | Never archived; the failure mode is silence |
Not a collapse — the same year, LiteLLM ⭐ 56.2k kept compounding, Bifrost ⭐ 7.2k won on measured overhead, Envoy AI Gateway reached v1.0 (Jun 23) as the first CNCF-backed production-stable option, and the coding-agent tier plus token-compression proxies became the fastest-growing categories in this list.
Three things that changes for you: check the last commit before the feature matrix; prefer code you can keep over a control plane you cannot; and if your traffic already lives in one cloud, that cloud's native gateway is now a real answer.
📉 The full picture — the consolidation evidence, the six-check evaluation standard, and three dated predictions with their disproof conditions: LANDSCAPE.md
The questions people actually ask (sourced from real threads) — answered first:
| You're asking… | The answer |
|---|---|
| "Cheapest way to hit many models right now?" | OpenRouter (~5.5% credit fee, ~340 models) — or 0% markup on your own keys: Vercel / Cloudflare AI Gateway → Cost-first |
| "Which free tiers still work, and what are the real limits?" | OpenRouter :free: 50 req/day (<$10 credits) or 1,000/day ($10+ top-up), 20 req/min shared (official limits). Eleven providers verified row-by-row in the free-tier table. The catch with "free": your prompts may train someone's model — check the fine print |
| "How much does the model choice matter?" | 106× — the same 100K-token report costs $0.03 (DeepSeek) vs $3.01 (GPT-5.5) → computed tables · calculator |
| "How much latency does the gateway itself add?" | Independently measured (nobody else does): Bifrost 0.62 ms · Portkey OSS 2.65 ms · LiteLLM 5.83 ms per request → data |
| "Will my prompt-cache discount still work through it?" | Often no — and it's silent. The most under-claimed discount in most bills → caching through a gateway |
| "Who sees my prompts?" | The gateway does, always — and routers range from ZDR-by-default to training on your prompts by ToS. See the data-retention matrix |
| "Sick of LiteLLM — what else?" | LiteLLM alternatives, compared honestly (overhead-measured: it's 10× heavier than Bifrost) |
| "Will it break my Claude Code / Codex / Cursor?" | The #1 gateway failure — but we measured it. Routing Claude Code to an OpenAI model, LiteLLM & Bifrost translate cleanly (3/3); Portkey OSS doesn't offer the path → the independent test. Still run your agent (tools + streaming) through it, and pin the version |
The list tells you which gateway; these chapters teach you how they work and why they fail — same evidence bar (dated, sourced, reproducible). Chapter map + reading order: HANDBOOK.md.
- 📕 The compatibility surface - Why gateways break Claude Code: three wire protocols field-by-field, five translation failure modes each anchored to a verified GitHub issue, the measured fidelity results, and a 10-minute self-test for your own gateway.
- 📗 Routing & model selection: the research landscape - Cost-aware cascades, learned routers, ensembling and self-routing — with the honest counter-evidence on when routing doesn't pay.
- 📘 Observability: what to measure and why - The OTel GenAI conventions, the metric tiers that separate instrumented from blind, and silent model drift.
- 📙 Anatomy of an AI gateway - The request lifecycle read from seven gateways' source at pinned commits: where the cache, the budget check and the retry boundary actually sit, why metering rarely survives a crash, and the six conditions under which you shouldn't run a gateway at all.
- 📕 Failover & reliability - What actually happens when a provider fails mid-request: only one of six gateways retries by default, what a client sees when the stream dies after the first token, whether a retry can be billed twice, and three providers' incompatible 429 contracts.
- 📗 Caching economics - Reads are 0.1× but writes are 1.25–2×, so caching loses money below a 21.7% hit rate — the formula, the four providers' non-portable rules, semantic-cache false hits with receipts, and a 30-second self-test.
- 📘 Virtual keys, budgets & metering - How a gateway counts your money: what a virtual key really scopes, pre-spend reservation vs post-hoc metering under concurrency, how streamed and reasoning and cached tokens get mis-counted, and what spend disappears when the process dies.
- 📕 MCP & agent gateways - Why agent traffic is not completion traffic: six properties change at once, MCP went stateless on 2026-07-28, tool-level authorization and secret brokering, and the DNS-rebinding CVE class that hit five codebases.
- ✅ All eight chapters are live — the map and reading order live in
HANDBOOK.md, linked above.
Every project in the list that clears the last row here, sorted by stars — a fast orientation. Archived projects are held back (TensorZero went read-only in June 2026) so the table stays things you can actually adopt today. Stars auto-refresh daily; full context (features, license, caveats) is in each linked section.
| Gateway | Stars | What it is | Jump to |
|---|---|---|---|
| headroom | ⭐ 66k | Context-compression proxy, not a router — fewer tokens per call | Self-hosted |
| LiteLLM | ⭐ 56.2k | The default OSS proxy + SDK — OpenAI format to 100+ providers | Self-hosted |
| CLIProxyAPI |
⭐ 47k | Wraps coding-CLI subscriptions (Claude Code, Codex…) into APIs | Self-hosted |
| OmniRoute |
⭐ 46.1k | Coding-agent token-saver across 231+ providers | Self-hosted |
| new-api | ⭐ 45k | The most active relay/billing hub for teams | China |
| Kong | ⭐ 44k | Mature API gateway with AI plugins (semantic cache, guard) | Enterprise |
| sub2api |
⭐ 36.7k | Pools subscription accounts behind one endpoint | China |
| Claude Code Router | ⭐ 36.6k | Route Claude Code and agent CLIs to any model/provider | Smart routing |
| one-api | ⭐ 36.3k | The original LLM API management / distribution system | China |
| MLflow AI Gateway | ⭐ 27.5k | Unified endpoints + governance in the MLflow platform | Observability |
| 9router |
⭐ 25.2k | BYOK local proxy, subscription→cheap→free fallback | Self-hosted |
| freellmapi |
⭐ 18.3k | Stacks 28 providers' free tiers behind one endpoint | Cost-first |
| Apache APISIX | ⭐ 17k | Cloud-native API + AI gateway (ai-proxy plugins) |
Enterprise |
| aisuite | ⭐ 16.1k | Andrew Ng's unified multi-provider client (a library) | Self-hosted |
| Portkey Gateway | ⭐ 12.7k | Fast TypeScript gateway, 1,600+ models, 50+ guardrails | Self-hosted |
| opencodex | ⭐ 9.4k | Translates Codex's Responses API to any provider | Smart routing |
| Chat Nio (CoAI) | ⭐ 9.3k | Multi-tenant gateway with a built-in credit/billing panel | Self-hosted |
| Higress | ⭐ 9.1k | Alibaba's AI-native gateway on Envoy/Istio | China |
| NVIDIA Dynamo | ⭐ 7.7k | Datacenter-scale, KV-cache-aware inference routing | K8s |
| Manifest |
⭐ 7.4k | Self-hosted TS router, 300+ models behind one /auto endpoint |
Self-hosted |
| Bifrost | ⭐ 7.2k | Go gateway, lowest independently-measured overhead | Self-hosted |
Hosted leaders (SaaS, no GitHub stars): OpenRouter (~340 models, ~5.5% fee) · Vercel AI Gateway & Cloudflare AI Gateway (0% markup) → Cost-first.
Stars measure popularity, not fitness for your need — that's what the category sections and the evidence-based scorecard are for.
Stars are cumulative; they never go down when a project stops shipping. Every tracked repo below is either archived or has taken no commit in six months. Refreshed daily from the GitHub API, same pass as the star counts — so this cannot go stale while the stars stay current.
| Project | Last commit | Status |
|---|---|---|
pydantic/pydantic-ai-gateway |
2026-03-30 | ⛔ Repo archived (read-only) |
tensorzero/tensorzero |
2026-06-11 | ⛔ Repo archived (read-only) |
lm-sys/RouteLLM |
2024-08-10 | |
EinStack/glide |
2024-08-12 | |
bricks-cloud/BricksLLM |
2025-01-05 | |
Helicone/ai-gateway |
2025-11-21 | |
fruitbars/simple-one-api |
2025-12-25 | |
songquanpeng/one-api |
2026-01-09 | |
lasso-security/mcp-gateway |
2026-01-22 | |
InftyAI/llmaz |
2026-01-26 | |
VoAPI/VoAPI |
2026-01-27 | |
Veloera/Veloera |
2026-02-11 |
A quiet repo is not automatically a bad pick — a stable proxy may simply be finished, and a listed project may have moved its work elsewhere. It is a question to ask before you depend on it, not a verdict.
🎯 Pain point: "I want many models for the least money and zero ops."
Where the line is: stolen-quota / reverse-engineered access is excluded from this list outright; transparent resellers are listed — but flagged New & unverified and folded below until an independent fidelity check passes.
Start here — the options with a track record:
- OpenRouter - The dominant model marketplace: ~340 models (live API count) behind one OpenAI-compatible API, pay-as-you-go with automatic failover; ~5.5% fee when buying credits. $113M Series B (May 2026), ~8M users.
- Vercel AI Gateway - Hundreds of models at provider list price (0% markup), $5/month free credits, zero-data-retention option; pairs naturally with the AI SDK.
- Cloudflare AI Gateway - Free control plane in front of your own provider keys: caching, dynamic routing, unified billing, and dollar-denominated spend limits (2026 beta).
- Requesty - EU-friendly OpenRouter alternative: 400+ models, sub-20ms failover, ~5% markup.
- Eden AI - Unified API for 500+ models plus vision/OCR/speech; EU-based, ~5.5% platform fee.
- Helicone AI Gateway (cloud) - Passthrough billing at 0% markup with observability bundled. Acquired by Mintlify (2026-03) and in maintenance mode — fine to use today, but weigh the roadmap risk.
- GPT-Load ⭐ 6.3k - High-performance Go proxy that rotates pools of API keys across channels to maximize quota usage.
- freellmapi ⭐ 18.3k - OpenAI-compatible proxy (MIT) that stacks the free tiers of 28 LLM providers behind one
/v1endpoint — smart routing, automatic failover, per-key quota tracking, encrypted key storage.⚠️ Stacking provider free quotas behind one endpoint can carry provider-ToS / account-ban risk — the repo itself says "personal experimentation only" — and the operator sells a paid live-catalog subscription ($19/yr; the router itself stays MIT, self-hosted, on your own keys). - AIMLAPI - One OpenAI/Anthropic-compatible endpoint fronting 400+ models (chat, image, video, audio, embeddings); prepaid, OpenRouter-style aggregator.
- Novita AI - Unified API to 200+ open-source models (DeepSeek/Qwen/Llama…) with load balancing, autoscaling and failover; also a GPU cloud.
- Glama Gateway - OpenAI-compatible gateway to 100+ models with consolidated billing, caching and logging (OSS core glama-ai/lightport).
New & unverified relays (17) — transparent resellers awaiting independent fidelity checks; read the relay watch-list first
- Loop Gateway - OpenAI-compatible proxy that meters every request in Bitcoin sats instead of dollars. 311 models via OpenRouter at a 15% markup. No accounts, no email, no card; top up over Lightning, get a bearer token. Three auth rails (prepaid bearer, L402, Cashu). Hosted at api.loopxxi.com. New & unverified (anonymous; its public GitHub repo has since been removed, so treat it as a closed hosted relay) — it resells frontier models through the operator's own OpenRouter account at a 15% markup, and account-less + crypto-prepaid means no recourse if it swaps models or vanishes; confirm fidelity with canary_check.py and only top up what you can afford to lose.
- nullsink (repo) - Account-less, metered proxy for frontier-model APIs, paid in Monero or Bitcoin. No accounts, no email, no card; mint a bearer token, prepay on-chain, and point the official SDKs at one base URL. ~10% markup taken once at top-up; no IP logging, no request logs; payment and token kept unlinkable. Self-hostable single binary (TypeScript/Bun, AGPL-3.0), live at nullsink.is. New & unverified (repo created 2026-06, ⭐ 12) — account-less + crypto-prepaid + no logs means no recourse if it swaps models or vanishes; confirm fidelity with canary_check.py and only top up what you can afford to lose.
- RunAPI - Hosted OpenAI-compatible relay (
https://runapi.ai/v1) spanning LLM + media-generation (image, video, music/audio) behind one developer account; ships an active CLI and SDKs (runapi-ai) though the gateway itself is closed/hosted. New & unverified (self-submitted) — confirm model fidelity (e.g. with canary_check.py) before relying on it in production. - TeamoRouter - Hosted SaaS gateway speaking OpenAI, Anthropic Messages and Gemini APIs over 500+ providers, with quality/latency/cost-aware "agentic routing" and a channel-dilution check that reroutes when a provider's output quality degrades; native Claude Code / Codex support, MCP, pay-as-you-go plus plans (Alipay/WeChat). New & unverified (the routing / quality-detection behavior is the operator's own description) — confirm model fidelity (e.g. with canary_check.py) before relying on it in production.
- KeepRouter - OpenAI- and Anthropic-compatible gateway: one key fronts 50+ models (Claude, GPT, Gemini, Mistral, Qwen, Kimi, GLM, DeepSeek, MiMo, MiniMax). Native /v1/messages means it works with Claude Code and the Anthropic SDK, not just the OpenAI SDK. Prepaid pay-as-you-go, billed at cost with 0% token markup (top-up fee 8%+$0.35); one genuinely free $0 model plus trial credit for new accounts. Bilingual EN/简体中文, Merchant of Record Paddle; not available in mainland China. New & unverified — confirm fidelity with canary_check.py before relying on it in production.
- AI快站 (aifast.club) - Operator-submitted OpenAI- and Anthropic-compatible relay aimed at the mainland-China market (Cursor / Claude Code / Codex / Dify integration docs). Its current model list and rates are published live at
/api/ratio_config, with service status at/api/status— read those rather than a fixed snapshot. New & unverified (closed-source, self-submitted) — confirm model fidelity with canary_check.py before relying on it in production. - ApiFlux - Hosted OpenAI-, Anthropic- and Gemini-compatible gateway with dedicated connection guides for Claude Code, Codex CLI and OpenCode, plus per-key usage and request logs in the console; the operator advertises all models at 85% of list price. New & unverified (self-submitted; below-list pricing makes independent fidelity checks especially important) — confirm model fidelity with canary_check.py before relying on it in production.
- NovAI - Hosted OpenAI-compatible relay fronting Chinese frontier models (DeepSeek, Qwen, GLM, Kimi, MiniMax, Doubao, Hunyuan) under one key; unlike the other Chinese-model relays here it also exposes image (Doubao Seedream) and video (Doubao Seedance) generation on the same endpoint, per-token pay-as-you-go with trial credit for new accounts. New & unverified (self-submitted) — confirm model fidelity (e.g. with canary_check.py) before relying on it in production.
- FlintAPI (repo) - Hosted OpenAI-compatible relay over Chinese LLMs (DeepSeek, Qwen, Kimi, GLM, MiniMax); the operator now describes it as a smart-routing engine that dispatches each prompt to a best-suited model rather than a plain aggregator, with free trial credit for new accounts. New and unverified (routing behavior is the operator's own description) — confirm model fidelity (e.g. with canary_check.py) before relying on it in production.
- FlowBar - Hosted OpenAI-compatible relay reselling dozens of models (GPT, Claude, Gemini, DeepSeek, Qwen, GLM, Kimi) below OpenRouter. Its differentiator is payment reach for buyers outside the US card system: Alipay/WeChat, USDT/USDC, PayPal and Apple/Google Pay through a local-methods processor. New and unverified — confirm model fidelity (e.g. with canary_check.py) before relying on it in production.
- Meshs One - Hosted OpenAI-compatible relay fronting Chinese frontier models (DeepSeek-V4, Qwen3.7-Max, MiniMax-M3) under one key, per-token pay-as-you-go (its
/v1endpoint returns anew_api_error, so it appears to run on new-api). New & unverified — closed-source and brand-new; confirm model fidelity (e.g. with canary_check.py) before relying on it in production. - CoderPlan - Hosted OpenAI-compatible relay fronting Claude/GPT/Gemini/DeepSeek/Grok for the China market, per-token pay-as-you-go, ¥10 minimum top-up with Alipay/WeChat; Hong Kong/Singapore nodes (API base
api.coderplan.ai/v1, which returns anew_api_error, so it appears to run on new-api). New and unverified — confirm model fidelity (e.g. with canary_check.py) before relying on it in production. - lxg2it ModelRouter (repo) - Solo-built, OpenAI-compatible router over 7+ providers (Anthropic, OpenAI, Google, Cerebras, Groq, Grok, GLM) with tiered automatic fallback that selects the cheapest available model. Free tier plus a paid tier advertised at 0% markup on Anthropic (a deposit fee may apply — verify current pricing). New and unverified — the public repo is a thin, unlicensed core stub still seeing fresh commits (routing logic lives in the closed hosted service; no license file) — confirm model fidelity (e.g. with canary_check.py) before relying on it in production.
- OpenPaths (repo) - Hosted OpenAI-compatible router auto-routing across 15+ providers (OpenAI, Anthropic, Gemini, Groq, xAI, DeepSeek, Mistral) under one API, spanning chat, image, video, music, speech, embeddings and transcription. New & unverified — despite the "open source" framing the GitHub repo is a no-code, unlicensed marketing mirror (canonical code lives on the third-party Codex Infinity platform and is agent-maintained), so treat it as a closed hosted relay and confirm model fidelity (e.g. with canary_check.py) before relying on it in production.
- RouterPlex - Hosted OpenAI-compatible gateway to 25+ models (GPT, Claude, Gemini, DeepSeek, Qwen, Kimi and others) across 11 providers; prepaid, billed per token at published vendor list rates, no subscription. $5 free credit for new accounts. New and unverified, closed-source — confirm model fidelity (e.g. with canary_check.py) before relying on it in production.
- TierUp - Hosted OpenAI-compatible gateway exposing four fixed performance tiers (tier-1…tier-4) instead of model names, each mapped server-side to a current best-value model; routes through OpenRouter and prices ~50% under the underlying models' retail, transparently subsidized during an early product-market-fit phase (solo-built, ~zero production users, tier 1 currently free). New & unverified — confirm model fidelity (e.g. with canary_check.py) before relying on it in production.
- AllRouter - Hosted OpenAI- and Anthropic-compatible relay: one key over ~25 models (Kimi K3, Claude, GPT, Gemini, DeepSeek, GLM, Grok), advertised at the providers' own list prices with no markup, plus a free GLM/Gemma tier the operator says runs on its own GPUs; Alipay/WeChat Pay accepted, and an MCP entry point (
npx @allrouter/mcp-server). Its/v1endpoint returns anew_api_error, so it appears to run on new-api. New & unverified (self-submitted; the linked GitHub repo is a docs/marketing mirror, not the gateway's source, so treat it as a closed hosted relay) — at-list-price resale leaves no visible margin, which makes an independent fidelity check especially worthwhile; confirm with canary_check.py before relying on it in production.
💡 Squeeze more from any gateway: enable semantic caching (Bifrost, Zuplo; Kong's
ai-semantic-cacheis Enterprise), set spend limits (Cloudflare, Zuplo, Pydantic/Logfire), and route easy prompts to cheap models (see Smart routing).
One of the most-asked questions in the ecosystem, and the internet's answers are mostly stale. Every row below was re-verified against the provider's own docs (machine-readable, verified 2026-07-09, CI-enforced ≤30-day re-review). "Unpublished" means the provider now hides the numbers behind a login — we say so instead of quoting third-hand figures.
| Provider | What's free (verified limits) | Notable free models | Card? | The catch |
|---|---|---|---|---|
OpenRouter :free |
50 req/day (<$10 lifetime top-up) → 1,000 req/day ($10+ once); 20 req/min shared across all :free models |
rotating :free pool |
❌ | free routes hit third-party providers that may train on your data (per-provider policy — check the free/paid routing settings) |
| Google Gemini API | Flash-class models free; per-model RPM/RPD unpublished since 2026 (login-only in AI Studio); daily quota resets midnight PT | Gemini 3.5 Flash · 3.1 Flash-Lite · 2.5 Flash · Gemma 4 | ❌ | free-tier content is "used to improve our products" (their pricing page's words) |
| Groq | per-model: Llama-3.3-70B 30 RPM / 1K req/day / 100K tok/day · GPT-OSS-120B 30 RPM / 1K RPD / 200K TPD · Llama-3.1-8B 14.4K RPD / 500K TPD | GPT-OSS-120B/20B · Llama-4-Scout · Llama-3.3-70B · Qwen3-32B | ❌ (card only to upgrade) | daily token caps burn fast on 70B+; limits are org-level |
| Cerebras | 5 RPM / 30K TPM / 1M tokens/day, flat across models | GPT-OSS-120B · GLM-4.7 · Gemma-4-31B | ❔ not stated | 5 RPM = one interactive user; "all models" marketing vs 3 models on the actual table |
| GitHub Models | any GitHub account: low-tier 15 RPM / 150 req/day, high-tier 10 RPM / 50 req/day; 8K in / 4K out per request | GPT-5 · o4-mini · Llama 4 · Phi-4 · DeepSeek-R1 | ❌ | explicitly experimentation-only; the 8K/4K per-request caps rule out long context |
| Cloudflare Workers AI | 10,000 neurons/day (≈4M input tok/day on Llama-3.2-1B, ≈314K on GPT-OSS-120B — our arithmetic from official rates) | Llama-3.3-70B · GPT-OSS-120B/20B · DeepSeek-R1-distill | ❔ not stated | neurons meter compute — output burns 5–10× faster than input |
| Mistral | free Experiment mode; exact limits unpublished (per-account, Admin Console) | Large · Small · Codestral (reported) | ❌ | trains on your data by default — free users must manually toggle it off |
| Cohere | trial key: 1,000 calls/month; Chat 20 RPM | Command A · Command R+ | ❌ | evaluation scale only |
| SambaNova | 20 RPM / 20 req/day / 200K tok/day | DeepSeek-V3.1 · Llama-3.3-70B · GPT-OSS-120B | ❌ | 20 req/day = demo only (very fast inference though) |
| Hugging Face | $0.10/month provider-passthrough credits (PRO: $2/mo) | 200+ provider-routed (DeepSeek-V3 …) | ❌ | $0.10 ≈ a handful of big-model requests |
| Z.ai (GLM) | GLM Flash models $0 in AND out; rate limits unpublished (per-key, login) | GLM-4.7-Flash · GLM-4.5-Flash · GLM-4.6V-Flash | ❔ not stated | opaque shifting concurrency limits; China-HQ provider — weigh your data sensitivity |
Trial credits ≠ free tiers (they expire): NVIDIA build.nvidia.com (1,000 requests at signup, +4,000 with a business email — staff-forum figures; prototyping-only license) and Alibaba Cloud Model Studio international (per-model quotas, hard 90-day expiry, Singapore region). Recently discontinued — ignore stale listicles: Together AI killed its -free models (now $5 minimum prepaid, "does not currently offer free trials"), Moonshot/Kimi requires a $1 top-up to start, and xAI's much-cited data-sharing credits are no longer documented on any public page. Full evidence per row: data/free_tiers.json. A wrong row is a bug — report it.
Pain point: "My keys, my infra, no per-token middleman fee."
- LiteLLM ⭐ 56.2k - The default choice: Python SDK + proxy server speaking OpenAI format to 100+ providers (local Ollama included), with virtual keys, budgets, load balancing and guardrails; the admin UI / virtual keys need a Postgres database.
- Portkey Gateway ⭐ 12.7k - Fast TypeScript gateway (1,600+ models, 50+ guardrails) that also powers Portkey's commercial LLMOps platform.
- CLIProxyAPI ⭐ 47k - Go gateway that wraps coding-agent CLI subscriptions (Claude Code, Codex, Gemini, Grok, Antigravity) into OpenAI/Gemini/Claude/Codex-compatible APIs with multi-account pools, round-robin load balancing and a management API; one of the highest-starred OSS gateways in the space. BYO accounts — but routing OAuth coding-tier subscriptions through an API can violate provider ToS, so weigh account-ban risk. Companion: CPA-Manager-Plus ⭐ 2.5k — a self-hosted management panel + observability dashboard for it (requests, usage, cost, quota, failures, account health); the same ToS caveat applies.
- 9router ⭐ 25.2k - MIT self-hosted BYOK local proxy that auto-routes across 40+ providers with subscription→cheap→free fallback, multi-account load balancing and token compression; cost-first and very popular. Two cautions, and the security one is the bigger: 19 published advisories — 6 critical, 11 high, 12 of them in July 2026 alone — clustered on unauthenticated access to the proxy and the provider keys it holds, and three carry no recorded fix, two of those critical (see the supply-chain matrix below). Run it on a network you control, pinned to the newest release. Separately, its free/OAuth coding-tier routing (Claude Code, Codex, Kiro) carries provider-ToS/account-ban risk.
- OmniRoute ⭐ 46.1k - MIT self-hosted TypeScript gateway: one endpoint to 231+ providers (50+ free), plugging Claude Code / Codex / Cursor / Cline / Copilot into free Claude/GPT/Gemini with stacked token compression (15–95% savings), 17 routing strategies, smart auto-fallback and MCP/A2A. A 2026 breakout of the coding-agent "token-saver" wave — genuine code (not a relay farm), but its free/OAuth coding-tier routing carries provider-ToS/account-ban risk.
- Chat Nio (CoAI) ⭐ 9.3k - Multi-tenant "one-stop" gateway with a built-in admin + credit/subscription billing panel over 200+ models / 35+ providers, priority-based load balancing and model caching — the same commercial-panel genre as the new-api / one-api / VoAPI entries here.
- TensorZero ⭐ 11.7k - Archived June 2026
⚠️ (company wound down; repo read-only, Apache-2.0 code + community forks remain). Rust gateway unified with observability, evals, experimentation and optimization. - Bifrost ⭐ 7.2k - Go gateway from Maxim AI claiming ~50x LiteLLM throughput; adaptive load balancing, cluster mode, MCP support, a built-in web UI, and documented Ollama support for local models — zero-config start via
npxor Docker. - Otari ⭐ 374 - Mozilla AI's self-hosted gateway (Apache-2.0, Python): one OpenAI- and Anthropic-compatible endpoint over 40+ providers via any-llm, with virtual keys, per-user/per-key budgets enforced before a request runs, and
/v1/usagespend tracking; single-Docker start, optional hosted platform (otari.ai). Launched April 2026 and one of July's fastest risers. - Traceloop Hub ⭐ 226 - High-scale gateway written in Rust from the Traceloop team (OpenLLMetry / OTel-for-LLMs); OpenTelemetry-native observability built in.
- Helicone ⭐ 6.1k - Observability-first platform (YC W23) with a Rust ai-gateway ⭐ 620.
- Plano ⭐ 7k - AI-native proxy and data plane for agents (formerly Arch Gateway / archgw).
- AxonHub ⭐ 5k - Go gateway: call 100+ LLMs from any SDK behind one OpenAI/Anthropic-compatible endpoint, with built-in failover, load balancing, cost control and end-to-end tracing. BYOK self-hosted.
- Manifest ⭐ 7.4k - Self-hosted TypeScript router (MIT): one OpenAI-compatible
/autoendpoint (plus/v1/messagesfor Anthropic clients) over 300+ models / 31+ providers, mixing API keys, local models (Ollama/LM Studio) and coding-tier subscriptions with complexity/header-based routing, cost tracking, budgets and failover. BYO accounts — routing OAuth coding-tier subscriptions through an API can carry provider-ToS risk. - LLM Gateway ⭐ 1.5k - Open-source OpenRouter alternative: route, manage and analyze requests across providers.
- APIPark ⭐ 1.8k - Cloud-native LLM API management and distribution platform.
- Pydantic AI Gateway ⭐ 192 - BYOK gateway with cost caps and OTel;
⚠️ repo archived, now folded into Pydantic Logfire. - OptiLLM ⭐ 4.2k - Optimizing inference proxy that boosts accuracy via test-time compute techniques.
- aisuite ⭐ 16.1k - Andrew Ng's unified multi-provider client. A library rather than a deployable proxy — fits when you don't want network hops.
- Shepherd Model Gateway (SMG) ⭐ 457 - Engine-agnostic gateway in Rust: one OpenAI/Anthropic-compatible endpoint over vLLM/SGLang/TRT-LLM + cloud providers, with KV-cache-aware routing and WASM plugins.
- RelayPlane ⭐ 194 - MIT, local-first proxy (npm): 11 providers behind one endpoint with per-request cost attribution and hard daily/hourly budget caps.
- SentryNode Gateway ⭐ 0 - Open-core (Apache-2.0) AI proxy for cost governance / FinOps routing: adaptive model routing, budget caps and audit logging. Early-stage; the public repo currently ships a demo scaffold.
- GoModel ⭐ 1.1k - Lightweight single-binary Go gateway (open-source LiteLLM alternative) exposing one OpenAI/Anthropic-compatible API across 18+ providers with caching, guardrails and usage/cost tracking; fast-growing, though its throughput-vs-LiteLLM figures are vendor-run.
- OpenGateLLM ⭐ 180 - Production-grade open-source GenAI gateway from France's Etalab (powers the government's "Albert" assistant): one OpenAI-compatible API over self-hosted + provider models, with auth, rate limits and usage tracking. Distinct public-sector / EU-sovereignty angle.
- TokenHub ⭐ 950 - Private enterprise AI gateway (Apache-2.0) with role-based workspaces for users, team leaders and administrators — from astaxie, creator of the beego framework.
- dario ⭐ 340 - Local OpenAI- and Anthropic-compatible proxy (MIT) that brings a Claude Pro/Max subscription to Cursor, Cline, Aider and the Agent SDK, with session-affinity routing, multi-account pooling and drift detection.
⚠️ Routing OAuth coding-tier subscriptions through an API carries provider-ToS / account-ban risk (same cohort as CLIProxyAPI). - Aurora Gateway ⭐ 33 - Go single-binary gateway that auto-discovers providers from environment variables, with provider pools, exact/semantic/prompt caching, guardrails, audit logging and usage analytics (Apache-2.0). Early-stage (launched mid-2026, single maintainer); its headline speed-vs-LiteLLM figures are vendor-run.
- headroom ⭐ 66k - Apache-2.0 context-compression layer shipping as a library, a drop-in local proxy (
headroom proxy) and an MCP server: it shrinks tool outputs, logs, files and RAG chunks reversibly before they reach the model. Not a multi-provider router — reach for it when the bill is driven by context size rather than model choice, and pair it with a router. Its 15–20% (coding agents) / 60–95% (JSON) savings figures are the project's own. - OpenRelay ⭐ 2.3k - MIT local proxy that auto-discovers the AI quota you already own — Claude Desktop/Code, Kiro, Copilot, Cursor, Windsurf, Codex, Gemini CLI — alongside 45 direct API or local endpoints, then lets any tool draw on any of them. BYO accounts, no resold quota.
⚠️ Routing coding-tier subscriptions through an API carries provider-ToS / account-ban risk (same cohort as CLIProxyAPI). - Paritok ⭐ 1k - Apache-2.0 compression gateway for coding agents: fronts any
BASE_URLagent (Claude Code, Cursor, Codex, OpenHands) and compresses conversation context with an open-weights 4B code model, to fit more turns in the same window. Same niche as headroom, different mechanism — a learned model rather than content-aware rules. Early-stage (repo opened 2026-07); its 25%→85% savings curve is vendor-run. - copilot-api ⭐ 962 - MIT gateway that re-exposes a GitHub Copilot subscription (plus Codex and OpenCode Go) as OpenAI- and Anthropic-compatible APIs for any client.
⚠️ Same provider-ToS / account-ban risk as the other subscription-wrapping proxies here. - CliRelay ⭐ 897 - MIT Go gateway aimed at coding CLIs: one OpenAI/Claude/Gemini/Codex-compatible endpoint plus a multi-tenant web console with request logs and per-tenant spend quotas — the part most single-user coding proxies omit once you have to hand keys to a team.
- Swobu ⭐ 6 - Local-first Go gateway (single binary, AGPL-3.0) that terminates OpenAI, Anthropic Messages and remote-MCP client traffic and routes it across providers, regions, accounts and local engines (Ollama, LM Studio, vLLM) with failover; clients point at one workspace endpoint and name a route, and the stated goal is to preserve tools, reasoning and session state across a switch. BYOK — no resold quota. Early-stage (still on 1.0 release candidates, single maintainer).
- Unified AI System ⭐ 3 - Apache-2.0 local-first Node.js gateway and MCP server for Codex/Cursor/Cline: explicit provider/model selection, streaming chat, nine governed MCP tools and prompt enhancement that runs without calling a provider. Starts with no credentials against a fake provider, which makes it cheap to try. Early-stage public preview — fallback candidates are reported but not yet executed automatically.
- Token Efficiency ⭐ 0 - MIT Python/Vue gateway bundling prompt-token compression, semantic caching, a multi-tenant portal and cost analytics over OpenAI/Anthropic/MiniMax/GLM/Kimi/DeepSeek; ships docker-compose and Grafana dashboards. Early-stage (repo opened 2026-08, single maintainer); its compression/savings figures are vendor-run, and a hosted edition shares the name.
⚠️ Stale but historically notable: BricksLLM ⭐ 1.2k (PII masking, per-key limits; inactive since early 2025), Glide ⭐ 160 (inactive since 2024).
Pain point: "Audit logs, PII redaction, RBAC, on-prem, and the EU AI Act."
⚠️ The EU AI Act deadline moved — check which half applies to you before you buy on urgency. The Digital Omnibus (Regulation (EU) 2026/1744, in force 2026-07-27) pushed the Annex III high-risk obligations — the ones that mandate Article 12 event logging, the reason most teams shop for an auditable gateway — from 2026-08-02 to 2027-12-02, and AI embedded in Annex I regulated products to 2028-08-02. What did start on 2026-08-02 is Article 50 transparency: disclosing AI interaction, marking synthetic content, labelling deepfakes. Those are application-layer duties your gateway mostly cannot discharge for you. Verified 2026-08-12.
⚖️ Coverage honesty: of the vendors below, only Kong and Apache APISIX carry scores in the 5-axis scorecard — the rest are listed for coverage, not evaluated (mostly closed-source platforms our evidence rubric can't independently verify). Treat their capability descriptions as vendor claims; the identity/governance paywalls we could verify are in the SSO-tax table.
Who paywalls SSO / SCIM / RBAC / audit logs? Six of nine vendors gate SSO behind an enterprise tier — Cloudflare is the only one shipping it free on every plan, and Vercel the only one with a sticker price for the tax ($300/month). Full tier-by-tier matrix, primary-sourced: the SSO-tax table (machine-readable).
- Kong AI Gateway ⭐ 44k - Mature API gateway with AI plugins: semantic caching/routing, prompt guard, token rate-limiting; Konnect for managed control plane.
- Apache APISIX ⭐ 17k - Cloud-native API + AI gateway with
ai-proxy/ai-proxy-multiplugins. - Envoy AI Gateway ⭐ 1.9k - CNCF-aligned GenAI access on Envoy Gateway, backed by Tetrate and Bloomberg.
- kgateway ⭐ 5.7k - CNCF API/AI gateway, the base of Solo.io's commercial Gloo AI Gateway.
- TrueFoundry AI Gateway - Enterprise gateway with routing, guardrails and RBAC, deployable into your K8s/VPC.
- nexos.ai - Enterprise AI gateway/orchestration from the Nord Security founders (€30M Series A, Oct 2025).
- Tyk AI Studio - AI governance suite: budgets, model catalogs, guardrails on Tyk's gateway.
- Gravitee Agent Mesh - LLM Proxy, MCP Proxy and A2A support inside Gravitee APIM.
- WSO2 AI Gateway - Egress management for LLM traffic: model routing, semantic caching, guardrails.
- F5 AI Gateway - Containerized AI traffic gateway; data-leakage detection via the LeakSignal acquisition (announced Jul 2025).
- IBM API Connect AI Gateway - AI Gateway capability inside IBM API Connect: policy enforcement, masking and audit for LLM traffic (its dedicated docs line has since moved to the standalone DataPower Interact Gateway below).
- IBM DataPower Interact Gateway - Standalone gateway built on the DataPower Nano Gateway governing LLM and MCP traffic at runtime: an MCP gateway exposing existing APIs as agent-discoverable MCP tools, plus governed endpoints to external LLM providers (announced Jul 2026).
- MuleSoft AI / Omni Gateway - Governs LLM, MCP and agent traffic alongside classic APIs.
- Lunar.dev ⭐ 481 - Egress consumption gateway repositioned around MCP/agent governance.
- KrakenD AI Gateway - High-performance, stateless Go API gateway (krakend/krakend-ce ⭐ 2.7k) with an AI proxy + prompt-security layer.
- Broadcom Layer7 AI Gateway - LLM traffic governance, threat protection and quotas on the mature Layer7 API platform.
- Cequence AI Gateway - API-security-first AI gateway: discovery, guardrails and threat protection for LLM/agent traffic.
- Axway Amplify AI Gateway - Centralized control plane on Axway's Amplify platform governing LLM/MCP/agent traffic with business-logic model routing, RBAC, spend caps, prompt-injection controls and RAG integration, from a 10× Gartner MQ API-management Leader.
- Red Hat Connectivity Link - Kubernetes-native gateway (built on the Kuadrant project, successor to 3scale) unifying AI gateway, API management and multicluster connectivity; powers OpenShift AI Models-as-a-Service as the front door governing external and self-hosted LLM endpoints.
- Sensedia AI Gateway - Gartner-recognized APIM vendor's agnostic AI gateway governing LLMs, MCP servers and AI agents with multi-model routing, guardrails, cost controls and observability across a multi-cloud control plane.
- Ambassador Edge Stack - Envoy-based, Kubernetes-native API gateway (OSS core emissary-ingress ⭐ 4.5k) whose AI Gateway layer adds LLM-provider routing, token rate-limiting and fallback — a peer to Kong/Tyk/APISIX in the API-vendor cohort.
- Shim - Compliance-oriented AI gateway (closed-source, hosted/on-prem): PII and secret redaction spanning Turkish TCKN, SSN and cloud/API keys, audit trails positioned for EU AI Act / GDPR / KVKK evidence, plus semantic caching and cost optimization. Young product — verify certification claims directly.
Pain point: "We're already committed to one cloud — give us the native path."
- AWS Bedrock - Multi-model access via the unified Converse API, cross-region inference, and AgentCore Gateway for tools/MCP.
- Azure API Management — GenAI gateway - Token limits, semantic caching and load balancing in front of Azure OpenAI / AI Foundry.
- Google Apigee + Vertex AI - LLM gateway patterns on Apigee with Vertex Model Garden as the managed hub.
- Cloudflare AI Gateway — See Cost-first; the strongest free first-party option.
- Vercel AI Gateway — GA, 0% markup, ZDR option; the default for Next.js/AI SDK shops.
- Databricks Unity AI Gateway - Mosaic AI Gateway folded into Unity Catalog, adding agent + MCP governance.
- Tencent Cloud AI Gateway - Tencent's first-party cloud-native intelligent gateway bundling LLM + MCP + Agent gateways with protocol conversion, cost/performance-based routing, and unified access to Hunyuan + third-party models.
Pain point: "Domestic models (Qwen/DeepSeek/GLM/Kimi), CNY payment, key distribution & billing for teams."
- new-api ⭐ 45k - The most active one-api fork, now a "unified AI model hub": protocol conversion, multi-user key distribution and billing behind a polished admin UI, Rerank/Realtime endpoints. Not China-only — the panel fronts OpenAI-compatible, Claude and Gemini channels too, a fit for any family or small team wanting per-user keys and budgets. AGPL-3.0.
- one-api ⭐ 36.3k - The original LLM API management & distribution system (OpenAI/Azure/Claude/Gemini/DeepSeek/Doubao…); development has slowed.
- Higress ⭐ 9.1k - Alibaba's AI-native gateway on Envoy/Istio, first-class Tongyi/DeepSeek support; hosted version at higress.ai.
- GPT-Load ⭐ 6.3k - Smart API-key rotation multi-channel proxy in Go.
- one-hub ⭐ 2.9k - Fork of one-api with better non-OpenAI function calling and stats.
- simple-one-api ⭐ 2.3k - Single binary adapting Qianfan/Spark/Hunyuan/MiniMax/DeepSeek to the OpenAI interface.
- Octopus ⭐ 2.4k - Personal LLM API aggregation gateway unifying multiple providers behind one endpoint, with load balancing and OpenAI/Anthropic protocol conversion (Go + Next.js).
- Veloera ⭐ 1.6k - Newer relay platform in the one-api/new-api lineage.
- uni-api ⭐ 1.2k - Lightweight single-config unified API manager, no frontend.
- CrossLink ⭐ 396 - Unified LLM proxy gateway (Apache-2.0) with load balancing, failover, rate limiting, budget management, content auditing, caching and an MCP gateway layer.
- APIPark ⭐ 1.8k - China-origin, cloud-native AI & API gateway with an open developer portal.
- VoAPI ⭐ 1.1k - Polished new-api-lineage relay/billing panel (Go), focused on UI and operations.
- done-hub ⭐ 801 - Fork of one-api/new-api with richer billing and channel management.
- sub2api ⭐ 36.7k - Go relay platform that pools Claude/OpenAI/Gemini/Antigravity subscription accounts (OAuth, session keys, API keys) behind one OpenAI/Anthropic-compatible endpoint, adding cost-sharing "carpool" billing (Stripe/Alipay/WeChat), key distribution and per-token rate limits. One of 2026's fastest-rising China-ecosystem relays — but account-pooling sits adjacent to the resold-relay category this list excludes; BYO accounts and vet before use.
- AIHelms ⭐ 942 - GPL-3.0 enterprise AI-asset governance platform (FastAPI + Vue3): a unified gateway over OpenAI/Azure/Claude/DeepSeek with token scheduling, internal-vs-external dual pricing, cost attribution, SSO, audit logging and centralized MCP-tool/skill distribution. Governance-first rather than throughput-first — closer to the enterprise-compliance section than to the relay panels above.
- Bella OpenAPI ⭐ 237 - MIT gateway from Lianjia/Beike that goes past chat completion: text embedding, ASR, TTS and image generation behind one API, with billing, rate limiting and resource management. Its distinguishing claim is provenance — the capabilities come out of a large operator's own production stack.
- AI Proxy ⭐ 525 - Self-hosted Go gateway from the Sealos team that accepts OpenAI/Claude/Gemini protocols, converts between them, and adds multi-channel routing, load balancing, rate limiting, multi-tenant isolation, and a caching/web-search/reasoning plugin layer.
- metapi ⭐ 3.2k - Self-hosted "router of routers": aggregates your accounts across new-api/one-api/OneHub/DoneHub/Veloera/AnyRouter/sub2api into one key, with cost/balance/utilization-weighted smart routing, channel cool-down/retry, model auto-discovery and OpenAI⇄Claude conversion (TypeScript, MIT). Routing software only — vet the upstream relays it points at.
- Volcengine AI Gateway - ByteDance's cloud AI gateway: unified access, routing and governance for Doubao + third-party models.
⚠️ This list deliberately excludes reverse-engineered / resold "free-api" relays — and not on principle alone. Two 2026 measurement studies found systematic fraud across the relay population: Real Money, Fake Models measured model-identity failures in 45.8% of fingerprint tests and output divergence up to 47%; Your Agent Is Mine caught routers injecting malicious code and exfiltrating planted API keys. The precise boundary: stolen-quota / reverse-engineered access is excluded outright, while transparent resellers are listed but flagged New & unverified in Cost-first until independently fidelity-checked. If you're forced to vet one anyway, use the canary-diff test in How to choose safely.
Pain point: "Agents call tools now — govern MCP traffic like you govern APIs." The newest category (2025–2026).
- agentgateway ⭐ 4.3k - Linux Foundation proxy for agentic traffic (LF Projects, per its own CHARTER — not a CNCF-hosted project): MCP governance and agent-to-agent (A2A) communication.
- Lunar.dev MCPX ⭐ 481 - Gateway for managing MCP server consumption.
- Tetrate Agent Router Service - Managed Envoy AI Gateway fleet: LLM + MCP gateway with guardrails (~5% fee).
- Zuplo AI Gateway - Programmable policies: USD spend limits, prompt-injection detection, secret masking, MCP support.
- NetFoundry MCP/LLM Gateways - Zero-trust gateways for AI deployments (launched June 2026).
- AWS AgentCore Gateway — Tool/MCP gateway inside Bedrock AgentCore.
- IBM ContextForge ⭐ 4.3k - MCP gateway/registry federating many MCP servers behind one endpoint with auth, rate limits and observability.
- Docker MCP Gateway ⭐ 1.5k - Docker-maintained
docker mcpCLI plugin that runs and federates MCP servers as containers behind one endpoint, with secret management, call interception and a deployment-time per-tool enable-list (identity-free — not per-caller authorization). - MetaMCP ⭐ 2.6k - Aggregates MCP servers into one endpoint with middleware (auth, filtering) and a management UI.
- ToolHive ⭐ 2k - Go platform that runs MCP servers in isolated containers and fronts them with a unified, secured gateway (access policies, "virtual MCP" aggregation).
- Microsoft MCP Gateway ⭐ 778 - Microsoft-maintained reverse proxy + management layer for MCP servers: session-aware stateful routing and lifecycle management on Kubernetes.
- 1MCP ⭐ 483 - Unified MCP server (TypeScript) aggregating many MCP servers behind one endpoint, with HTTP access and CLI-based discovery for agents.
- mcpproxy-go ⭐ 313 - Local Go MCP proxy that federates multiple MCP servers behind one endpoint, with BM25 tool-search filtering, token reduction, and auto-quarantine/security scanning of new servers.
- MCPJungle ⭐ 1.2k - Self-hosted MCP registry + gateway for central tool governance in enterprises.
- Obot ⭐ 939 - Open-source agent platform with an MCP gateway for governing tool access.
- Director ⭐ 481 - Middleware to run, secure and observe MCP servers behind one connection.
- Lasso MCP Gateway ⭐ 385 - Security-first MCP gateway: plugin guardrails, secret masking, threat detection.
- Armorer Guard ⭐ 40 - Local Rust MCP proxy that wraps stdio servers and inspects tool-call arguments for prompt injection, credential leakage, exfiltration, and risky actions.
- fak ⭐ 30 - Security-first agent/MCP firewall: a single dependency-free Go binary (Apache-2.0) fronting any OpenAI/Anthropic/MCP backend, where a default-deny capability allow-list adjudicates every tool call and suspicious tool results are quarantined out of the model's context, plus bearer/
x-api-keyauth, anX-Trace-Idaudit trail and Prometheus/metrics. New and early-stage. - Archestra ⭐ 4.2k - Kubernetes-native MCP gateway with OAuth On-Behalf-Of user-delegated tool access, an A2A agent-to-agent gateway, and deterministic dual-LLM / "lethal trifecta" guardrails plus per-environment egress and cost limits, built for enterprise agent deployments ($13.5M funding).
- Unla ⭐ 2.2k - Lightweight Go MCP gateway that turns existing REST/gRPC APIs and MCP servers into standardized MCP endpoints with zero code changes, behind one gateway with multi-tenant sessions, OAuth, hot-reload config and a management UI.
- Jarvis Registry ⭐ 2.8k - Enterprise MCP/agent gateway fronting internal tools behind one authenticated MCP-over-SSE/HTTP endpoint with OAuth2/OIDC identity (Keycloak/Cognito/Entra), tool-level RBAC/ACL, agent orchestration, and OpenTelemetry/Prometheus observability.
- MCP Gateway & Registry ⭐ 859 - Enterprise MCP gateway + registry centralizing access to many MCP servers behind one OAuth-protected endpoint, with virtual MCP servers, semantic tool discovery, A2A agent discovery and fine-grained governance/audit; AWS-aligned.
- Nexus (Grafbase) ⭐ 434 - Rust AI router from Grafbase that aggregates MCP servers (STDIO/SSE/HTTP) and LLM providers behind one endpoint with context-aware fuzzy tool search, OAuth2/TLS security, rate limiting and OpenTelemetry.
- Pomerium ⭐ 4.9k - Identity-aware access proxy with MCP support: policy-based auth in front of MCP servers.
- Open Connector ⭐ 4.6k - Open-source auth gateway (Apache-2.0, OOMOL Lab) connecting AI agents to 1000+ SaaS providers through SDK, CLI, MCP, HTTP and OpenAPI — it governs agent→SaaS tool credentials and access rather than LLM completion traffic.
- toolport ⭐ 153 - Local-first MCP gateway (MIT): one port for every tool and every AI client, with lazy tool discovery (~90% token savings, per its docs), tool integrity checks + quarantine, and secrets kept in the OS keychain.
These cut across the need-based sections above — routing intelligence, observability, and Kubernetes infra that complement whichever gateway you picked.
Pain point: "Send each prompt to the cheapest model that can handle it."
💸 Routing a coding agent specifically (Claude Code / Codex / Cursor)? The dedicated deep-dive classifies claude-code-router, OmniRoute, 9router, CLIProxyAPI and sub2api by mechanism — BYO-key vs own-OAuth vs pooled accounts — with the actual ToS clauses and dated ban reports: Coding-agent routers (2026): savings vs. ban risk.
🧭 The theory behind these picks — cost-aware cascades, learned routers, multi-model ensembling, self-routing, and the benchmarks that test them (with the honest counter-evidence on when routing doesn't pay): the routing & model-selection chapter in 📖 Learn above is the dated, sourced research map, sibling of the observability chapter.
⚠️ The most common gateway failure isn't routing — it's translation. Across 2025–26 issue trackers, the single largest bug category on every major gateway is corrupted tool-call / thinking-block / streaming translation: Portkey's most-commented issue (#980, tool_use ids lost), OpenRouter's AI-SDK thinking-mode breakage (filed three times: #245), Claude Code erroring through LiteLLM (#13373) and new-api (#1854) — "claude code" appears in 465 LiteLLM issues (repo:BerriAI/litellm "claude code" is:issue, 2026-07-29 — a plain search withoutis:issuereturns ~2,000 because it counts pull requests too). "OpenAI-compatible" is a spectrum, not a checkbox (LangChain's own compat issue). We measured it — pointing Claude Code at an OpenAI model, LiteLLM and Bifrost translate cleanly (3/3) while Portkey OSS doesn't offer the path (and the/v1/messagestransport shifts across versions, so pin it). Before committing, still run your actual agent (tool calls + streaming + thinking) through the gateway, not just a hello-world completion.
- Not Diamond - SOTA model-routing intelligence; powers OpenRouter's Auto router.
- Martian - Pioneer commercial model router; Accenture partnership.
- Inworld Router - One API for 200+ models with real-time complexity-based routing and 0% markup (pass-through pricing); adds first-party realtime inference for open models. Research preview.
- RouteLLM ⭐ 5.3k - LMSYS's open router framework (research-grade; inactive since 2024 but still the canonical paper/code).
- OpenRouter Auto — One model id (
openrouter/auto) that routes per-prompt. - Unify - Early neural LLM router (company since pivoted to agents).
- Bifrost adaptive load balancing / Cloudflare dynamic routing — routing built into gateways themselves.
- Claude Code Router ⭐ 36.6k - Route Claude Code (and other agent CLIs) to any model/provider — DeepSeek, Qwen, local — by request type.
- opencodex ⭐ 9.4k - Universal provider proxy (MIT) for OpenAI Codex and Claude Code: translates Codex's Responses API to any provider (Claude, Gemini, Grok, DeepSeek, local Ollama) with streaming, tool calls, reasoning tokens and images intact — to Codex what Claude Code Router is to Claude Code.
⚠️ Its ChatGPT account-pool management sits in the same provider-ToS / account-ban-risk cohort as CLIProxyAPI. - ClawRouter ⭐ 6.6k - Agent-native LLM router (TypeScript) with local sub-ms routing across 41+ models, built so autonomous agents can pay per call via x402/USDC with no signup or API key. The routing client is open-source — but its account-less hosted access (8 free models + crypto pay-per-use) is resold access: verify model fidelity with canary_check.py and prefer your own keys in production.
- workweave/router ⭐ 947 - Go router for agentic systems: routes each prompt to the right model in <50ms behind one OpenAI-compatible endpoint, pitched as a drop-in endpoint swap that cuts 40–70% of cost.
- UncommonRoute ⭐ 685 - MIT drop-in OpenAI proxy that routes by prompt difficulty; markets hard numbers (≈82% cost savings, 79.4% accuracy, 93.4% pass rate) and integrates with Claude Code / Cursor / Codex.
- OrcaRouter Lite ⭐ 418 - MIT self-hosted single-workspace router (BYOK, OpenAI-compatible) from Continuum AI with a managed hosted upgrade path; ranks at/near the top of the RouterArena leaderboard.
- RouterArena ⭐ 118 - Open evaluation framework + live leaderboard for LLM routers (standardized datasets, cost/quality metrics) — pick a router on data, in the spirit of this list's benchmarks.
- vLLM Semantic Router ⭐ 5.1k - Mixture-of-models router that picks a model per prompt by intent/complexity; a vLLM project.
- NVIDIA LLM Router ⭐ 336 - NIM-based blueprint routing each prompt to the best model by task and complexity.
- LLMRouter ⭐ 2.3k - Research framework for graph/learned cost–quality model routing.
- Orq.ai - Hosted routing control plane: 500+ models across 30+ providers with retries, fallbacks, caching and governance (BYOK).
- NadirClaw ⭐ 638 - Self-hosted, OpenAI-compatible router (Python) that sends simple prompts to cheap/local models and hard ones to premium, with a trained cascade verifier to cut API cost 40–70%.
- ngrok AI Gateway - Managed proxy routing to OpenAI/Anthropic/Google + local Ollama/vLLM/LM Studio, with automatic failover, key rotation, and CEL traffic-policy controls (PII redaction).
🎯 Pain point: "Anthropic/OpenAI sell a ~90% cache-read discount — do I still get it through a router, and is it even worth it?"
Often no, and the failure is silent. This is one of the most-asked, worst-answered questions in the ecosystem — real users repeatedly discover their cache discount vanished in transit: native_tokens_cached stuck at 0 through OpenRouter in Zed (zed#52576), the OpenRouter AI-SDK provider shipping broken cache options (ai-sdk-provider#35), and a long tail of "caching doesn't work" threads — sometimes it works but the router simply doesn't report it. At production scale, only 28% of LLM calls show any cached input while system prompts eat 69% of input tokens — the largest unclaimed discount in most AI bills.
Verify it in 30 seconds — never trust vibes: send the same long-system-prompt request twice, then diff the usage fields:
OpenAI-style: usage.prompt_tokens_details.cached_tokens > 0 on the 2nd call?
Anthropic-style: usage.cache_read_input_tokens > 0 on the 2nd call?
If the second call shows 0, you're paying full price at best — and if the gateway wrote the cache but never hit it, about a fifth more than full price on Anthropic 5m / OpenAI 5.6+, because the 1.25× write is billed either way (why). Switch the route (provider-direct, or a gateway that passes cache_control through) and re-test.
Two different "caches" — they stack:
- 💵 Provider prompt caching (reads at 0.1× base input — 90% off — but writes cost 1.25–2×, so it only pays above a break-even hit rate: see caching economics) — the gateway must pass through cache headers/params and report back the usage fields. LiteLLM supports Anthropic
cache_controlpassthrough; check any gateway against the test above. - 🗄️ Gateway-side response caching (exact or semantic) — Kong, Bifrost, Zuplo, Cloudflare AI Gateway serve repeated/similar requests from their own cache at ~$0; this stacks on top of provider caching. See the cache column in Quick comparison.
Pain point: "Who spent what, on which model, and why did quality drop?"
🔎 How to evaluate a gateway's observability (table-stakes vs differentiating vs advanced, grounded in the OpenTelemetry GenAI conventions): see BENCHMARKS → Part 6. For the research landscape — theory, seminal papers, company writing, standards & open problems: see the observability survey.
- Helicone ⭐ 6.1k - Logs, costs, sessions, prompt experiments; one-line proxy integration.
- TensorZero ⭐ 11.7k - Archived June 2026
⚠️ (repo read-only; Apache-2.0 code + community forks remain). Gateway + observability + evals in one Rust binary, data stays in your ClickHouse. - Portkey - Full LLMOps suite over its OSS gateway: traces, budgets, prompt management.
- vLLora (ex-LangDB) ⭐ 813 - Agent debugging and observability from the LangDB team.
- Braintrust Proxy ⭐ 408 - Caching proxy wired into Braintrust evals.
- MLflow AI Gateway ⭐ 27.5k - Unified endpoints + governance inside the MLflow platform.
- Respan (ex–Keywords AI) - One endpoint to 250+ models with routing/fallback/caching, plus built-in observability and evals.
- claude-tap ⭐ 3k - MIT local intercepting proxy and trace viewer for coding-agent traffic (Claude Code, Codex CLI, Gemini CLI, Cursor CLI, OpenCode, Kimi, Pi, Hermes): the exact requests, responses and tool calls your agent sends. Wider agent coverage than ccglass; debugging-first rather than spend-first.
- ccglass ⭐ 679 - Local proxy + web dashboard (MIT) that shows exactly what your coding agent (Claude Code, Codex, Kimi) sends to the model.
Pain point: "Routing to self-hosted models (vLLM/Ollama) inside the cluster, GPU-aware."
- Gateway API Inference Extension ⭐ 738 - The Kubernetes standard for inference-aware routing.
- AIBrix ⭐ 5k - Cost-efficient control plane for vLLM on K8s (ByteDance-origin).
- llm-d ⭐ 4k - K8s-native distributed inference serving (Red Hat/Google/IBM-backed).
- Higress ⭐ 9.1k / Kong ⭐ 44k / Envoy AI Gateway ⭐ 1.9k - all implement inference-extension-style routing.
- Traefik Hub AI Gateway - LLM routing/security in Traefik's commercial runtime.
- Inference Gateway ⭐ 230 - Small cloud-native gateway unifying cloud + local (Ollama) providers.
- Olla ⭐ 273 - Lightweight Go proxy + load balancer for LLM infra: intelligent routing and automatic failover across inference backends (Ollama, vLLM, LM Studio, OpenAI-compatible).
- KServe ⭐ 5.8k - The standard model-inference platform on K8s; LLM serving with an inference-gateway / OpenAI-compatible runtime.
- GPUStack ⭐ 5.5k - Manage GPU clusters and serve LLMs behind one OpenAI-compatible endpoint.
- vLLM Production Stack ⭐ 2.5k - Reference K8s stack to serve vLLM at scale with a KV-cache-aware routing layer.
- NVIDIA Dynamo ⭐ 7.7k - NVIDIA's datacenter-scale distributed inference framework whose Endpoint Picker (EPP) plugin for the Gateway API Inference Extension does KV-cache-aware, LLM-aware request routing at the gateway layer over vLLM/SGLang/TensorRT-LLM backends.
- llmaz ⭐ 308 - K8s-native inference platform fronting heterogeneous backends (vLLM, SGLang, TGI, llama.cpp, TensorRT-LLM) with Envoy AI Gateway-based model routing and token rate-limiting, Gateway-API inference-pool routing, and LLM-metric HPA plus Karpenter autoscaling. Maintained but slower cadence (still v0.1.x).
Stars auto-refresh daily. ✅ built-in · ➕ via plugin/paid tier · ❌ not available · n/d = not documented in the project's own docs.
| Project | Type | Stars | License | Multi-provider | Fallback / LB | Caching | Guardrails | Cost tracking | Web UI | Local (Ollama) |
|---|---|---|---|---|---|---|---|---|---|---|
| LiteLLM | OSS proxy + SDK | ⭐ 56.2k | MIT¹ | ✅ 100+ | ✅ | ✅ | ✅ | ✅ | ✅ needs DB | ✅ |
| new-api | OSS relay/billing | ⭐ 45k | AGPL-3.0 | ✅ | ✅ | ➕ | ➕ | ✅ | ✅ | n/d |
| one-api | OSS relay/billing | ⭐ 36.3k | MIT | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
| Kong AI Gateway | OSS API gateway | ⭐ 44k | Apache-2.0 | ✅ | ✅ | 💰 semantic (Ent) | ✅ | ✅ | ✅ Kong Manager | ✅ |
| Apache APISIX | OSS API gateway | ⭐ 17k | Apache-2.0 | ✅ | ✅ | ➕ | ➕ | ➕ | ➕ separate Dashboard | ➕ OpenAI-compat |
| Portkey Gateway | OSS gateway + SaaS | ⭐ 12.7k | MIT | ✅ 1600+ | ✅ | ✅ | ✅ 50+ | ➕ SaaS | ✅ logs console | ✅ |
| TensorZero | OSS LLMOps · |
⭐ 11.7k | Apache-2.0 | ✅ | ✅ | ✅ | ➕ | ✅ | ✅ | ✅ OpenAI-compat |
| Higress | OSS AI-native gateway | ⭐ 9.1k | Apache-2.0 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ console | ✅ |
| GPT-Load | OSS key-pool proxy | ⭐ 6.3k | MIT | ✅ | ✅ key rotation | ❌ | ❌ | ➕ | ✅ | n/d |
| Bifrost | OSS gateway (Go) | ⭐ 7.2k | Apache-2.0 | ✅ | ✅ adaptive | ✅ | ✅ | ✅ | ✅ | ✅ |
| Helicone | OSS observability + gateway | ⭐ 6.1k | Apache-2.0 | ✅ | ✅ | ✅ | ➕ | ✅ | ✅ | ➕ SDK logging |
| Envoy AI Gateway | OSS K8s gateway | ⭐ 1.9k | Apache-2.0 | ✅ | ✅ | ➕ | ➕ | ✅ | ❌ | ➕ OpenAI-format |
| OpenRouter | SaaS marketplace | — | Commercial | ✅ ~340 | ✅ | ✅ | ➕ | ✅ | ✅ | ❌ |
| Vercel AI Gateway | SaaS (0% markup) | — | Commercial | ✅ 100s | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ |
| Cloudflare AI Gateway | SaaS control plane | — | Commercial (free tier) | ✅ | ✅ dynamic | ✅ | ✅ | ✅ budgets | ✅ | n/d |
¹ LiteLLM core is MIT; the repo contains a separately licensed enterprise directory.
💰 = paid tier only · Deploy weight (self-hosted rows, each from its own deploy docs — often the deciding factor on a home server): single process with SQLite/config-file state — Bifrost (npx or Docker, zero-config start), GPT-Load, new-api, one-api (SQLite default; MySQL/Postgres optional), Portkey OSS (Node/npx); needs external state — LiteLLM (Postgres required for virtual keys / budgets / admin UI), Kong (Postgres, or DB-less declarative config), APISIX (etcd, or standalone YAML mode), TensorZero (ClickHouse), Helicone (all-in-one Docker image bundling Postgres + ClickHouse + MinIO), Higress (Docker Compose or K8s). "Local (Ollama)" = the project's own docs name Ollama (or, where marked, a generic OpenAI-compatible/self-hosted backend path).
📂 Browse the raw data (machine-readable, CC0): models & pricing JSON · cost table CSV · gateway scorecard CSV. Every cost cell is regenerated from this data by a unit-tested script.
The full directory at a glance — browse the sections below by your need.
Gateways get bought for nine distinct jobs. Find yours, jump straight to the evidence:
| Your requirement | The question it answers | Where to look |
|---|---|---|
| 🔀 Routing & failover | "One provider went down — did my app?" | Quick comparison · Smart routing |
| 💰 Cost control | "Who can spend what, and where does it stop?" | Cost-first · cost tables · calculator |
| 📊 Observability | "Which key, which model, which prompt — and why did quality drop?" | Observability section · what to measure · research survey |
| 🛡️ Security & compliance | "Can I prove to an auditor where prompts went?" | Enterprise & compliance · scorecard |
| 📦 Supply-chain trust | "Is the gateway itself safe to run?" | How to choose safely (step 8) |
| ⚡ Caching & rate limits | "Stop paying twice for the same answer; survive 429s" | Quick comparison cache column |
| 🏠 Local models on one box (Ollama / LM Studio) | "One proxy on my home server fronting OpenAI + Anthropic + local Ollama" | The Local (Ollama) column in Quick comparison — e.g. LiteLLM, Bifrost, one-api — plus Manifest and Olla |
| ☸️ K8s / GPU cluster | "Route to vLLM/Ollama inside my cluster, GPU-aware" | Kubernetes-native & inference infra |
| 🤖 Agent & MCP governance | "My agents call tools — who's watching that traffic?" | MCP & agent gateways |
| 🔍 Model fidelity / relay trust | "Am I getting the model I'm paying for?" | canary_check.py · watch-list |
How common is each job? Survey-grounded. From the Amplify Partners 2026 AI Engineering Report (1,000+ engineers, with Notion & Vercel): 87% actively run multiple models together — routing is the default architecture, not the edge case (44% route by task type, 11% by cost). 75% adjust how ambitiously they use AI because of cost (40% say it regularly shapes ambition), and cost is the #2 most-monitored production metric after quality — which is why per-key cost attribution anchors the observability axis. 89% of agent-running teams now grant write permissions while guardrails stay primitive — the case for agent/MCP governance. And one number worth reading twice: only 20% put reliability in their top-3 selection criteria — yet peer-reviewed status-page measurements show the OpenAI and Anthropic APIs each fail about every 2 days (median MTBF 1.99 / 2.09 days) with ~1h median recovery, and only 6.15% of incidents get a postmortem (ICPE 2025); in production traffic, rate limits alone caused ~⅓ of all LLM errors (Datadog, Mar 2026). Engineers underweight failover until an event like the June-2026 Fable 5 pullout takes single-provider stacks offline for three weeks. Multi-provider routing is cheap insurance precisely because it's underpriced.
Start by matching the gateway's trust level to your data's sensitivity — this one call decides most of the rest:
| Your data | Route it to | Don't |
|---|---|---|
| 🔴 Secrets / regulated (PII, PHI, financial, source code, keys) | First-party direct + ZDR (Azure / Bedrock / Vertex) or a gateway self-hosted in your VPC | …send it through any third-party relay — full stop |
| 🟡 Internal / business | Compliant hosted (Cloudflare, Vercel, Portkey) or self-hosted (LiteLLM, Bifrost) | …use an unvetted relay; get ZDR in writing |
| 🟢 Low-stakes / public / throwaway (demos, scraped public text) | Cheapest wins — a gray relay can even be economically rational here | …skip the canary test: assume model-swap + data-harvest until you've proven otherwise |
The mistake is using one trust tier for all your traffic. Sensitive prompts through a $0.50/M relay is how keys leak; throwaway prompts through a FedRAMP endpoint is how you overpay 100×. Match the tier to the data.
Then, whatever tier you're in:
- 💰 Check the markup. Marketplaces charge 0–6% — for high volume, self-hosting or 0%-markup gateways (Vercel, Helicone cloud) pay for themselves fast.
- 🔬 Verify model fidelity (canary-diff test). Some relays silently downgrade or quantize models — and quantization spread is normal even on mainstream infrastructure: OpenRouter's own docs list eight quantization levels (int4 → fp32) a provider may serve for the same open-weight model (that's why the
quantizationsfilter exists), and Moonshot's official K2-Vendor-Verifier measures tool-call schema accuracy for the same K2 model ranging from 100% (first-party API) down to ~83% across third-party hosts. Send fixed "canary" prompts — a known-hard reasoning question plus a tokenizer/fingerprint probe — through the gateway and direct to the provider, then diff the outputs —scripts/canary_check.pyautomates exactly this (relay vs. official → a verdict you can attach to a watch-list report). 2026 research found model-identity failures in 45.8% of fingerprint tests (arXiv:2603.01919). Community monitors apiranking.com and rate.linux.do (browser-only) track relay authenticity/stability — usable as signal if you must vet one, but listing there is not endorsement, and this list includes none of them. - 🔒 Mind data flow. Every gateway sees your prompts. For sensitive data: self-host, or require ZDR (zero data retention) in writing.
- ⚖️ License check before embedding. new-api is AGPL-3.0; LiteLLM has an enterprise-licensed directory; "open core" ≠ everything free.
- 🩺 Project health. Star count ≠ maintenance. Check last release date — several once-popular gateways (BricksLLM, Glide, RouteLLM) are effectively unmaintained; this list labels them. Star counts can also be bought: brand-new "router/proxy 2026" repos increasingly show star-inflation patterns, so check the repo's created date against its star curve (and its issue/fork activity) before trusting a star-sorted GitHub search.
- 🚫 Avoid gray-market relays reselling reverse-engineered or stolen-quota access. Beyond account-ban risk, 2026 research caught relays serving poisoned models and exfiltrating planted secrets (Your Agent Is Mine) — and the most-visible relay "rankings" are often paid press releases or carry affiliate links. Account bans and data leaks are your risk, not theirs. Caught one swapping models, harvesting data, or vanishing with your balance? Report it — with evidence — and we'll build the community watch list together.
- 🪤 Watch for the 2026 bait: "unlimited" plans that throttle. The scam vocabulary has moved upmarket — from fake relays to official-but-cheap subscription plans that are quietly speed-throttled into uselessness: "it's unlimited in practice because it's so slow there's no chance you'll spend your quota… the pay-as-you-go API is orders of magnitude faster than the subscription" (r/ClaudeCode on Z.ai's GLM coding plan, echoed here). Before buying any flat-rate plan: benchmark its throughput against the same vendor's pay-as-you-go API — a 10× speed gap is the tell.
- 📦 Treat the gateway itself as supply chain. It sees every prompt and holds every provider key, so its own security posture is a buying criterion: in March 2026 LiteLLM's PyPI releases v1.82.7/.8 were backdoored via a CI-token compromise (quarantined in ~3h), and in June 2026 a LiteLLM RCE chain (CVE-2026-42271) entered CISA's KEV catalog — two different failure modes in one quarter, on the most-deployed OSS gateway. Practical hygiene: pin exact versions (never
latest), watch the project's security advisories + KEV, patch the control plane fast, keep the admin UI off the public internet, and pre-vet any third-party relay with an audit tool like api-relay-audit ⭐ 787 (checks prompt injection, model substitution, tool-call rewriting, SSE anomalies). Per-gateway posture — who signs releases, ships SBOMs, and what got exploited — is in the supply-chain matrix below.
The #1 trust question, with no neutral cross-vendor answer anywhere. Here's one, from primary sources (machine-readable, verified 2026-07-29). Self-hosted gateways are omitted — you own the data plane.
| Hosted gateway | Logs bodies by default | ZDR / no-log mode | Trains on your prompts | Default retention |
|---|---|---|---|---|
| Vercel AI Gateway | ❌ no | ❌ no (own); upstream opt-in filter | 0 days own layer; 30d for ZDR-excluded models | |
| Eden AI | ❌ no | ✅ default | ❌ no (own) | not retained after processing |
| Requesty | ❌ no (bodies) | ✅ default | ❌ no (own); upstream opt-in | 0 days (30d EU if you enable logging) |
| OpenRouter | ❌ no | ✅ opt-in (zdr:true, global/per-req) |
❌ no (own); |
0 days unless you opt into logging |
| Cloudflare AI Gateway | ✅ yes | ✅ opt-in (zdr:true/header) — OpenAI + Anthropic only, silently falls back otherwise |
❌ no (Trust Hub, "without customer consent") | storage-limit based |
| Portkey (cloud) | ✅ yes | ❔ not documented | 3d free · 30d Prod | |
| Helicone (cloud) | ✅ yes (logging proxy) | ❔ not documented (2023-24 legal docs, silent) | 7d Hobby · 1mo Pro · 3mo Team · Forever Ent | |
| Martian | ❔ not documented | ❌ none documented | 🔴 YES — by ToS (licenses your Input Data to train its models) | not documented |
| First-party cloud | Logs bodies by default | ZDR | Trains on your prompts | Default retention |
|---|---|---|---|---|
| AWS Bedrock | inherit, not zero |
data_retention_mode: none) — approval-gated on models that require retention |
❌ no (except opt-in provider_data_share) |
0d only where you set (and are allowed) none |
| Azure OpenAI | ⚙️ only human-review-flagged | ❌ no | conditional ( |
|
| Google Vertex AI | ⚙️ configurable — |
❌ no | 90d abuse logs · 24h cache · 30d under the Advanced-AI addendum · 0d only otherwise | |
| OpenAI (direct API) | ✅ yes (30-day abuse monitoring) | ❌ no (API, since 2023) | 30 days — ✅ the NYT going-forward hold ended 2025-09-26; residual holds + Safety Retention remain |
The two things everyone misses: (1) On every router, your real exposure is the upstream you route to, not the router's own policy — a "ZDR-by-default" router still hits non-ZDR (or training) providers unless you turn on a ZDR-only / no-training filter. (2) "We delete after 30 days" is not always binding — Azure quietly dropped its 30-day commitment, Vertex's Advanced-AI addendum can override ZDR outright, and OpenAI's going-forward court hold (now ended, 2025-09-26) showed how fast a deletion promise can be suspended by litigation. When it matters, get ZDR in the contract, and read the specific upstream endpoint's policy — not just the front-door marketing. Full evidence + source links per row:
data/data_retention.json.
🔑 Your gateway holds every provider key and sees every prompt — it's the highest-value box in your AI stack, and in 2026 attackers stopped brute-forcing it and started shipping it to you pre-backdoored. Posture below is machine-checked from repos, registries and CVE.org (full evidence + incident record, verified 2026-07-28).
Read the advisory column correctly: many published advisories + fast fixes = a working disclosure process, not extra danger. The worst row here is the one with a single unpatched CVE on a dormant repo.
| Gateway | Signed releases | SBOM / provenance | SECURITY.md | 2025–26 advisories (notable) |
|---|---|---|---|---|
| LiteLLM | ✅ cosign (Docker, since v1.83.0) | ❌ | ✅ | 12 — 3 critical (Host-header auth bypass, SQLi in key verification, OIDC cache collision) + the KEV-listed MCP RCE chain; fixes land fast — run ≥1.84.0 |
| new-api | ✅ cosign | ✅ SBOM + checksums | ✅ | 14 — critical billing integer-overflow (self-crediting) & root-token leak; Stripe-webhook bypass; 4× SSRF — patch fast if you sell quota through it |
| 9router | ❌ none found | ❌ | ❌ | 19 — 6 critical, 11 high; 12 in July 2026 alone. Recurring class: unauthenticated access to the proxy and the keys it holds. /api/providers leaking every API key) |
| one-api | ❌ raw binaries, no checksums | ❌ | ❌ | 1 — unpatched XSS; last release Feb 2025 |
| Kong | ✅ cosign (EE images) | ✅ SLSA documented | ✅ | 1 (EE request smuggling, fixed per-branch); OSS repo: none published |
| APISIX | ✅ GPG (ASF) | ASF process | ✅ | 7 — 2026 cluster is auth-plugin bypasses (jwt-auth, openid-connect, cas-auth) — run ≥3.17.0 if an auth plugin fronts your LLM routes |
| Portkey Gateway | ❌ no npm attestations | ❌ | ✅ | 1 (SSRF, fixed 1.14.0) |
| Bifrost | ❌ none found | ❌ | ✅ | 1 (SSRF deny-list gaps, fixed 1.5.16) |
| Higress | ❌ none found | ❌ | ✅ | 0 published — absence of advisories ≠ absence of bugs |
| Helicone ai-gateway | ❔ unknown | ❌ | ❌ | 1 low SSRF — no patched version listed |
Three receipts that make this a buying axis, not paranoia:
- 🧾 2026-03-24 — the real LiteLLM package, backdoored on PyPI (v1.82.7/.8): a credential stealer with
.pthpersistence that ran on every Python start, exfiltrating SSH keys and cloud/CI secrets. Not a typosquat — the attackers stole the project's PyPI tokens through a compromised Trivy in its own CI (the TeamPCP campaign, which chained Trivy → npm worm → Checkmarx Actions → PyPI in one week). Pulled the same day (~40 min per the vendor, ~3 h per NHS England's timeline); the response — Mandiant forensics, rebuilt CI, cosign signing — set the disclosure bar for the category. (Datadog Security Labs) - 🧾 The Shai-Hulud npm worms went after AI tooling camouflage-first: the Nov 2025 wave (796 packages) spoofed commits as the Anthropic Claude Code GitHub App and exfiltrated through a fake
api.anthropic[.]comlookalike; the May 2026 wave hit Mistral AI and Guardrails AI packages, and OpenAI rotated code-signing certs after two employee devices were exposed. If your gateway's dependency tree touches npm, this is your blast radius. (Wiz) - 🧾 Running a public relay is an attack, not just a business model: a 2026 measurement of 428 public relays caught 9 actively injecting malicious code into returned tool calls (two triggered only after 50 calls, or only in YOLO-mode agent sessions), 17 harvesting planted AWS credentials, and one draining a planted ETH key (Your Agent Is Mine) — the same lesson as the package incidents: verify, don't trust (that's what the canary-diff is for).
And one anti-receipt, because evidence cuts both ways: two "critical Kong auth-bypass CVEs" circulating in AI-generated blogspam (CVE-2026-29413/-29414) do not exist at CVE.org — this table counts registry-verified advisories only.
This list tells you which gateway to start with; these two open-source tools — from this list's maintainer (disclosed) — help you prove it behaves before trusting it in production:
- 🧰 llm-gateway-bench (live dashboard) — black-box benchmark for any OpenAI-compatible gateway/relay: TTFT & throughput, success rate, price multiple, plus fidelity probes (model-echo, fake-streaming, usage inflation, context truncation). Test your own gateway with your own key and compare it to the best.
- 🧰 modelprobe - a tiny, dependency-free Go availability prober: point it at a base URL + key and it reports, per model, is it up and how fast. One static binary — drop it in CI or a cron on a $5 VM.
Built on evidence, not hearsay. Newer or unusually cheap relays we've listed but not yet independently fidelity-checked sit here as "vet before use." Run the canary-diff test and report your verdict to move an entry to ✅ verified or ⛔ confirmed-problematic. The script diffs across one or more models in a single pass (--model a,b) and adds a tokenizer/fingerprint probe — system_fingerprint mismatch and prompt_tokens divergence on identical prompts — an independent tell beyond text similarity. A passing canary from a project's own team is logged as self-reported — reaching ✅ verified takes an independent reproduction by someone unaffiliated.
| Relay | Listed in | Status | Why it's here |
|---|---|---|---|
| FlintAPI (repo) | Cost-first | Relay over Chinese LLMs (DeepSeek/Qwen/Kimi/GLM/MiniMax); operator-described smart routing (prompt → best-suited model), not independently verified; model fidelity unconfirmed. | |
| RunAPI | Cost-first | Hosted multi-modal OpenAI-compatible relay (LLM + image/video/audio); runapi.ai/v1 live (401 unauth); active org with CLI/SDKs but the gateway is closed/hosted; self-submitted; model fidelity unconfirmed. |
|
| NovAI | Cost-first | Hosted Chinese-model relay (DeepSeek/Qwen/GLM/Kimi/MiniMax/Doubao/Hunyuan) + Doubao image/video on one endpoint; aiapi-pro.com/v1 live; self-submitted; model fidelity unconfirmed. |
|
| TeamoRouter | Cost-first | Hosted multi-protocol (OpenAI/Anthropic/Gemini) relay over 500+ providers; teamorouter.com/v1 live (401); operator-claimed agentic routing + channel-dilution detection, unverified; model fidelity unconfirmed. |
|
| FlowBar | Cost-first | Resells frontier models (GPT/Claude/Gemini) below OpenRouter, differentiating on payment reach (Alipay/WeChat, USDT/USDC, PayPal, Apple/Google Pay); closed-source; model fidelity unconfirmed. | |
| lxg2it ModelRouter (repo) | Cost-first | Solo-built router reselling Anthropic/OpenAI/Google frontier models at an advertised 0% markup (deposit fee may apply). A canary-diff posted by the project's own side passed (mean sim 1.0 on Opus 4.8); not yet independently reproduced. Public repo is a thin unlicensed stub (committed again in 2026-06) — routing is closed/hosted. | |
| Loop Gateway | Cost-first | Anonymous closed relay (public GitHub repo since removed) reselling 311 frontier models through its own OpenRouter account at a 15% markup, account-less + crypto-only; model fidelity unconfirmed. | |
| nullsink (repo) | Cost-first | Account-less, no-logs, Monero/Bitcoin-only relay proxying OpenAI/Anthropic through the operator's own account at ~10% markup; repo ⭐ 12, model fidelity unconfirmed. | |
| Meshs One | Cost-first | New hosted relay (appears new-api-based) reselling Chinese frontier models (DeepSeek/Qwen/MiniMax) per-token; closed-source, self-submitted, model fidelity unconfirmed. | |
| CoderPlan | Cost-first | China-market relay fronting Claude/GPT/Gemini/DeepSeek/Grok per-token; API base api.coderplan.ai/v1 returns new_api_error (new-api-based), closed-source; model fidelity unconfirmed. |
|
| KeepRouter | Cost-first | Hosted OpenAI+Anthropic-compatible gateway (native /v1/messages) fronting 50+ models, prepaid at-cost with 0% token markup (8%+$0.35 top-up fee); /v1/models endpoint live (OpenAI-format); model fidelity unconfirmed. |
|
| RouterPlex | Cost-first | Hosted relay to 25+ models across 11 providers at vendor list rates; api.routerplex.com/v1 live (LiteLLM-style auth error, so it appears LiteLLM-based); closed-source, self-submitted; model fidelity unconfirmed. |
|
| AI快站 (aifast.club) | Cost-first | Operator-submitted China-market relay (OpenAI+Anthropic compatible); live model list/rates at /api/ratio_config, status at /api/status; closed-source; model fidelity unconfirmed. |
|
| TierUp | Cost-first | Tier-based relay routing through OpenRouter at ~50% of retail (transparently subsidized, solo-built, ~zero production users — all self-disclosed); api.tierup.ai/v1 live (OpenAI-format); resold access, model fidelity unconfirmed. |
|
| OpenPaths (repo) | Cost-first | Hosted multi-provider router (15+ providers, multi-modal) with auto-routing; the "open-source" GitHub repo is a no-code, unlicensed marketing mirror pointing to a third-party platform, so treat as closed/hosted; model fidelity unconfirmed. | |
| ApiFlux | Cost-first | Hosted multi-protocol (OpenAI/Anthropic/Gemini) relay with Claude Code / Codex CLI / OpenCode guides; apiflux.ai/v1 live (401); operator advertises all models at 85% of list price — below-list pricing is itself a signal to weigh; self-submitted; model fidelity unconfirmed. |
|
| AllRouter | Cost-first | Hosted OpenAI+Anthropic relay over ~25 models advertised at providers' official list prices with no markup, plus a free own-GPU GLM/Gemma tier; allrouter.ai/v1 live (401, new_api_error → new-api-based); the submitted repo is a docs mirror, not source; self-submitted; model fidelity unconfirmed. |
Nothing is ⛔ confirmed-problematic yet — that status needs a reproducible canary verdict or a documented incident, never hearsay.
A running digest of fresh model, pricing and gateway evals — newest first, every entry dated and sourced. This is the fast-moving signal layer; for our own reproducible cost tables and model scorecard, see the full evaluation set. Spotted a new eval worth tracking? Add it.
| Date | Category | Finding | Source |
|---|---|---|---|
| 2026-07-29 | 🤖 MCP | One of five MCP gateways has adopted the stateless rewrite, one day after it shipped. Probed via code search on the two markers the 2026-07-28 revision introduces and removes — server/discover and Mcp-Session-Id: agentgateway implements the new discovery RPC (3 hits) and keeps the old handshake, with a written version-negotiation design that bridges pre- and post-rewrite clients and servers; Lunar MCPX, Envoy AI Gateway and Docker MCP Gateway return zero server/discover hits and remain on the session model the spec deleted. If your agent fleet is mixed-version right now, that negotiation layer is the difference between a working proxy and a fallback path. Per-gateway evidence: data/mcp_gateways.json. |
agentgateway MCP README |
| 2026-07-29 | 💰 Pricing | The cheapest model this week is not in our cost table, and the reason is the point. Qwen3.7 Flash (listed 2026-07-27, 1M context, five regions on Alibaba's own model page) is priced by input length: $0.03/$0.13 per 1M below 32K, $0.10/$0.40 to 256K, $0.20/$0.80 above. Run it through our four scenarios and the "cheapest" answer splits: it beats DeepSeek V4-Flash on the generation-heavy report ($0.013 vs $0.028) and the input-heavy summary ($0.011 vs $0.015), and loses on the coding-agent session ($0.025 vs $0.021) and the 1M-token chatbot month ($0.50 vs $0.21) — because DeepSeek is flat across the full 1M window. It stays out of the computed cost tables until Alibaba's billing page carries it: that page still stops at 3.6-Flash, so the figures above are from OpenRouter's machine-readable API, not a vendor price sheet. | OpenRouter API |
| 2026-07-28 | 💰 Pricing | Monthly re-verification of our 13-model pricing snapshot against official provider pages: 12/13 exact. The one failure was a silent retirement, not a reprice — Grok 4 (retired 2026-05-15) kept serving through its old slug at grok-4.3 rates ($1.25/$2.50), 2.4× cheaper than the $3/$15 list price we carried. Gateway-config lesson: pin models by dated slug and watch retirement notices, not just price pages — a retired slug can silently change both the model and the bill. | models.json |
| 2026-07-27 | 💰 Pricing | The API pricing market now spans 194 models across 15 providers — a ~2,500× spread over the full input/output range: cheapest listed input Llama 3.1 8B ($0.02/M) vs priciest listed output Claude Mythos 5 ($50.00/M) (our arithmetic from the site's listed rates); cheapest flagship input is DeepSeek V4 Flash ($0.14/M). | aipricing.guru |
| 2026-07-10 | 🔌 Cross-format | The hardest path, measured across 3 gateways — an Anthropic client (e.g. Claude Code) routed to an OpenAI model, the single most-filed "tool calls break" complaint. Neutral CI runner: LiteLLM v1.91.1 — 3/3 · Bifrost — 3/3 · Portkey OSS — not offered (its /v1/messages is Anthropic-provider-only). LiteLLM + Bifrost both translate cleanly; Portkey OSS doesn't expose the path in header-config self-host. Version note: LiteLLM's /v1/messages transport changed (≤1.57.x Chat Completions → ≥~1.9x + Bifrost use the OpenAI Responses API, which KeyError('created_at')s against a chat-completions-only upstream) — so pin your version. Reproducible: node probe/xformat.mjs --gateway <name> --messages-url <your /v1/messages> (method: 10-minute self-test). |
xformat.json |
| 2026-07-09 | 🆓 Free tiers | Free-tier audit across 11 providers, each row re-verified against the provider's own docs: Google now hides Gemini free-tier limits behind a login; Mistral's free mode trains on your data by default (manual opt-out); Together AI's -free models are gone ($5 minimum prepaid); Kimi was never free ($1 to start). Machine-readable, CI-enforced ≤30-day re-review. |
free_tiers.json |
| 2026-07 | 🔌 Fidelity | First independent protocol-fidelity test — does the gateway relay tool-calls/streaming/usage intact (the #1 real-world failure)? LiteLLM 3/3 · Bifrost 3/3 · Portkey OSS 1/3 — Portkey OSS's custom-host streaming threw an internal error on a clean CI runner (non-streaming fine; hosted product untested). Reproducible: node probe/fidelity.mjs --gateway <name> --gateway-url <your /v1/chat/completions>; --gateway self-test checks the probe itself in one command. |
llm-gateway-bench |
| 2026-07 | ⏱️ Performance | First independent gateway-overhead comparison (same neutral CI runner, mock upstream, no vendor claims): Bifrost 0.62 ms · Portkey OSS 2.65 ms · LiteLLM 5.83 ms added per request. Bifrost's "fastest" claim holds directionally (~10× vs LiteLLM, not the marketed 50×); Portkey's "<1 ms" didn't reproduce on shared CI hardware. Reproducible: node probe/overhead.mjs; PRs add more gateways. |
llm-gateway-bench |
| 2026-07 | 📈 Adoption | Multi-model is now the default architecture — of 1,000+ surveyed AI engineers, 87% actively use multiple models together (44% route by task type, 11% by cost), 75% adjust usage because of cost, and cost is the #2 most-monitored production metric after quality. Only 20% rank reliability top-3 — failover stays underpriced. | Amplify Partners |
| 2026-07-02 | 🛡️ Reliability | Anthropic pulled Fable 5 & Mythos 5 offline globally for ~3 weeks under a US export-control order, then restored them once Commerce lifted it (back on the Claude platform/Code by Jul 2) — a live reminder that single-provider stacks have no fallback, and multi-provider routing is the mitigation. | CNBC |
| 2026-06-23 | 🚀 Gateway | Envoy AI Gateway reached v1.0 (production GA) — the CNCF/Envoy-backed, Kubernetes-native multi-provider data plane (provider failover, token rate-limiting, MCP support) graduates to stable. | Envoy |
| 2026-06 | 📈 Adoption | ChatGPT hit ~900M weekly active users and >2.5B queries/day — demand scaling about as fast as the price spread. | DemandSage |
| 2026-04 | 📡 Telemetry | Production telemetry confirms the survey data — across 1,000+ orgs' live LLM traffic: >70% run 3+ models, OpenAI's share fell 75% → 63% in a year (Gemini +20pp, Claude +23pp), rate limits caused ~⅓ of all LLM errors in March (8.4M), and only 28% of calls show any cached input while system prompts eat 69% of input tokens — routing, failover and caching are measurably underused. | Datadog |
| 2026-01 | 📡 Telemetry | 100 trillion tokens of real gateway traffic analyzed (OpenRouter × a16z): open-weight models reached ~⅓ of token volume; no single OSS model holds >20–25% for long (rapid turnover); and a 10% price cut moves usage only ~0.5–0.7% — quality, not price, drives model switching. | arXiv |
| 2025-12 | 💰 Spend | Enterprise LLM-API spend keeps flipping providers — Anthropic 40% · OpenAI 27% (was 50% in 2023) · Google 21% of $12.5B enterprise model-API spend (n=495; disclosure: Menlo is an Anthropic investor). Provider churn at this scale is the business case against hard-wiring one vendor. | Menlo Ventures |
💸 Same ¥100 (≈ $14.66) — how much can each model read? The ~500× spread, made concrete.
How many input+output tokens ¥100 buys, by model (blended estimate · snapshot 2026-06-21 · aipricing.guru):
| Tier | Model | Tokens / ¥100 | ≈ Chinese chars |
|---|---|---|---|
| 🥇 Rock-bottom | DeepSeek V4 Flash | 35.2M | ~26.4M |
| 🥇 Rock-bottom | GPT-4.1 nano | 29.6M | ~22.2M |
| 🥇 Rock-bottom | GPT-5.4 nano | 10.2M | ~7.7M |
| 💚 Value | GPT-5.4 mini | 2.83M | ~2.1M |
| 💚 Value | DeepSeek V4 Pro | 2.83M | ~2.1M |
| 🧠 Reasoning | o3 | 1.48M | ~1.1M |
| 🏁 Flagship | Gemini 2.5 Pro | 1.31M | ~0.98M |
| 🏁 Flagship | GPT-5.5 | 0.42M | ~0.32M |
| 🏁 Flagship | GPT-5.5 Pro | 0.07M | ~0.05M |
One line: ¥100 reads ~26M Chinese characters on DeepSeek V4 Flash — roughly 52× the Three-Body trilogy — but only ~50K on GPT-5.5 Pro, about one short story. Choosing a model is choosing the scale factor on your money; the Cost-first gateways exist to exploit exactly this spread.
Reverse-chronological, last ~3 months. Older items retire into the chapter or data file that carries their evidence — they are not deleted, just moved out of "new". Item review: 2026-07-29 · last prune: 2026-08-12.
- 2026-08-07 · Cloudflare merged Workers AI and AI Gateway into one wallet — prepaid AI Gateway credits now pay for Workers AI inference under a single balance, and credit-funded frontier models (Kimi K2.6, Kimi K2.7-code, GLM-5.2) get 50 RPM per account per model versus 20 on standard Workers AI billing. Worth noting if you picked Cloudflare specifically as a free control plane in front of your own keys: it is now also a billing layer. (changelog)
- 2026-08-03 · agentgateway bridges both MCP protocol eras — v1.4/v1.4.1 support the 2026-07-28 stateless revision, detecting support via
server/discoverand federating a mix of legacy and new-era MCP servers behind one gateway. That mixed-fleet path is the practical migration route now that sessions, the initialize handshake and SSE resumability are gone. (announcement) - 2026-08-02 · The EU AI Act deadline that drives gateway buying moved by 16 months — the Digital Omnibus (Regulation (EU) 2026/1744, in force 2026-07-27) pushed Annex III high-risk obligations, including the Article 12 event logging most audit-gateway shortlists exist to satisfy, from 2026-08-02 to 2027-12-02 (Annex I embedded systems to 2028-08-02). What started on schedule is Article 50 transparency — AI-interaction disclosure, synthetic-content marking, deepfake labelling — which is mostly an application-layer duty your gateway cannot discharge for you. (sourced in the Enterprise & compliance callout above)
- 2026-07-30 · OpenAI cut GPT-5.6 API prices — Luna $1/$6 → $0.20/$1.20 per M tokens (−80%) and Terra $2.50/$15 → $2/$12 (−20%); Sol unchanged at $5/$30. A frontier-family model at Luna's new rate resets the cheap end of a lot of routing maths — re-run your own numbers before trusting any cost table dated earlier, including the snapshot behind this one. (CNBC · OpenAI)
- 2026-07-28 · MCP went stateless — the biggest protocol break since it shipped. Spec revision 2026-07-28 removes protocol-level sessions and the
Mcp-Session-Idheader, deletes theinitialize/notifications/initializedhandshake (every request now carries its protocol version and capabilities in_meta), adds a mandatoryserver/discoverRPC, replaces the HTTP GET stream andresources/subscribewith a singlesubscriptions/listenPOST stream, and drops SSE resumability (Last-Event-IDand event IDs) — a broken stream now means starting over. Verified against the repo at the tagged commit on 2026-07-29: the newschema/2026-07-28/schema.tscontains zero references toInitializeRequestorsessionId, where 2025-11-25 had four. For anyone running an MCP gateway this is structural: session-affinity routing stops being necessary and stateless requests load-balance freely, but resumability is now the gateway's problem, not the protocol's. (changelog) - 2026-07-28 · The July flagship wave rebased the whole scoreboard — Claude Opus 5 (Jul 24) takes the AA-Index top spot (60.7 on the v4.1 scale) and the SWE-bench Verified lead (96.0), Fable 5 settles in at Arena #1 (1508±6), Kimi K3 (Jul 16, weights on HF Jul 27) becomes the top open-weights model, and GPT-5.6 Sol/Terra/Luna (Jul 9), Grok 4.5 (Jul 8) and Gemini 3.6 Flash (Jul 21) fill out the month. Our scoreboard + cost tables are rebuilt on the v4.1 scale with single-source columns (see
BENCHMARKS.md); the monthly pricing re-verification came back 12/13 exact — see the evaluations digest for the one that broke. - 2026-07 · Stripe is reportedly in talks to acquire OpenRouter at a ~$10B valuation (WSJ, Jul 23 — unconfirmed: talks could still collapse or another buyer could emerge) — ~7.7× the $1.3B mark from its May Series B, and the strongest signal yet in the consolidation trend line below (Portkey→Palo Alto, Helicone→Mintlify, TensorZero shutdown). (TNW, PYMNTS)
- 2026-06 · LiteLLM RCE added to CISA's KEV catalog — CVE-2026-42271 (an MCP command-injection) chains with a Starlette auth-bypass into unauthenticated remote code execution that can reach master keys and provider credentials (KEV-listed Jun 8; further CVEs Jun 16–22). Distinct from March's PyPI supply-chain attack — patch and lock down the gateway control plane. (CSA)
- 2026-06 · Envoy AI Gateway hit v1.0 (Jun 23) — the first production-stable open-source AI gateway built on CNCF Envoy: one API across 16 providers plus a native MCP gateway (backed by Tetrate, Bloomberg, Nutanix, Tencent). (release)
- 2026-06 · Hyperscalers converged on AI-gateway governance — Databricks shipped Unity AI Gateway (smart routing + hard spend caps) at Data+AI Summit, Azure API Management's AI-gateway features reached GA at Build, and AWS extended Bedrock AgentCore Gateway at Summit NY. Runtime governance is now table stakes. (Databricks)
- 2026-06 · Anthropic pulled Fable 5 & Mythos 5 offline globally under a US export-control directive (Jun 12–13), then restored them after the Dept of Commerce lifted the controls (Jun 30) — Fable 5 was back on the Claude platform, Claude.ai and Claude Code by Jul 2. The canonical "this is why you keep multi-provider failover" event of the year. (Fortune, CNBC)
- 2026-06 · GLM-5.2 is the new leading open-weight model — Z.ai's MIT-licensed 744B-param MoE (40B active, 1M context, open-weighted mid-June) tops the open-weight tier of the Artificial Analysis Intelligence Index (score 51), taking the crown from the previous open leaders. (Artificial Analysis)
- 2026-06 · TensorZero shut down — the VC-backed open-source LLMOps gateway ($7.3M seed) archived its repo on June 11 (last commit 2026-06-11,
archived: truevia the GitHub API; the shutdown was reported the next day), as first-party clouds ship native gateway/observability features and squeeze independents. (byteiota) - 2026-06 · NetFoundry launched zero-trust MCP and LLM gateways; Cisco Investments joined its Series A. (PR Newswire)
- 2026-05 · Palo Alto Networks completed its acquisition of Portkey (announced Apr 30, closed May 29), making the AI gateway the control plane for its Prisma AIRS security platform — a sign gateways are becoming core security infrastructure. (Palo Alto Networks)
- 2026-05 · OpenRouter raised a $113M Series B led by CapitalG at a $1.3B valuation — ~8M users, ~100T tokens/month. (TechCrunch)
- Trend · MCP gateways emerged as a distinct category; spend-limit enforcement became table stakes; the EU AI Act drives the compliance bucket, though its high-risk deadline slipped to 2027-12-02; new-api overtook one-api as the most active China-ecosystem relay; and an independent-gateway shakeout is underway — Portkey (→Palo Alto) and Helicone (→Mintlify) acquired, TensorZero shut down, and the consolidation kept going (Katanemo→DigitalOcean, TrueFoundry→Seldon, Langfuse→ClickHouse).
- 2026-08-12 · router-for-me/CLIProxyAPI v7.2.129 — v7.2.129
- 2026-08-12 · archestra-ai/archestra platform-v1.3.32 — platform: v1.3.32
- 2026-08-11 · BerriAI/litellm v1.96.2 — v1.96.2
- 2026-08-11 · tashfeenahmed/freellmapi v0.7.0 — v0.7.0
- 2026-08-11 · yym68686/uni-api v1.7.230 — Release 1.7.230
- 2026-08-11 · RelayPlane/proxy v1.9.43 — v1.9.43
- 2026-08-11 · caozhiyuan/copilot-api v2.1.1 — v2.1.1
- 2026-08-11 · askalf/dario v5.5.9 — v5.5.9
- 2026-08-11 · swobuforge/swobu v1.0.0 — v1.0.0
- 2026-08-11 · ENTERPILOT/GoModel v0.1.74 — v0.1.74
- 2026-08-11 · looplj/axonhub v1.0.0-beta7 — v1.0.0-beta7
- 2026-08-11 · smart-mcp-proxy/mcpproxy-go v0.55.0 — v0.55.0
📖 A short, vetted shelf — every link below was HTTP-checked live (2026-06-15). These are the concepts the comparison tables assume; read them before you commit to a gateway.
What an AI gateway actually is
- The 2026 AI Engineering Report - Amplify Partners (with Notion & Vercel), 2026 — the survey behind the numbers this list assumes: 87% of 1,000+ engineers run multiple models together, 75% are cost-constrained, cost is the #2 production metric, and inference is the most bought (vs built) layer of the stack.
- State of AI Engineering - Datadog, 2026-04 — what the surveys claim, measured: live telemetry from 1,000+ orgs (>70% on 3+ models, rate limits ≈ ⅓ of LLM errors, only 28% of calls cached, system prompts = 69% of input tokens). Telemetry, not opinions — with a self-disclosed customer-base skew.
- State of AI: An Empirical 100-Trillion-Token Study - OpenRouter × a16z, 2026-01 — two years of real multi-provider gateway traffic: open-weight models at ~⅓ of tokens, rapid model turnover, and near-zero price elasticity (quality drives switching). The primary dataset on how routing actually behaves at scale.
- LLM Gateway: The One Decision That Removes 100 AI Engineering Decisions - Latent.Space (swyx), 2025-02 — why one gateway choice collapses routing, caching, observability and guardrails into a single control plane.
- AI Gateway — overview — Cloudflare — first-party docs defining the pattern: one endpoint in front of many providers, with caching, rate limiting, analytics and cost tracking.
- AI Gateway documentation - Kong — how gateway concerns (provider-agnostic routing, PII sanitization, token rate-limiting) map onto mature API-gateway infrastructure.
Routing & fallback
- Routing & load balancing - LiteLLM — cross-provider routing, weighted load balancing and tiered fallbacks from the most-deployed open-source gateway.
- Router architecture (fallbacks & retries) - LiteLLM — how retries-within-group and cross-group fallbacks escalate on 429s and connection errors — the mechanics for judging reliability.
- Load balancing - Portkey — weighted, sticky distribution across providers, models and keys so no single provider becomes a bottleneck.
- FrugalGPT: Using LLMs While Reducing Cost and Improving Performance - Chen, Zaharia & Zou (Stanford), 2023 — the foundational paper behind cost-aware routing: model cascades that try cheap-first and escalate only when needed.
- An Empirical Characterization of Outages and Incidents in Public Services for LLMs - Chu et al. (VU Amsterdam), ICPE 2025 — the peer-reviewed case for failover: across 8 LLM services, a failure lands roughly every 2 days per API (median MTBF 1.99–2.09 days), median recovery ~1h, and only 6.15% of incidents get a postmortem. Their conclusion: treat failure as normal operating procedure.
- LLM inference prices have fallen rapidly but unequally across tasks - Epoch AI, 2025-03 — the economics that make routing rational: the price of a fixed capability level falls 9×–900× per year (median 50×), so yesterday's flagship task is today's budget-tier route.
Semantic caching
- GPTCache documentation - Zilliz — the de-facto open-source semantic cache: embedding + vector-similarity vs. exact-match.
- GPTCache: An Open-Source Semantic Cache for LLM Applications - Fu Bang, EMNLP 2023 — the peer-reviewed case for similarity-matched caching to lift hit rates and cut cost/latency.
Prompt caching (it's a prefix match)
- Prompt caching - Anthropic — the authoritative spec: cache key from exact bytes up to a breakpoint, write/read pricing, and TTLs.
- Prompt caching - OpenAI — cache hits require an exact prefix; put static instructions first and variable content last to maximize reuse.
Reasoning-token cost
- Building with extended thinking - Anthropic — reasoning/thinking tokens are billed and consume the output budget — the economics to grasp before enabling reasoning models behind a gateway.
Security & guardrails
- OWASP Top 10 for LLM Applications - OWASP, 2025 — the standard risk taxonomy; prompt injection is LLM01, the checklist any gateway's guardrails must answer to.
- Design patterns for securing LLM agents against prompt injection - Simon Willison, 2025-06 — six concrete architectural defenses (Dual LLM, Plan-Then-Execute, Action-Selector, …).
- LLM Prompt Injection Prevention Cheat Sheet - OWASP — a defense-in-depth checklist for what a gateway's guardrail layer should implement.
MCP & agent gateways
- Model Context Protocol — specification - The open standard any MCP gateway must speak and govern.
- Building effective agents - Anthropic, 2024 — when to use workflows vs. agents and the composable patterns (routing, orchestrator-workers) the traffic flowing through an agent gateway is made of.
- LLM Powered Autonomous Agents - Lilian Weng, 2023 — the canonical map of agent architecture (planning, memory, tool use) — what an MCP/agent gateway sits in front of and governs.
Observability
- AI Gateway observability - Cloudflare — per-request logs, token usage, cost estimation and OpenTelemetry export across all providers.
- How to monitor your LLM API costs - Helicone — practical cost-per-query tracking and spotting caching / model-downgrade opportunities.
- Your AI Product Needs Evals - Hamel Husain, 2024 — why systematic evals (not vibes) are how you actually catch quality regressions in the request/response data your gateway logs.
Self-hosting economics
- Automatic prefix caching - vLLM — KV-block prefix caching (and per-request cache isolation), the mechanism behind the savings when you self-host behind your own gateway.
In-depth, data-backed comparisons for the questions people actually search:
- 📘 Coding-agent routers (2026) - claude-code-router vs OmniRoute vs 9router vs CLIProxyAPI vs sub2api: savings vs. ban risk
- 📘 LiteLLM vs OpenRouter vs Portkey (2026) - which AI gateway should you use?
- 📘 LiteLLM alternatives (2026) - 8 gateways compared by cost, security & self-hosting
- 📘 OpenRouter alternatives (2026) - 0%-markup, EU-residency & self-hosted options compared
- 📘 Cloudflare vs Vercel AI Gateway (2026) - which 0%-markup hosted gateway?
- 📘 Best self-hosted AI gateway in 2026 - LiteLLM vs Bifrost vs Portkey vs Kong
- 📘 one-api vs new-api vs LiteLLM - Choosing a China-market LLM API gateway (Chinese)
More comparisons coming. Suggest one via an issue.
What is an AI gateway (LLM gateway)? A proxy between your code and LLM providers: one OpenAI-compatible endpoint and key for many models, adding routing, failover, caching, rate limits, cost tracking and guardrails. See the intro.
AI gateway vs LLM router — what's the difference? A router decides which model gets each request (e.g. cheap vs flagship); a gateway is the full proxy layer (auth, caching, observability, guardrails) that usually includes routing. See smart routing.
What's the best open-source AI gateway? LiteLLM is the default for breadth (Python, 100+ providers). For raw performance pick Bifrost (Go); for enterprise K8s pick Kong or Higress. Full list under self-hosted.
LiteLLM vs OpenRouter — which should I use? OpenRouter is hosted (zero ops, ~5.5% fee, ~340 models); LiteLLM is self-hosted (your keys, your infra, $0 markup). Hosted to start, self-host when volume justifies it. Cost math in the evaluation set.
What's the cheapest way to call many LLMs? Two different "cheapest": 0% markup on your own keys (Vercel / Cloudflare AI Gateway — you hold each provider account) vs one balance, no key management (OpenRouter, whose ~5.5% fee buys the catalog + failover). Same split as the 10-second answers table. For lowest token cost, route bulk work to cheap models — a 100K-token report runs $0.03 on DeepSeek vs $3.01 on GPT-5.5. See cost-first.
Are AI gateways safe? Who sees my prompts? Every gateway sees your prompts. For sensitive data self-host or require zero-data-retention in writing; check the gateway scorecard for compliance/security ratings and known CVEs.
Do I even need a gateway? Often not yet. If you use one provider, one wire format, one team's budget, and can tolerate an outage longer than an hour, an SDK's built-in retries plus a fallback library cover it — and you skip a component that sits on 100% of your traffic and ships releases you must patch. The handbook works through the six conditions and the tripwires that flip the answer: the case against a gateway.
Will a gateway retry my failed request? Probably not, unless you configured it. Read at pinned commits across six open-source gateways, exactly one retries at the LLM layer by default — Bifrost, Portkey OSS and new-api all ship a default of 0, Kong OSS has no AI-level retry, and Envoy writes no retry count of its own. Details and per-gateway config: retry vs failover.
Does prompt caching actually save money? Only above a break-even hit rate. Reads cost 0.1× base input, but writes cost 1.25–2×, so caching a prefix you rarely hit again is a loss: the break-even is 21.7% on Anthropic's 5-minute tier and OpenAI 5.6+, 52.6% on Anthropic's 1-hour tier. The formula, and which variable actually dominates it: caching economics.
Will a gateway break my coding agent?
It can, and four of the five ways are silent. Translating between Anthropic and OpenAI wire formats can drop tool-call identity, reshape or buffer the stream, misreport usage, truncate the system prompt, or strip cache_control — the last one bills cached input at 10× with no error anywhere. Each failure mode is anchored to a real issue, with a ten-minute self-test: the five failure modes.
Key terms used in the tables above (click to expand)
| Term | Meaning |
|---|---|
| AI gateway / LLM gateway | a proxy between your app and LLM providers; one endpoint and key for many models. |
| LLM router | the part that decides which model serves each request (cheap vs flagship, by cost or quality). |
| Fallback | automatically retry on another model/provider when the first fails or times out. |
| Load balancing (LB) | spread traffic across keys/providers to dodge rate limits and outages. |
| Semantic caching | return a cached answer when a new prompt is semantically similar to a past one (not just identical). |
| Prompt / cached input | providers bill reused prompt prefixes at a steep discount (≈0.1×); the gateway must not mangle the prefix or the cache misses. |
| Guardrails | input/output checks: prompt-injection detection, PII redaction, content filtering, schema enforcement. |
| Virtual keys | per-user/team keys the gateway issues in front of your real provider keys, with their own budgets and limits. |
| ZDR (zero data retention) | provider/gateway contractually does not store your prompts or completions. |
| BYOK | bring your own key: the gateway uses your provider accounts rather than reselling tokens. |
| Markup | the gateway's fee on top of provider token cost — marketplaces typically 0–6%; resold relays can run far higher, which is why the list prices them per task rather than trusting the label. |
| MCP gateway | governs agent ↔ tool traffic (Model Context Protocol), the agentic counterpart to an LLM gateway. |
| Protocol translation | converting between wire formats (OpenAI ↔ Anthropic ↔ Gemini) on the fly — the #1 source of "gateway broke my agent" bugs; see the compatibility chapter in 📖 Learn. |
| Passthrough | same format in and out, no translation — cheap and hard to get wrong; fidelity risk lives almost entirely on the translated paths. |
| SSE (server-sent events) | the streaming transport all three protocols use; their event shapes differ completely, which is why streams break in translation. |
| TTFT (time to first token) | how long before the first streamed token arrives; buffered "fake streaming" destroys it even when total latency looks fine. |
| Tool / function calling | the model returns a structured "call this function with these args" payload instead of text; agents live on it, and its shape differs per protocol. |
| Thinking / reasoning tokens | hidden chain-of-thought tokens reasoning models emit — billed as output, so a "cheap" reasoning model can out-cost a flagship. |
| Context window | maximum tokens a model accepts per request; several providers double the price above a threshold (e.g. 200K). |
| KV cache | the inference-server memory holding attention state; KV-aware routing sends a follow-up to the replica that already holds your prefix. |
| Cache hit rate | share of input tokens served from cache. Gross savings ≈ hit rate × discount × prefix share — but cache writes cost 1.25–2×, so below a break-even hit rate caching is a net loss. |
| RPM / TPM | requests / tokens per minute — the two axes of provider rate limits; rate limits caused ~⅓ of LLM errors in production telemetry (Datadog, 2026-03). |
| Cascade routing | try a cheap model first, escalate to a stronger one only when the cheap answer fails a check — the classic cost-cut pattern. |
| Circuit breaker | after repeated upstream failures, stop sending traffic there for a cooldown window instead of retrying into a dead provider. |
| Canary test | send fixed probe prompts through a gateway/relay and diff against the official API to detect model swaps or quantization — scripts/canary_check.py automates it. |
| Quantization | serving a model at reduced numeric precision (int4…fp32); cheaper and often fine, but undisclosed quantization is a fidelity problem — OpenRouter exposes it as a filter. |
| Model fidelity | whether the model you're billed for is the model actually answering, at full quality — the thing canary tests and fingerprint probes check. |
| Data plane / control plane | the request path (data) vs the config/admin/analytics layer (control); several "open-source" gateways open-source the data plane and sell the control plane. |
| SPOF (single point of failure) | the gateway itself is one — it sits on 100% of your traffic, which is why its own reliability and supply-chain posture are selection criteria. |
| OTel (OpenTelemetry) / span | the vendor-neutral telemetry standard and its unit of traced work; the GenAI semantic conventions define LLM-specific attributes (still mostly experimental in 2026). |
| SBOM | software bill of materials — a machine-readable dependency inventory; part of the supply-chain posture matrix above. |
| Open core / SSO tax | business model where base features are open and identity/governance (SSO, SCIM, RBAC, audit logs) sit behind the enterprise tier — quantified in the SSO-tax table. |
| AGPL | a copyleft license (used by new-api): network use counts as distribution, so embedding it in a commercial SaaS triggers source-sharing obligations — read before you build on it. |
| Relay / 中转 | a hosted reseller endpoint fronting other people's model access; transparent ones exist, but the category carries model-swap and data-harvest risk — verify before trusting. |
On June 10 I ran Claude Code hard for ~13 hours, and the bill came to ≈ $788. One look at the per-model breakdown told the whole story: the flagship (Fable 5) alone was $617 — 78% of the bill — while the cheap model (Haiku) did 242 real tasks for $1.70. I hadn't done anything clever to rack that up; I'd done the opposite — defaulted every request to the most capable (and most expensive) model because I couldn't be bothered to set up routing.
The fix wasn't "stop using good models." It was route by task — default to a cheap model, escalate to a flagship only when the work is genuinely hard. That's exactly what an AI gateway is for. While I was at it, I couldn't find a single gateway list organized by what you actually need, that scored the options honestly (CVEs and all), and shipped reproducible cost numbers instead of vibes. So I built one — that's this repo.
No vendor money, no affiliate links, CC0. If it saves you one surprise bill, it did its job. ⭐ Star it so the next person mid-$788-day finds it.
Everything behind the tables is machine-readable, CC0, and refreshed by CI — consume it directly (like models.dev, the raw files are the API):
| Dataset | Raw URL | Refresh |
|---|---|---|
| 5-axis gateway scorecard + per-gateway observability evidence | data/gateways_eval.json · CSV |
reviewed ≤30 days (CI-enforced) |
| Model pricing + benchmark snapshot | data/models.json · cost CSV |
reviewed ≤30 days (CI-enforced) |
| Gateway incident/reality check | data/gateway_reality.json |
on change (drift-gated) |
| Data-retention / ZDR / logging posture (per hosted gateway + cloud) | data/data_retention.json |
on policy change |
| Supply-chain security posture (signing/SBOM/advisories per OSS gateway + incident record) | data/supply_chain.json |
reviewed ≤30 days (CI-enforced) |
| MCP & agent gateway capability matrix (transports, spec revision, tool-level authz, secret brokering) | data/mcp_gateways.json |
reviewed ≤30 days (CI-enforced) |
| Verified free-tier / rate-limit table (11 providers + discontinued list) | data/free_tiers.json |
reviewed ≤30 days (CI-enforced) |
| Stars + latest releases for ~80 tracked gateways | data/projects.json · data/releases.json |
daily |
| Measured gateway overhead (Bifrost/Portkey/LiteLLM) | overhead.json |
monthly CI |
| Protocol-fidelity results (tool-calls/streaming/usage relay) | fidelity.json |
monthly CI |
| Multi-source price triangulation (litellm/openrouter/models.dev) | prices.json |
every 6h |
Attribution appreciated, not required. If you build on it, tell us — we'll link you.
Contributions welcome! Please read CONTRIBUTING.md first. Inclusion criteria, in short: the project must be an actual gateway/proxy/router for LLM or agent traffic (not an SDK wrapper or chat UI), publicly available, and active within the last 12 months — or clearly labeled as stale.
Listed here? You're welcome to show the badge — 
CONTRIBUTING.md. It never changes your scores; those follow the rubric and evidence only.
This list lives in the awesome-list ecosystem. If it doesn't have what you need, these well-maintained neighbors might — and the gateways here sit between their tools and the models:
- Awesome-LLMOps - The broader LLMOps landscape (serving, fine-tuning, observability) this list's gateways plug into.
- Awesome-LLM - Models, papers and the wider LLM ecosystem.
- awesome-langchain - LangChain tools and LLM app frameworks that call through these gateways.
- awesome-mcp-servers - MCP servers to put behind the MCP & agent gateways here.
- awesome-free-llm-apis - CC0 list of permanently-free LLM APIs, from the team behind the Manifest router listed here; pairs with our verified free-tier limits table.
Maintain a related list and think this belongs in yours? Open an issue — cross-linking helps every list's readers.