diff --git a/AGENTS.md b/AGENTS.md index 108b5a0..e318f8b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,7 +5,7 @@ Next.js 16 (App Router) + fumadocs site for [bitrouter.ai](https://bitrouter.ai) ## Documentation rules - **English only.** The site no longer ships localized docs — do not create `.zh.md` translation files or reintroduce i18n plumbing. Old `/zh/*` URLs 301 to the English pages via `next.config.ts` redirects. -- Docs are plain Markdown (`.md`), import-free, using only the whitelisted global components — see `docs/CONTRIBUTING.md` for the full authoring contract. +- Docs are **`.mdx`**, import-free, using only the whitelisted global components — see `docs/CONTRIBUTING.md` for the full authoring contract. Write Markdown; the extension is what makes `` / `` / `` actually render (fumadocs-mdx picks its processor by extension, and `.md` silently drops those blocks). - Lint with `pnpm lint:docs` (`scripts/check-docs.mjs`) after editing docs. - **Always use the scripts — never hand-edit generated output.** Anything produced by a generator is regenerated at `prebuild`, so manual edits are silently lost: - API reference (`content/docs/reference//`) — edit `openapi.yaml`, then `pnpm generate:openapi`. diff --git a/cli-overlays/providers.md b/cli-overlays/providers.md index 80504f8..3fe162c 100644 --- a/cli-overlays/providers.md +++ b/cli-overlays/providers.md @@ -3,7 +3,7 @@ title: providers description: Provider management — list the provider catalog, and log in to subscription providers with OAuth. --- -Two ways a provider becomes available: **BYOK** (its API key in the environment — see [BYOK](/docs/gateway-and-routing/byok)) and **`providers login`** (OAuth against subscription providers such as Claude or Codex subscriptions — see [Integrations](/docs/integrations)). `providers list` shows the catalog and which providers are active in the current config. +Two ways a provider becomes available: **BYOK** (its API key in the environment — see [BYOK](/docs/gateway-and-routing/bring-your-own-provider)) and **`providers login`** (OAuth against subscription providers such as Claude or Codex subscriptions — see [Integrations](/docs/integrations)). `providers list` shows the catalog and which providers are active in the current config. ## @providers login diff --git a/content/docs/(guide)/integrations/openrouter.md b/content/_unpublished/integrations/openrouter.md similarity index 100% rename from content/docs/(guide)/integrations/openrouter.md rename to content/_unpublished/integrations/openrouter.md diff --git a/content/_unpublished/integrations/unsloth.mdx b/content/_unpublished/integrations/unsloth.mdx deleted file mode 100644 index 6035767..0000000 --- a/content/_unpublished/integrations/unsloth.mdx +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: Unsloth -description: Serve a model with Unsloth Studio and register its OpenAI-compatible endpoint as a BitRouter provider. ---- -import { Callout } from 'fumadocs-ui/components/callout'; - -[Unsloth](https://unsloth.ai) is best known for fast fine-tuning, and **Unsloth Studio** also serves models locally behind an OpenAI-compatible API — handy for routing a model you just fine-tuned, or any Unsloth GGUF. The server exposes `/v1/chat/completions` (and an Anthropic `/v1/messages` surface), which BitRouter fronts as one provider block. - -## Prerequisites - -- BitRouter installed, with a `bitrouter.yaml` ([scaffold one](/docs/integrations/local-models#scaffold-a-config) with `bitrouter init`). -- Unsloth Studio serving a model: - - ```bash - unsloth run --model unsloth/gemma-3-27b-it-GGUF:UD-Q4_K_XL - ``` - - This starts the server, opens the Studio UI, and **prints your endpoint URL and `sk-unsloth-…` API key** — note both. The port is typically `8000` or `8888`. - -## Add Unsloth to BitRouter - -```yaml -# bitrouter.yaml -providers: - unsloth: - api_base: http://localhost:8000/v1 # use the port Studio printed - api_protocol: - - "*": chat_completions - api_key: ${UNSLOTH_API_KEY} - models: - - id: unsloth/gemma-3-27b-it-GGUF -``` - - -**Unsloth requires a key.** Unlike Ollama / vLLM / LM Studio, Unsloth Studio authenticates every request with an `Authorization: Bearer sk-unsloth-…` header. Export the key it printed and reference it in the block: - -```bash -export UNSLOTH_API_KEY=sk-unsloth-xxxxxxxxxxxx -``` - -`api_key` resolves from the environment at load time. - - -Confirm the exact model id the server reports — it's what goes under `models`: - -```bash -curl http://localhost:8000/v1/models -H "Authorization: Bearer $UNSLOTH_API_KEY" -``` - - -**Port clash with vLLM.** Unsloth Studio and vLLM both default to `:8000`. If you run both, point `api_base` at whichever port Studio actually printed. - - -## Route to it - -```bash -bitrouter route unsloth:unsloth/gemma-3-27b-it-GGUF -``` - -Then [start BitRouter and send a request](/docs/integrations/local-models#start-bitrouter-and-send-a-request). - -## Learn more - -- [Unsloth — API endpoint guide](https://unsloth.ai/docs/basics/api) -- [Unsloth — llama-server & OpenAI endpoint](https://unsloth.ai/docs/basics/inference-and-deployment/llama-server-and-openai-endpoint) -- [Model fallback](/docs/gateway-and-routing/model-fallback) · [Provider selection](/docs/gateway-and-routing/provider-selection) diff --git a/content/docs/(guide)/gateway-and-routing/acp-gateway.md b/content/docs/(guide)/gateway-and-routing/acp-gateway.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/acp-gateway.md rename to content/docs/(guide)/gateway-and-routing/acp-gateway.mdx diff --git a/content/docs/(guide)/gateway-and-routing/advisor.md b/content/docs/(guide)/gateway-and-routing/advisor.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/advisor.md rename to content/docs/(guide)/gateway-and-routing/advisor.mdx diff --git a/content/docs/(guide)/gateway-and-routing/bring-your-own-model.mdx b/content/docs/(guide)/gateway-and-routing/bring-your-own-model.mdx new file mode 100644 index 0000000..fc83a82 --- /dev/null +++ b/content/docs/(guide)/gateway-and-routing/bring-your-own-model.mdx @@ -0,0 +1,105 @@ +--- +title: Bring Your Own Model +description: Put a model you serve yourself — a local runner, a fine-tune, a private GPU cluster — behind BitRouter, and route to it with the same ids, fallbacks, and variants as any hosted model. +--- + +Bring your own model means the weights are yours: a model running on your laptop, a fine-tune you serve on your own GPUs, or an inference endpoint inside your own network. BitRouter fronts it as one more provider, so agents address it by id and never learn the difference between your GPU and a hosted API. + +This is the mirror image of [bring your own provider](/docs/gateway-and-routing/bring-your-own-provider): there, someone else's model on your account; here, your model on whatever hardware you like. + +## The shape: one provider block + +Any endpoint that speaks a protocol BitRouter already talks — OpenAI-compatible `/v1/chat/completions` in the common case — becomes a provider in `bitrouter.yaml`: + +```yaml +# bitrouter.yaml +providers: + my-cluster: # an id you choose + api_base: https://llm.internal.example/v1 + api_key: ${MY_CLUSTER_TOKEN} # omit if the endpoint is keyless + api_protocol: + - "*": chat_completions # upstream wire format + models: + - id: my-team/llama-3.1-8b-finetune # the id your endpoint serves +``` + +| Field | What it is | +| --- | --- | +| `api_base` | Your endpoint's base URL — the part before `/chat/completions`. | +| `api_key` | Optional. Use a `${VAR}` reference; it resolves from the environment at load time, so no secret lands in the committed file. | +| `api_protocol` | The upstream wire format. `chat_completions` for anything OpenAI-compatible (also the inferred default, so the block is optional); `messages`, `generate_content`, and `responses` are the other known values. | +| `models[].id` | The model id **exactly as your endpoint serves it**. This is what you'll route to. | + +Scaffold a starter file with `bitrouter init` (writes `./bitrouter.yaml` with `skip_auth: true`; `-c ` puts it elsewhere), then check it before you ship: + +```bash +bitrouter config validate -c bitrouter.yaml +``` + +## Already running a local model server? + +The common runners each have a page with their port, id conventions, and auth quirks — the provider block is the same shape: + + + + + + + + + +**Most local servers need no key.** Ollama and vLLM accept anonymous loopback requests, so their blocks have no `api_key`. Add one only if you launched the server with auth (e.g. `vllm serve --api-key `) — or if the server requires it, as Unsloth does. + + +## Route to it + +Point the CLI at the model to make it the active route: + +```bash +bitrouter route my-cluster:my-team/llama-3.1-8b-finetune +``` + +Then start the proxy — it listens on `127.0.0.1:4356` by default — and send a normal request: + +```bash +curl http://127.0.0.1:4356/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "my-cluster:my-team/llama-3.1-8b-finetune", + "messages": [{"role": "user", "content": "Hello from my own model"}] + }' +``` + +The provider-qualified form (`my-cluster:`) pins the request to that exact endpoint. The bare model id also works — BitRouter cascades it to whichever active provider declares it, which is what you want when the same model name is served in more than one place. + +## Your model is a first-class routing target + +Once declared, your endpoint is eligible for the rest of the gateway: + +- [**Model fallbacks**](/docs/gateway-and-routing/model-fallback) — put your model first in the `models` list and a hosted model after it. Requests run on your hardware and only spill over when it errors or is saturated. +- [**Virtual models**](/docs/gateway-and-routing/virtual-model) — give that pairing a single name. Self-hosted, the `models` section of `bitrouter.yaml` does it with a `priority` strategy over an ordered `endpoints` list; on Cloud, a saved `@name`. +- [**Provider selection**](/docs/gateway-and-routing/provider-selection) and [**model variants**](/docs/gateway-and-routing/model-variants) rank *among* the providers serving a model, so they matter here once your endpoint is one of several serving the same id — a model only your cluster serves has nothing to re-rank. + +## Compatibility switches + +Most endpoints need nothing beyond `api_base` and a model id. When an upstream is strict about a field, set it per model under `compatibility`: + +```yaml + models: + - id: my-team/llama-3.1-8b-finetune + compatibility: + chat_completions: + token_limit_field: max_completion_tokens # or max_tokens + supports_store: false + supports_stream_options: false +``` + +`token_limit_field` picks the output-token field the upstream expects — current OpenAI models want `max_completion_tokens`, while some older compatible servers still require `max_tokens`. BitRouter otherwise preserves the caller's spelling and translates the semantic limit across the Chat, Messages, Responses, and Generate Content surfaces for you. + +## Serving your model to others + +Everything above is private: your config, your endpoint, your traffic. To make a model you host **discoverable and routable by anyone on the network** — including from BitRouter Cloud — list it in the open registry instead, by opening a PR with a provider manifest. See [Register as a provider](/docs/guides/register-as-a-provider) for the manifest schema, the `models_url` BitRouter polls, and the payment modes. + + +**Cloud can't reach a private endpoint.** BitRouter Cloud routes to registry providers, and there's no per-account way to attach your own endpoint to a Cloud namespace. To use a model only you serve, either run BitRouter yourself in front of it ([self-host](/docs/guides/self-host)) or publish it to the registry. + diff --git a/content/docs/(guide)/gateway-and-routing/byok.md b/content/docs/(guide)/gateway-and-routing/bring-your-own-provider.mdx similarity index 89% rename from content/docs/(guide)/gateway-and-routing/byok.md rename to content/docs/(guide)/gateway-and-routing/bring-your-own-provider.mdx index 306c44e..e93545f 100644 --- a/content/docs/(guide)/gateway-and-routing/byok.md +++ b/content/docs/(guide)/gateway-and-routing/bring-your-own-provider.mdx @@ -1,9 +1,11 @@ --- -title: External providers (BYOK) -description: Bring your own provider key — use your own account, pay the provider directly, and let BitRouter route requests without a rev share or per-token fee. Works self-hosted, locally, and on BitRouter Cloud. +title: Bring Your Own Provider +description: Route through your own provider account — pay the provider directly at list price, with no rev share or per-token fee. Works self-hosted, locally, and on BitRouter Cloud. --- -**BYOK** (bring your own key) routes your requests using your own provider account, not BitRouter's. You pay the provider directly at their list price — BitRouter takes no rev share, adds no per-token fee, and never holds your keys in plaintext. +Bring your own provider — **BYOK** (bring your own key) in the API and CLI — routes your requests using your own provider account, not BitRouter's. You pay the provider directly at their list price — BitRouter takes no rev share, adds no per-token fee, and never holds your keys in plaintext. + +Where [bring your own model](/docs/gateway-and-routing/bring-your-own-model) is about weights you serve yourself, this page is about *someone else's* model reached on *your* account. BYOK works the same way regardless of deployment: diff --git a/content/docs/(guide)/gateway-and-routing/fusion.md b/content/docs/(guide)/gateway-and-routing/fusion.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/fusion.md rename to content/docs/(guide)/gateway-and-routing/fusion.mdx diff --git a/content/docs/(guide)/gateway-and-routing/guardrails.md b/content/docs/(guide)/gateway-and-routing/guardrails.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/guardrails.md rename to content/docs/(guide)/gateway-and-routing/guardrails.mdx diff --git a/content/docs/(guide)/gateway-and-routing/mcp-gateway.md b/content/docs/(guide)/gateway-and-routing/mcp-gateway.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/mcp-gateway.md rename to content/docs/(guide)/gateway-and-routing/mcp-gateway.mdx diff --git a/content/docs/(guide)/gateway-and-routing/meta.json b/content/docs/(guide)/gateway-and-routing/meta.json index 434af5e..5254e50 100644 --- a/content/docs/(guide)/gateway-and-routing/meta.json +++ b/content/docs/(guide)/gateway-and-routing/meta.json @@ -3,24 +3,11 @@ "icon": "Route", "collapsible": false, "pages": [ - "---Models---", - "provider-selection", "model-fallback", + "provider-selection", + "virtual-model", "model-variants", - "presets", - "structured-outputs", - "byok", - "---Tools---", - "mcp-gateway", - "server-tools", - "websearch", - "web-fetch", - "---Agents---", - "acp-gateway", - "subagent", - "advisor", - "fusion", - "---Governance---", - "guardrails" + "bring-your-own-model", + "bring-your-own-provider" ] } diff --git a/content/docs/(guide)/gateway-and-routing/model-fallback.md b/content/docs/(guide)/gateway-and-routing/model-fallback.mdx similarity index 99% rename from content/docs/(guide)/gateway-and-routing/model-fallback.md rename to content/docs/(guide)/gateway-and-routing/model-fallback.mdx index 954c984..7bd21be 100644 --- a/content/docs/(guide)/gateway-and-routing/model-fallback.md +++ b/content/docs/(guide)/gateway-and-routing/model-fallback.mdx @@ -1,5 +1,5 @@ --- -title: Model Fallback +title: Model Fallbacks description: Pass an ordered list of models and BitRouter walks the list when the primary fails. --- diff --git a/content/docs/(guide)/gateway-and-routing/model-variants.md b/content/docs/(guide)/gateway-and-routing/model-variants.mdx similarity index 96% rename from content/docs/(guide)/gateway-and-routing/model-variants.md rename to content/docs/(guide)/gateway-and-routing/model-variants.mdx index 609ecc6..c3df680 100644 --- a/content/docs/(guide)/gateway-and-routing/model-variants.md +++ b/content/docs/(guide)/gateway-and-routing/model-variants.mdx @@ -47,7 +47,7 @@ The suffix behaves identically on `/v1/messages` (Anthropic) and `/v1beta/models The profile affects provider *ranking* only. Everything else keys off the base model id: - [Guardrail](/docs/gateway-and-routing/guardrails) model allowlists/denylists and BYOK rules judge `anthropic/claude-sonnet-4.6:cost` exactly as `anthropic/claude-sonnet-4.6` — a profile can never widen or bypass a policy. -- [BYOK](/docs/gateway-and-routing/byok) providers still rank ahead of platform providers; the profile orders providers *within* each tier. +- [BYOK](/docs/gateway-and-routing/bring-your-own-provider) providers still rank ahead of platform providers; the profile orders providers *within* each tier. - Billing is unchanged — you pay the selected provider's rate for the base model. ## Seeing which profile a request used diff --git a/content/docs/(guide)/gateway-and-routing/presets.md b/content/docs/(guide)/gateway-and-routing/presets.md deleted file mode 100644 index 0966310..0000000 --- a/content/docs/(guide)/gateway-and-routing/presets.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: Presets -description: Save a named @preset per namespace — a reusable bundle of base model, system prompt, params, and routing rules you invoke inline with @name. ---- - -A **preset** is a named, reusable routing configuration you save once on a namespace and invoke inline by putting `@` in the `model` field. Where a [model variant](/docs/gateway-and-routing/model-variants) (`:cost`) only re-ranks providers for one request, a preset can also **substitute the base model**, **prepend a system prompt**, **set default generation params**, and **restrict which providers are eligible** — all behind a single short token. - -Like a variant, the token lives in the `model` string itself, so it needs no body fields and no SDK — it works the same on the OpenAI, Anthropic, and Google surfaces. A request that uses `@fast` looks exactly like any other request; the preset is resolved server-side before routing. - -## Invoking a preset - -Put `@` where you would normally put a model id. The grammar is `@[/][:]`: - -| `model` value | Resolves to | -| --- | --- | -| `@fast` | The preset `fast`; its saved base model and overrides apply. | -| `@fast:cost` | The preset `fast`, with the [`:cost` variant](/docs/gateway-and-routing/model-variants) overriding the preset's own `sort`. | -| `@fast/openai/gpt-5` | The preset `fast`, but routed to `openai/gpt-5` instead of the preset's saved model. | - -A bare model id with no leading `@` — `anthropic/claude-sonnet-4.6` — is untouched and routes exactly as it does today. Presets are purely additive. - -## What a preset can set - -Every field is optional. An empty preset is valid (it just resolves to its base model unchanged). - -| Field | Effect | -| --- | --- | -| `model` | The base model to route to (e.g. `openai/gpt-5-mini`). If omitted, the request must supply a base inline (`@name/`). | -| `system_prompt` | A system prompt applied when the request doesn't already set one. | -| `params` | Default generation params (`temperature`, `max_tokens`, `top_p`, …), merged in for keys the request didn't set. | -| `routing.sort` | A default routing profile (`balanced` / `cost` / `latency` / `throughput`) — the same axes as [model variants](/docs/gateway-and-routing/model-variants). | -| `routing.only` | A provider allow-list. Routing is restricted to these `provider_name`s. | -| `routing.ignore` | A provider deny-list. These providers are dropped from the chain. | - -## Presets are defaults; the request always wins - -A preset supplies *defaults*. Anything the caller sets explicitly on the request takes precedence: - -- **Base model** — an inline `@name/` (or a body that already names a model) overrides the preset's `model`. If neither the preset nor the request supplies a base, the request is rejected `400`. -- **Profile** — an explicit `:profile` suffix overrides the preset's `routing.sort`; with neither, routing is `balanced`. -- **System prompt** — the preset's `system_prompt` is applied only if the request didn't send one. An explicit system message always wins. -- **Params** — preset params are merged key-by-key, and only for keys the request omitted. A `temperature` in the request body beats the preset's. - -## Creating a preset - -Presets are scoped to a namespace. Create them in the console under **Settings → Routing Presets**, or with the management API: - -```bash -curl -X POST https://api.bitrouter.ai/v1/namespaces/{nsid}/routing-presets \ - -H "Authorization: Bearer $BRK_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "name": "fast", - "model": "openai/gpt-5-mini", - "system_prompt": "Be terse.", - "params": { "temperature": 0.1 }, - "routing": { "sort": "latency", "only": ["openai"] } - }' -``` - -Then invoke it from any inference surface: - -```bash -curl http://127.0.0.1:4356/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "@fast", - "messages": [{"role": "user", "content": "Summarize this in one line."}] - }' -``` - -The full CRUD surface — `list`, `get`, `create`, `update`, `delete`, plus `disable`/`enable` — is documented under [Management API](/docs/reference/management/listRoutingPresets). Reading presets needs the `routing_preset:read` scope; creating or changing them needs `routing_preset:write`. - - -**`name` is the `@token`.** A preset name must match `[A-Za-z0-9_-]+` (the same character set the `@name` grammar accepts), so a name like `my-fast_v2` is fine but `my preset` is rejected at create time — a name you could never invoke is never stored. - - -## Enabling and disabling - -A preset can be disabled without deleting it (`POST …/routing-presets/{id}/disable`, re-enable with `/enable`, or toggle it in the console). A disabled preset is treated as if it doesn't exist: invoking its `@name` returns the same `400` as an unknown preset, while the definition is preserved for when you switch it back on. - -## Presets never change authorization - -Resolution happens *before* policy enforcement, and a preset can only ever **narrow** what a key could already do — never widen it: - -- [Guardrail](/docs/gateway-and-routing/guardrails) model allow/deny lists and BYOK rules judge the **resolved base model**, so a preset that substitutes `openai/gpt-5` is checked exactly as if you had asked for `openai/gpt-5` directly. A preset can't smuggle a request past a model denylist. -- `routing.only` / `routing.ignore` can only *remove* providers from the eligible set — they can never add a provider the request wasn't already allowed to reach. [BYOK](/docs/gateway-and-routing/byok) providers still rank ahead of platform ones. -- Billing is unchanged — you pay the selected provider's rate for the resolved base model. - -## Errors - -| Condition | Result | -| --- | --- | -| `@name` is unknown or disabled in the namespace | `400` (distinct from an unknown-model `404`) | -| The preset has no `model` and the request supplied no base | `400` | -| `routing.only` / `routing.ignore` leave no eligible providers | `400` (no providers available under the preset's constraints) | -| At create/update: invalid `name`, a `routing.sort` that isn't a known profile, or a `params` key that collides with a transport control (`model` / `messages` / `stream`) | `400` | - -## Presets vs. model variants - -The two features overlap deliberately — reach for whichever fits: - -- A [**model variant**](/docs/gateway-and-routing/model-variants) (`openai/gpt-4o:cost`) is anonymous and zero-setup: it re-ranks providers along one axis for a single request and nothing else. -- A **preset** (`@fast`) is named and saved: it captures a base model, a prompt, params, and provider constraints once, so callers invoke a tested configuration by name instead of repeating it. - -They compose — `@fast:cost` applies the preset and then overrides its routing profile with the inline variant. diff --git a/content/docs/(guide)/gateway-and-routing/provider-selection.md b/content/docs/(guide)/gateway-and-routing/provider-selection.mdx similarity index 91% rename from content/docs/(guide)/gateway-and-routing/provider-selection.md rename to content/docs/(guide)/gateway-and-routing/provider-selection.mdx index 595fab4..752d195 100644 --- a/content/docs/(guide)/gateway-and-routing/provider-selection.md +++ b/content/docs/(guide)/gateway-and-routing/provider-selection.mdx @@ -37,7 +37,7 @@ The same `provider.sort` field works on `/v1/messages` (Anthropic) and `/v1beta/ ## BYOK providers come first -If you've [added an external key](/docs/gateway-and-routing/byok) for a provider, BitRouter prefers that provider for any model it can serve — ahead of every non-BYOK provider, regardless of `provider.sort`. Your BYOK key bills against your own account at upstream list price with no rev share, and you opted into that provider explicitly; honoring that opt-in by default is the only choice that doesn't surprise you later. +If you've [added an external key](/docs/gateway-and-routing/bring-your-own-provider) for a provider, BitRouter prefers that provider for any model it can serve — ahead of every non-BYOK provider, regardless of `provider.sort`. Your BYOK key bills against your own account at upstream list price with no rev share, and you opted into that provider explicitly; honoring that opt-in by default is the only choice that doesn't surprise you later. Within the BYOK-eligible set, the `provider.sort` policy still applies. So `provider.sort: "latency"` plus BYOK keys for OpenAI and Anthropic ranks those two by TTFT first, and falls back to non-BYOK providers (also ranked by latency) only if both BYOK paths fail. diff --git a/content/docs/(guide)/gateway-and-routing/server-tools.md b/content/docs/(guide)/gateway-and-routing/server-tools.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/server-tools.md rename to content/docs/(guide)/gateway-and-routing/server-tools.mdx diff --git a/content/docs/(guide)/gateway-and-routing/structured-outputs.md b/content/docs/(guide)/gateway-and-routing/structured-outputs.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/structured-outputs.md rename to content/docs/(guide)/gateway-and-routing/structured-outputs.mdx diff --git a/content/docs/(guide)/gateway-and-routing/subagent.md b/content/docs/(guide)/gateway-and-routing/subagent.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/subagent.md rename to content/docs/(guide)/gateway-and-routing/subagent.mdx diff --git a/content/docs/(guide)/gateway-and-routing/virtual-model.mdx b/content/docs/(guide)/gateway-and-routing/virtual-model.mdx new file mode 100644 index 0000000..ed9948b --- /dev/null +++ b/content/docs/(guide)/gateway-and-routing/virtual-model.mdx @@ -0,0 +1,128 @@ +--- +title: Virtual Model +description: Define a named model of your own — a saved bundle of base model, system prompt, params, and routing rules — and invoke it inline with @name. +--- + +A **virtual model** is a model that doesn't exist upstream: a name you define once and invoke inline by putting `@` in the `model` field. Where a [model variant](/docs/gateway-and-routing/model-variants) (`:cost`) only re-ranks providers for one request, a virtual model can also **substitute the base model**, **prepend a system prompt**, **set default generation params**, and **restrict which providers are eligible** — all behind a single short token. + +Like a variant, the token lives in the `model` string itself, so it needs no body fields and no SDK — it works the same on the OpenAI, Anthropic, and Google surfaces. A request that uses `@fast` looks exactly like any other request; the virtual model is resolved server-side before routing. + + +**In the API and the console these are called _routing presets_.** The endpoints (`/v1/namespaces/{nsid}/routing-presets`), scopes (`routing_preset:read` / `:write`), and the `presets` section of `bitrouter.yaml` all keep that name. "Virtual model" is what the thing *is*; "preset" is what the field is called. + + +## Invoking a virtual model + +Put `@` where you would normally put a model id. The grammar is `@[/][:]`: + +| `model` value | Resolves to | +| --- | --- | +| `@fast` | The virtual model `fast`; its saved base model and overrides apply. | +| `@fast:cost` | The virtual model `fast`, with the [`:cost` variant](/docs/gateway-and-routing/model-variants) overriding its own `sort`. | +| `@fast/openai/gpt-5` | The virtual model `fast`, but routed to `openai/gpt-5` instead of its saved model. | + +A bare model id with no leading `@` — `anthropic/claude-sonnet-4.6` — is untouched and routes exactly as it does today. Virtual models are purely additive. + +## What a virtual model can set + +Every field is optional. An empty definition is valid (it just resolves to its base model unchanged). + +| Field | Effect | +| --- | --- | +| `model` | The base model to route to (e.g. `openai/gpt-5-mini`). If omitted, the request must supply a base inline (`@name/`). | +| `system_prompt` | A system prompt applied when the request doesn't already set one. | +| `params` | Default generation params (`temperature`, `max_tokens`, `top_p`, …), merged in for keys the request didn't set. | +| `routing.sort` | A default routing profile (`balanced` / `cost` / `latency` / `throughput`) — the same axes as [model variants](/docs/gateway-and-routing/model-variants). | +| `routing.only` | A provider allow-list. Routing is restricted to these `provider_name`s. | +| `routing.ignore` | A provider deny-list. These providers are dropped from the chain. | + +## Virtual models are defaults; the request always wins + +A virtual model supplies *defaults*. Anything the caller sets explicitly on the request takes precedence: + +- **Base model** — an inline `@name/` (or a body that already names a model) overrides the saved `model`. If neither the definition nor the request supplies a base, the request is rejected `400`. +- **Profile** — an explicit `:profile` suffix overrides `routing.sort`; with neither, routing is `balanced`. +- **System prompt** — `system_prompt` is applied only if the request didn't send one. An explicit system message always wins. +- **Params** — saved params are merged key-by-key, and only for keys the request omitted. A `temperature` in the request body beats the saved one. + +## Defining a virtual model + +Virtual models are scoped to a namespace. Create them in the console under **Settings → Routing Presets**, or with the management API: + +```bash +curl -X POST https://api.bitrouter.ai/v1/namespaces/{nsid}/routing-presets \ + -H "Authorization: Bearer $BRK_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "fast", + "model": "openai/gpt-5-mini", + "system_prompt": "Be terse.", + "params": { "temperature": 0.1 }, + "routing": { "sort": "latency", "only": ["openai"] } + }' +``` + +Then invoke it from any inference surface: + +```bash +curl http://127.0.0.1:4356/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "@fast", + "messages": [{"role": "user", "content": "Summarize this in one line."}] + }' +``` + +The full CRUD surface — `list`, `get`, `create`, `update`, `delete`, plus `disable`/`enable` — is documented under [Management API](/docs/reference/management/listRoutingPresets). Reading needs the `routing_preset:read` scope; creating or changing needs `routing_preset:write`. + + +**`name` is the `@token`.** A name must match `[A-Za-z0-9_-]+` (the same character set the `@name` grammar accepts), so `my-fast_v2` is fine but `my preset` is rejected at create time — a name you could never invoke is never stored. + + +## Enabling and disabling + +A virtual model can be disabled without deleting it (`POST …/routing-presets/{id}/disable`, re-enable with `/enable`, or toggle it in the console). A disabled one is treated as if it doesn't exist: invoking its `@name` returns the same `400` as an unknown name, while the definition is preserved for when you switch it back on. + +## Self-hosted: aliases over your own endpoints + +The `@name` form above is a BitRouter Cloud namespace feature. When you [self-host](/docs/guides/self-host), the `models` section of `bitrouter.yaml` gives you the other half of the idea — a named alias over an ordered list of endpoints, so one model name fails over across providers: + +```yaml +# bitrouter.yaml +models: + smart: + strategy: priority # walk the endpoints in declared order + endpoints: + - provider: anthropic + service_id: claude-sonnet-4-6 + - provider: openai + service_id: gpt-4o +``` + +Requests for `smart` route to Anthropic first and fall back to OpenAI. This composes with [bring your own model](/docs/gateway-and-routing/bring-your-own-model) — list your own endpoint first and a hosted model second, and you get free local inference with a hosted safety net behind one name. + +## Virtual models never change authorization + +Resolution happens *before* policy enforcement, and a virtual model can only ever **narrow** what a key could already do — never widen it: + +- Guardrail model allow/deny lists and BYOK rules judge the **resolved base model**, so substituting `openai/gpt-5` is checked exactly as if you had asked for `openai/gpt-5` directly. A virtual model can't smuggle a request past a model denylist. +- `routing.only` / `routing.ignore` can only *remove* providers from the eligible set — they can never add a provider the request wasn't already allowed to reach. [Your own provider keys](/docs/gateway-and-routing/bring-your-own-provider) still rank ahead of platform ones. +- Billing is unchanged — you pay the selected provider's rate for the resolved base model. + +## Errors + +| Condition | Result | +| --- | --- | +| `@name` is unknown or disabled in the namespace | `400` (distinct from an unknown-model `404`) | +| The definition has no `model` and the request supplied no base | `400` | +| `routing.only` / `routing.ignore` leave no eligible providers | `400` (no providers available under the constraints) | +| At create/update: invalid `name`, a `routing.sort` that isn't a known profile, or a `params` key that collides with a transport control (`model` / `messages` / `stream`) | `400` | + +## Virtual models vs. model variants + +The two features overlap deliberately — reach for whichever fits: + +- A [**model variant**](/docs/gateway-and-routing/model-variants) (`openai/gpt-4o:cost`) is anonymous and zero-setup: it re-ranks providers along one axis for a single request and nothing else. +- A **virtual model** (`@fast`) is named and saved: it captures a base model, a prompt, params, and provider constraints once, so callers invoke a tested configuration by name instead of repeating it. + +They compose — `@fast:cost` resolves the virtual model and then overrides its routing profile with the inline variant. diff --git a/content/docs/(guide)/gateway-and-routing/web-fetch.md b/content/docs/(guide)/gateway-and-routing/web-fetch.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/web-fetch.md rename to content/docs/(guide)/gateway-and-routing/web-fetch.mdx diff --git a/content/docs/(guide)/gateway-and-routing/websearch.md b/content/docs/(guide)/gateway-and-routing/websearch.mdx similarity index 100% rename from content/docs/(guide)/gateway-and-routing/websearch.md rename to content/docs/(guide)/gateway-and-routing/websearch.mdx diff --git a/content/docs/(guide)/guides/build-a-plugin.md b/content/docs/(guide)/guides/build-a-plugin.mdx similarity index 100% rename from content/docs/(guide)/guides/build-a-plugin.md rename to content/docs/(guide)/guides/build-a-plugin.mdx diff --git a/content/docs/(guide)/guides/cloud-api.md b/content/docs/(guide)/guides/cloud-api.mdx similarity index 100% rename from content/docs/(guide)/guides/cloud-api.md rename to content/docs/(guide)/guides/cloud-api.mdx diff --git a/content/docs/(guide)/guides/migrate-from-litellm.md b/content/docs/(guide)/guides/migrate-from-litellm.mdx similarity index 96% rename from content/docs/(guide)/guides/migrate-from-litellm.md rename to content/docs/(guide)/guides/migrate-from-litellm.mdx index c723b7d..c7c4d77 100644 --- a/content/docs/(guide)/guides/migrate-from-litellm.md +++ b/content/docs/(guide)/guides/migrate-from-litellm.mdx @@ -78,7 +78,7 @@ response = client.chat.completions.create( -Fallbacks and provider selection that you'd configure with `litellm.Router` move into BitRouter's [routing presets](/docs/gateway-and-routing/presets) and [model fallback rules](/docs/gateway-and-routing/model-fallback) — declared once, not per call site. +Fallbacks and provider selection that you'd configure with `litellm.Router` move into BitRouter's [virtual models](/docs/gateway-and-routing/virtual-model) and [model fallback rules](/docs/gateway-and-routing/model-fallback) — declared once, not per call site. ### From the LiteLLM Proxy @@ -114,11 +114,11 @@ To skip the local proxy entirely, point clients at `https://api.bitrouter.ai/v1` | LiteLLM concept | BitRouter equivalent | Docs | |---|---|---| -| `model_list` in `config.yaml` | Provider keys + routing presets | [Presets](/docs/gateway-and-routing/presets) | +| `model_list` in `config.yaml` | Provider keys + routing presets | [Virtual models](/docs/gateway-and-routing/virtual-model) | | `router_settings` (retries, fallback) | Model fallback rules | [Model fallback](/docs/gateway-and-routing/model-fallback) | | `routing_strategy` (least-busy, latency) | Provider selection | [Provider selection](/docs/gateway-and-routing/provider-selection) | | `cache` (Redis/DynamoDB backed) | Not built into the proxy — handle in app/edge if needed | — | -| Virtual keys + budgets + admin UI | Workspace keys (cloud); env-var keys (local) | [BYOK](/docs/gateway-and-routing/byok) | +| Virtual keys + budgets + admin UI | Workspace keys (cloud); env-var keys (local) | [BYOK](/docs/gateway-and-routing/bring-your-own-provider) | | Guardrails / PII / content filter | Agent firewall on the proxy hop | [Guardrails](/docs/gateway-and-routing/guardrails) | | Callbacks (Langfuse, Datadog, etc.) | Built-in spend + request logs; OTLP export | [OpenTelemetry](/docs/observability/opentelemetry) | | MCP Gateway | MCP gateway | [MCP](/docs/gateway-and-routing/mcp-gateway) | diff --git a/content/docs/(guide)/guides/migrate-from-openrouter.md b/content/docs/(guide)/guides/migrate-from-openrouter.mdx similarity index 96% rename from content/docs/(guide)/guides/migrate-from-openrouter.md rename to content/docs/(guide)/guides/migrate-from-openrouter.mdx index 4636ba1..8049274 100644 --- a/content/docs/(guide)/guides/migrate-from-openrouter.md +++ b/content/docs/(guide)/guides/migrate-from-openrouter.mdx @@ -105,7 +105,7 @@ If your OpenRouter integration sets any of these headers, you can safely drop th - + diff --git a/content/docs/(guide)/guides/migrate-from-tensorzero.md b/content/docs/(guide)/guides/migrate-from-tensorzero.mdx similarity index 95% rename from content/docs/(guide)/guides/migrate-from-tensorzero.md rename to content/docs/(guide)/guides/migrate-from-tensorzero.mdx index aa44f0e..94025b6 100644 --- a/content/docs/(guide)/guides/migrate-from-tensorzero.md +++ b/content/docs/(guide)/guides/migrate-from-tensorzero.mdx @@ -78,7 +78,7 @@ response = client.chat.completions.create( -A TensorZero **function** (a named prompt template + schema with one or more variants) has no single equivalent — the routing half maps to BitRouter, the templating/variant half stays in your app or moves to a [preset](/docs/gateway-and-routing/presets). See the [feature mapping](#feature-mapping) below. +A TensorZero **function** (a named prompt template + schema with one or more variants) has no single equivalent — the routing half maps to BitRouter, the templating/variant half stays in your app or moves to a [preset](/docs/gateway-and-routing/virtual-model). See the [feature mapping](#feature-mapping) below. ### From the gateway + ClickHouse stack @@ -115,9 +115,9 @@ To skip the local proxy entirely, point clients at `https://api.bitrouter.ai/v1` | TensorZero concept | BitRouter equivalent | Docs | |---|---|---| -| `[models.*]` + `[models.*.providers.*]` in `tensorzero.toml` | Provider keys (auto-detected) + the model registry | [BYOK](/docs/gateway-and-routing/byok), [Models](/docs/overview/supported-models#how-model-ids-work) | -| `[functions.*]` (named prompt + schema) | App-side, or a routing [preset](/docs/gateway-and-routing/presets) | [Presets](/docs/gateway-and-routing/presets) | -| `[functions.*.variants.*]` (per-variant model) | Routing preset variants / model ids | [Presets](/docs/gateway-and-routing/presets) | +| `[models.*]` + `[models.*.providers.*]` in `tensorzero.toml` | Provider keys (auto-detected) + the model registry | [BYOK](/docs/gateway-and-routing/bring-your-own-provider), [Models](/docs/overview/supported-models#how-model-ids-work) | +| `[functions.*]` (named prompt + schema) | App-side, or a routing [preset](/docs/gateway-and-routing/virtual-model) | [Virtual models](/docs/gateway-and-routing/virtual-model) | +| `[functions.*.variants.*]` (per-variant model) | Routing preset variants / model ids | [Virtual models](/docs/gateway-and-routing/virtual-model) | | `routing` / `retries` / `fallbacks` | Model fallback rules | [Model fallback](/docs/gateway-and-routing/model-fallback) | | `load_balancing` across providers | Provider selection | [Provider selection](/docs/gateway-and-routing/provider-selection) | | OpenAI-compatible `/openai/v1` endpoint | OpenAI-compatible `/v1` endpoint | [API Reference](/docs/reference/openai-compatible/createChatCompletion) | @@ -140,7 +140,7 @@ If your workflow depends on that closed feedback loop — collect inferences and **Before migration** - [ ] List the providers and models you actually route through TensorZero (skip the rest) - [ ] Separate gateway use from platform use — are you using evals / optimization / experimentation, or just routing? -- [ ] Note where `[functions.*]` templates and schemas live; plan to keep them app-side or move them to a [preset](/docs/gateway-and-routing/presets) +- [ ] Note where `[functions.*]` templates and schemas live; plan to keep them app-side or move them to a [preset](/docs/gateway-and-routing/virtual-model) - [ ] Decide cloud vs. local (or both — they share the endpoint) diff --git a/content/docs/(guide)/guides/register-as-a-provider.md b/content/docs/(guide)/guides/register-as-a-provider.mdx similarity index 100% rename from content/docs/(guide)/guides/register-as-a-provider.md rename to content/docs/(guide)/guides/register-as-a-provider.mdx diff --git a/content/docs/(guide)/guides/self-host.md b/content/docs/(guide)/guides/self-host.mdx similarity index 100% rename from content/docs/(guide)/guides/self-host.md rename to content/docs/(guide)/guides/self-host.mdx diff --git a/content/docs/(guide)/integrations/claude-code.md b/content/docs/(guide)/integrations/claude-code.mdx similarity index 100% rename from content/docs/(guide)/integrations/claude-code.md rename to content/docs/(guide)/integrations/claude-code.mdx diff --git a/content/docs/(guide)/integrations/claude-subscription.md b/content/docs/(guide)/integrations/claude-subscription.mdx similarity index 100% rename from content/docs/(guide)/integrations/claude-subscription.md rename to content/docs/(guide)/integrations/claude-subscription.mdx diff --git a/content/docs/(guide)/integrations/codex-subscription.md b/content/docs/(guide)/integrations/codex-subscription.mdx similarity index 100% rename from content/docs/(guide)/integrations/codex-subscription.md rename to content/docs/(guide)/integrations/codex-subscription.mdx diff --git a/content/docs/(guide)/integrations/codex.md b/content/docs/(guide)/integrations/codex.mdx similarity index 100% rename from content/docs/(guide)/integrations/codex.md rename to content/docs/(guide)/integrations/codex.mdx diff --git a/content/docs/(guide)/integrations/harnesses.md b/content/docs/(guide)/integrations/harnesses.mdx similarity index 100% rename from content/docs/(guide)/integrations/harnesses.md rename to content/docs/(guide)/integrations/harnesses.mdx diff --git a/content/docs/(guide)/integrations/hermes.md b/content/docs/(guide)/integrations/hermes.mdx similarity index 100% rename from content/docs/(guide)/integrations/hermes.md rename to content/docs/(guide)/integrations/hermes.mdx diff --git a/content/docs/(guide)/integrations/index.md b/content/docs/(guide)/integrations/index.mdx similarity index 96% rename from content/docs/(guide)/integrations/index.md rename to content/docs/(guide)/integrations/index.mdx index 4481d6c..8df6c80 100644 --- a/content/docs/(guide)/integrations/index.md +++ b/content/docs/(guide)/integrations/index.mdx @@ -15,9 +15,9 @@ Where your tokens come from — a subscription you already pay for, an aggregato - + ## Harnesses diff --git a/content/docs/(guide)/integrations/meta.json b/content/docs/(guide)/integrations/meta.json index 881c889..05cd92b 100644 --- a/content/docs/(guide)/integrations/meta.json +++ b/content/docs/(guide)/integrations/meta.json @@ -7,9 +7,9 @@ "models", "claude-subscription", "codex-subscription", - "openrouter", "ollama", "vllm", + "unsloth", "---Harnesses---", "harnesses", "claude-code", diff --git a/content/docs/(guide)/integrations/models.md b/content/docs/(guide)/integrations/models.mdx similarity index 83% rename from content/docs/(guide)/integrations/models.md rename to content/docs/(guide)/integrations/models.mdx index 1da1847..ef5b313 100644 --- a/content/docs/(guide)/integrations/models.md +++ b/content/docs/(guide)/integrations/models.mdx @@ -5,23 +5,25 @@ description: Bring any model source under BitRouter — a Claude or Codex subscr A **model source** is wherever your tokens actually come from. BitRouter puts every source behind one endpoint and one [registry](/docs/overview/supported-models#how-model-ids-work), so an agent addresses models by their `provider/model` id and never sees the difference between a subscription, an aggregator, and a GPU in your closet. -There are three shapes of source, by how you authenticate: +Set one up: - + ## How a source is wired +There are three shapes of source, by how you authenticate: + | Source | Auth | Where it's configured | | --- | --- | --- | | **Subscriptions** (Claude, Codex) | OAuth — your plan's login | `bitrouter providers login ` (no key, no yaml) | -| **Aggregators / hosted** (OpenRouter, …) | Bring-your-own API key | A provider block in `bitrouter.yaml` | -| **Self-hosted** (Ollama, vLLM) | Usually none — loopback | A provider block in `bitrouter.yaml` | +| **Aggregators / hosted** | Bring-your-own API key | A provider block in `bitrouter.yaml` | +| **Self-hosted** (Ollama, vLLM, Unsloth) | Usually none — loopback | A provider block in `bitrouter.yaml` | Subscriptions skip `bitrouter.yaml` entirely: `bitrouter providers login claude-code` or `bitrouter providers login openai-codex` stores a refreshing subscription credential for BitRouter to attach at request time. Everything else is a provider block — an `api_base`, an optional `api_key`, and the models that source serves. @@ -76,7 +78,7 @@ curl http://localhost:4356/v1/chat/completions \ The bare model name also works — BitRouter auto-cascades it to whichever active source declares it. The provider-qualified form (`openrouter:openai/gpt-4o`) pins the request to that exact source. -**Mix sources freely.** Declare a [virtual model](/docs/gateway-and-routing/model-fallback) whose endpoints list a local source first and a hosted one second: requests run on your own hardware for free and fail over to the hosted model on error or overload — one model name, automatic failover. +**Mix sources freely.** Declare a [virtual model](/docs/gateway-and-routing/virtual-model) whose endpoints list a local source first and a hosted one second: requests run on your own hardware for free and fail over to the hosted model on error or overload — one model name, automatic failover. -For the concepts behind this — provider selection, fallback, and registry detection — see [Models](/docs/overview/supported-models#how-model-ids-work). To serve models yourself, see the local-server integrations: [Ollama](/docs/integrations/ollama) and [vLLM](/docs/integrations/vllm). +For the concepts behind this — provider selection, fallback, and registry detection — see [Models](/docs/overview/supported-models#how-model-ids-work). To serve models yourself, see the local-server integrations — [Ollama](/docs/integrations/ollama), [vLLM](/docs/integrations/vllm), and [Unsloth](/docs/integrations/unsloth) — or [bring your own model](/docs/gateway-and-routing/bring-your-own-model) for the general shape. diff --git a/content/docs/(guide)/integrations/ollama.md b/content/docs/(guide)/integrations/ollama.mdx similarity index 100% rename from content/docs/(guide)/integrations/ollama.md rename to content/docs/(guide)/integrations/ollama.mdx diff --git a/content/docs/(guide)/integrations/openclaw.md b/content/docs/(guide)/integrations/openclaw.mdx similarity index 100% rename from content/docs/(guide)/integrations/openclaw.md rename to content/docs/(guide)/integrations/openclaw.mdx diff --git a/content/docs/(guide)/integrations/opencode.md b/content/docs/(guide)/integrations/opencode.mdx similarity index 100% rename from content/docs/(guide)/integrations/opencode.md rename to content/docs/(guide)/integrations/opencode.mdx diff --git a/content/docs/(guide)/integrations/pi.md b/content/docs/(guide)/integrations/pi.mdx similarity index 100% rename from content/docs/(guide)/integrations/pi.md rename to content/docs/(guide)/integrations/pi.mdx diff --git a/content/docs/(guide)/integrations/unsloth.mdx b/content/docs/(guide)/integrations/unsloth.mdx new file mode 100644 index 0000000..a37bad1 --- /dev/null +++ b/content/docs/(guide)/integrations/unsloth.mdx @@ -0,0 +1,72 @@ +--- +title: Unsloth +description: Register a local Unsloth server as a BitRouter provider — run or fine-tune a model on your own machine, served behind an OpenAI-compatible API. +--- + +[Unsloth](https://unsloth.ai) trains and runs open models locally. Its CLI serves whatever model you load behind an OpenAI-compatible API at `http://localhost:8888/v1` — the same port also answers Anthropic `/v1/messages` — so it drops into `bitrouter.yaml` as one provider block. That makes it the shortest path from a model you just fine-tuned to a model your agents can route to. + +## Prerequisites + +- BitRouter installed, with a `bitrouter.yaml` ([scaffold one](/docs/integrations/models#scaffold-a-config) with `bitrouter init`). +- Unsloth installed: + + ```bash + curl -fsSL https://unsloth.ai/install.sh | sh # macOS, Linux, WSL + ``` + +- Unsloth serving a model: + + ```bash + unsloth run --model unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL # default port 8888 + ``` + + This loads the model, opens the Studio UI, and **prints your endpoint URL and API key** — note both. + +## Add Unsloth to BitRouter + +```yaml +# bitrouter.yaml +providers: + unsloth: + api_base: http://localhost:8888/v1 + api_protocol: + - "*": chat_completions + api_key: ${UNSLOTH_API_KEY} + models: + - id: unsloth/gemma-4-26B-A4B-it-GGUF +``` + +The `models` id must match the name Unsloth serves — the model name as it appears in Studio, without the quantization tag you passed to `--model`. Confirm it against the server: + +```bash +curl http://localhost:8888/v1/models -H "Authorization: Bearer $UNSLOTH_API_KEY" +``` + + +**Unsloth requires a key.** Unlike Ollama and vLLM, it authenticates every request with an `Authorization: Bearer sk-unsloth-…` header — the server can load and run models, so it isn't left open. Export the key `unsloth run` printed (or one from **Settings → API** in Studio) and reference it in the block: + +```bash +export UNSLOTH_API_KEY=sk-unsloth-xxxxxxxxxxxx +``` + +`api_key` resolves from the environment at load time, so no secret lands in the committed file. + + + +**Serving with `llama-server` instead?** Unsloth also documents running a GGUF through `llama.cpp`'s `llama-server` directly. That's the same wiring with two changes: point `api_base` at the `--port` you chose, and use the `--alias` you set as the `models` id. + + +## Route to it + +```bash +bitrouter route unsloth:unsloth/gemma-4-26B-A4B-it-GGUF +``` + +Then [start BitRouter and send a request](/docs/integrations/models#start-bitrouter-and-send-a-request). Use the provider-qualified id to pin the request to Unsloth, or the bare model name to let BitRouter cascade. + +## Learn more + +- [Unsloth — API endpoint guide](https://unsloth.ai/docs/basics/api) +- [Unsloth — llama-server & OpenAI endpoint](https://unsloth.ai/docs/basics/inference-and-deployment/llama-server-and-openai-endpoint) +- [Bring your own model](/docs/gateway-and-routing/bring-your-own-model) — the general shape, for any endpoint you serve. +- [Model fallback](/docs/gateway-and-routing/model-fallback) — fail over from your own machine to a hosted model. diff --git a/content/docs/(guide)/integrations/vllm.md b/content/docs/(guide)/integrations/vllm.mdx similarity index 89% rename from content/docs/(guide)/integrations/vllm.md rename to content/docs/(guide)/integrations/vllm.mdx index c5f5304..f9327d7 100644 --- a/content/docs/(guide)/integrations/vllm.md +++ b/content/docs/(guide)/integrations/vllm.mdx @@ -33,10 +33,6 @@ The `models` id must match the name vLLM serves. By default that's the full Hugg **Optional auth.** vLLM is keyless by default. If you launched it with `--api-key ` (or `VLLM_API_KEY`), add `api_key: ${VLLM_API_KEY}` to the provider block — it resolves from the environment at load time. - -**Port clash with Unsloth.** vLLM and Unsloth Studio both default to `:8000`. If you run both, start one on another port (`vllm serve … --port 8001`) and update `api_base`. - - ## Route to it ```bash diff --git a/content/docs/(guide)/observability/evaluation.md b/content/docs/(guide)/observability/evaluation.mdx similarity index 100% rename from content/docs/(guide)/observability/evaluation.md rename to content/docs/(guide)/observability/evaluation.mdx diff --git a/content/docs/(guide)/observability/opentelemetry.md b/content/docs/(guide)/observability/opentelemetry.mdx similarity index 100% rename from content/docs/(guide)/observability/opentelemetry.md rename to content/docs/(guide)/observability/opentelemetry.mdx diff --git a/content/docs/(guide)/observability/tracing.md b/content/docs/(guide)/observability/tracing.mdx similarity index 100% rename from content/docs/(guide)/observability/tracing.md rename to content/docs/(guide)/observability/tracing.mdx diff --git a/content/docs/(guide)/overview/bitrouter-vs-litellm.md b/content/docs/(guide)/overview/bitrouter-vs-litellm.mdx similarity index 100% rename from content/docs/(guide)/overview/bitrouter-vs-litellm.md rename to content/docs/(guide)/overview/bitrouter-vs-litellm.mdx diff --git a/content/docs/(guide)/overview/bitrouter-vs-openrouter.md b/content/docs/(guide)/overview/bitrouter-vs-openrouter.mdx similarity index 100% rename from content/docs/(guide)/overview/bitrouter-vs-openrouter.md rename to content/docs/(guide)/overview/bitrouter-vs-openrouter.mdx diff --git a/content/docs/(guide)/overview/quickstart.md b/content/docs/(guide)/overview/quickstart.mdx similarity index 96% rename from content/docs/(guide)/overview/quickstart.md rename to content/docs/(guide)/overview/quickstart.mdx index 919630c..d76f4fe 100644 --- a/content/docs/(guide)/overview/quickstart.md +++ b/content/docs/(guide)/overview/quickstart.mdx @@ -123,7 +123,7 @@ bitrouter start # Proxy running at http://127.0.0.1:4356 ``` -BitRouter auto-detects any key set in the environment — no config file needed. Any provider whose key is present is immediately available. See [BYOK](/docs/gateway-and-routing/byok) for the full list of recognized variables, or [local & private models](/docs/integrations/models) to point BitRouter at Ollama, vLLM, or LM Studio for free. +BitRouter auto-detects any key set in the environment — no config file needed. Any provider whose key is present is immediately available. See [BYOK](/docs/gateway-and-routing/bring-your-own-provider) for the full list of recognized variables, or [local & private models](/docs/integrations/models) to point BitRouter at Ollama, vLLM, or LM Studio for free. For advanced routing rules, guardrails, or multi-account failover, scaffold a config file: @@ -234,7 +234,7 @@ bitrouter policy lock # forbid programmatic writes again - + diff --git a/content/docs/(guide)/overview/supported-models.md b/content/docs/(guide)/overview/supported-models.mdx similarity index 96% rename from content/docs/(guide)/overview/supported-models.md rename to content/docs/(guide)/overview/supported-models.mdx index 5682bd6..b061dfa 100644 --- a/content/docs/(guide)/overview/supported-models.md +++ b/content/docs/(guide)/overview/supported-models.mdx @@ -3,7 +3,7 @@ title: Supported Models description: The full catalog of models any BitRouter account can call — with pricing, reachable over your own keys or one hosted BitRouter Cloud account. --- -Every model BitRouter can route to is listed below. Reach any of them over your own provider keys ([BYOK](/docs/gateway-and-routing/byok), paid to the providers at their list price) or one [BitRouter Cloud](/docs/overview/quickstart) account — one sign-in, no upstream keys, billed per request with failed requests not charged. Running your own model? See [local & private models](/docs/integrations/models) (free). +Every model BitRouter can route to is listed below. Reach any of them over your own provider keys ([BYOK](/docs/gateway-and-routing/bring-your-own-provider), paid to the providers at their list price) or one [BitRouter Cloud](/docs/overview/quickstart) account — one sign-in, no upstream keys, billed per request with failed requests not charged. Running your own model? See [local & private models](/docs/integrations/models) (free). Prices are USD per **million tokens** — what a BitRouter Cloud request costs today, from the cheapest provider actually serving that model (`—` means no per-token provider is currently serving it). A provider listed in the [registry](https://github.com/bitrouter/bitrouter/tree/main/registry) but not currently reachable isn't priced here, so bringing your own key to one of them can beat these rates. Every model is served by one or more registered providers — membership lives in the public, open-source [registry](https://github.com/bitrouter/bitrouter/tree/main/registry), and anyone can [register a provider](/docs/guides/register-as-a-provider). diff --git a/content/docs/(guide)/overview/what-is-bitrouter.md b/content/docs/(guide)/overview/what-is-bitrouter.mdx similarity index 100% rename from content/docs/(guide)/overview/what-is-bitrouter.md rename to content/docs/(guide)/overview/what-is-bitrouter.mdx diff --git a/content/docs/(guide)/usage/cli/index.mdx b/content/docs/(guide)/usage/cli/index.mdx index e36d184..45a7f63 100644 --- a/content/docs/(guide)/usage/cli/index.mdx +++ b/content/docs/(guide)/usage/cli/index.mdx @@ -18,7 +18,7 @@ Running it bare is always safe: `bitrouter` probes for configured credentials wi | Variable | Effect | | --- | --- | -| `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `OPENROUTER_API_KEY`, `OPENCODE_ZEN_API_KEY` | Zero-config BYOK — auto-enables the provider. See [BYOK](/docs/gateway-and-routing/byok) | +| `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `OPENROUTER_API_KEY`, `OPENCODE_ZEN_API_KEY` | Zero-config BYOK — auto-enables the provider. See [BYOK](/docs/gateway-and-routing/bring-your-own-provider) | | `BITROUTER_API_KEY` | Cloud API key; enables the managed `bitrouter` provider | | `BITROUTER_HOME` | Config discovery override (see above) | | `BITROUTER_OAUTH_AS` | Override the OAuth authorization server for self-hosted Cloud | diff --git a/content/docs/(guide)/usage/cli/providers.mdx b/content/docs/(guide)/usage/cli/providers.mdx index 3cf4658..c968ee8 100644 --- a/content/docs/(guide)/usage/cli/providers.mdx +++ b/content/docs/(guide)/usage/cli/providers.mdx @@ -5,7 +5,7 @@ description: Provider management — list the provider catalog, and log in to su {/* GENERATED by scripts/generate-cli.mjs from .cli-snapshot.json (bitrouter 1.0.0-alpha.26). Edit cli-overlays/providers.md and re-run; do not edit this file. */} -Two ways a provider becomes available: **BYOK** (its API key in the environment — see [BYOK](/docs/gateway-and-routing/byok)) and **`providers login`** (OAuth against subscription providers such as Claude or Codex subscriptions — see [Integrations](/docs/integrations)). `providers list` shows the catalog and which providers are active in the current config. +Two ways a provider becomes available: **BYOK** (its API key in the environment — see [BYOK](/docs/gateway-and-routing/bring-your-own-provider)) and **`providers login`** (OAuth against subscription providers such as Claude or Codex subscriptions — see [Integrations](/docs/integrations)). `providers list` shows the catalog and which providers are active in the current config. ## `bitrouter providers` diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 6967140..3ef42a2 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -10,7 +10,7 @@ The docs site has two **tabs**, and each is a folder marked `"root": true` in its `meta.json`: - **Documentation** — `content/docs/(guide)/`. The parentheses make it a - *folder group*: fumadocs strips it from the URL, so `(guide)/overview/quickstart.md` + *folder group*: fumadocs strips it from the URL, so `(guide)/overview/quickstart.mdx` still publishes at `/docs/overview/quickstart`. It exists only to give the tab something to hang off. - **API Reference** — `content/docs/reference/`, generated from the BitRouter @@ -34,15 +34,26 @@ the `pages` list in `content/docs/(guide)/meta.json`. `usage/` holds the CLI and MCP server references — generated, so don't hand-author those. -## Authoring contract (plain Markdown) +## Authoring contract (import-free MDX) -Pages are plain Markdown (`.md`), not MDX with imports. The build enforces this: +Pages are `.mdx`, but you write plain Markdown in them — no imports, no JSX +beyond the whitelisted components. The build enforces this: 1. **Frontmatter** — every page needs `title:` (and ideally `description:`). 2. **No `import` / `export` lines.** A whitelisted set of components is available globally without imports: `Callout`, `Tabs`/`Tab`, `Cards`/`Card`, and (on the relevant pages) `ModelsTable`, `ProvidersTable`, `CompareTable`, `CalInline`. Any other `` tag fails the check. + +> [!IMPORTANT] +> **The `.mdx` extension is load-bearing — never author a docs page as `.md`.** +> `fumadocs-mdx` chooses its processor from the file extension alone +> (`filePath.endsWith(".mdx") ? "mdx" : "md"`), and no config option overrides +> it. In `.md`, a block like `…` parses as raw HTML and is +> dropped whole — tag *and body* — so the page renders with the content +> silently missing. `pnpm lint:docs` still passes, because the components are +> whitelisted either way. Docs were `.md` until 2026-08; the migration to +> `.mdx` restored the dropped blocks across 41 pages. 3. **Callouts** — prefer GitHub-style `> [!NOTE]` / `> [!WARNING]` blockquotes, or the `` component. 4. **Internal links** are site paths without extensions: `/docs/features/byok`, @@ -52,7 +63,7 @@ Pages are plain Markdown (`.md`), not MDX with imports. The build enforces this: ## Adding a page -1. Create `content/docs/(guide)/
/.md`. +1. Create `content/docs/(guide)/
/.mdx`. 2. Add `` to that section's `meta.json` `pages` list in the position you want it to appear in the nav. 3. Run `pnpm lint:docs` to check the authoring contract. diff --git a/lib/llms-txt.ts b/lib/llms-txt.ts index 1f3de45..713bf99 100644 --- a/lib/llms-txt.ts +++ b/lib/llms-txt.ts @@ -29,15 +29,18 @@ References: - [Agent Skills](https://github.com/bitrouter/agent-skills): Drop-in skills that teach an agent to install and use BitRouter - [BitRouter CLI](https://github.com/bitrouter/bitrouter): \`cargo install bitrouter\` — the Rust binary, setup wizard, and TUI dashboard -## Routing +## Gateway & Routing -- [Model Fallback](${BASE_URL}/docs/gateway-and-routing/model-fallback): Automatic fallback across models when an upstream fails +- [Model Fallbacks](${BASE_URL}/docs/gateway-and-routing/model-fallback): Automatic fallback across models when an upstream fails - [Provider Selection](${BASE_URL}/docs/gateway-and-routing/provider-selection): How models resolve to upstream providers, with cost and performance policies +- [Virtual Model](${BASE_URL}/docs/gateway-and-routing/virtual-model): Define a named model of your own — base model, prompt, params, and routing rules behind \`@name\` +- [Model Variants](${BASE_URL}/docs/gateway-and-routing/model-variants): Append \`:cost\`, \`:latency\`, or \`:throughput\` to a model id to pick a ranking axis inline +- [Bring Your Own Model](${BASE_URL}/docs/gateway-and-routing/bring-your-own-model): Put a model you serve yourself behind BitRouter and route to it like any hosted model +- [Bring Your Own Provider](${BASE_URL}/docs/gateway-and-routing/bring-your-own-provider): Route through your own provider account at list price, with no rev share or per-token fee ## Features - [Workspaces](${BASE_URL}/docs/cloud/workspaces): Per-team routing tables, keys, and observability -- [BYOK](${BASE_URL}/docs/cloud/byok): Bring your own provider keys at zero cost; auto-detected from env vars - [OpenTelemetry](${BASE_URL}/docs/observability/opentelemetry): Self-run OTLP export — traces and metrics of every request, pushed to a Collector, Honeycomb, Grafana, or Datadog - [Cloud Tracing](${BASE_URL}/docs/cloud/tracing): Hosted Activity view — spend, token, and latency KPIs plus a per-request log, nothing to operate - [Guardrails](${BASE_URL}/docs/gateway-and-routing/guardrails): Inspect, warn, redact, or block risky content at the proxy layer diff --git a/lib/mcp/format.test.ts b/lib/mcp/format.test.ts index 481f625..b247eee 100644 --- a/lib/mcp/format.test.ts +++ b/lib/mcp/format.test.ts @@ -41,7 +41,7 @@ describe("formatSearchResults", () => { const raw: RawSearchResult[] = [ { id: "1", url: "/docs/guides/routing/model-fallback", type: "page", content: "Model Fallback" }, { id: "2", url: "/docs/guides/routing/model-fallback#auto", type: "heading", content: "Automatic fallback" }, - { id: "3", url: "/docs/gateway-and-routing/byok", type: "page", content: "BYOK" }, + { id: "3", url: "/docs/gateway-and-routing/bring-your-own-provider", type: "page", content: "BYOK" }, ]; it("dedups by page and caps to the limit", () => { const hits = formatSearchResults(raw, 1); diff --git a/next.config.ts b/next.config.ts index e4b4693..86605bd 100644 --- a/next.config.ts +++ b/next.config.ts @@ -8,8 +8,8 @@ const finalPath = { comparison: "/docs/overview/bitrouter-vs-openrouter", "self-host-vs-cloud": "/docs/overview/quickstart#self-host-or-cloud", // gateway & routing "provider-selection":"/docs/gateway-and-routing/provider-selection","model-fallback":"/docs/gateway-and-routing/model-fallback", - "model-variants":"/docs/gateway-and-routing/model-variants","presets":"/docs/gateway-and-routing/presets", - "structured-outputs":"/docs/gateway-and-routing/structured-outputs","byok":"/docs/gateway-and-routing/byok", + "model-variants":"/docs/gateway-and-routing/model-variants","presets":"/docs/gateway-and-routing/virtual-model", + "structured-outputs":"/docs/gateway-and-routing/structured-outputs","byok":"/docs/gateway-and-routing/bring-your-own-provider", "local-models":"/docs/integrations/models","guardrails":"/docs/gateway-and-routing/guardrails", "observability":"/docs/observability/opentelemetry","opentelemetry":"/docs/observability/opentelemetry", "tracing":"/docs/observability/tracing","telemetry":"/docs/observability/opentelemetry", @@ -45,9 +45,9 @@ pairs.push( ["/docs/features/provider-selection", "/docs/gateway-and-routing/provider-selection"], ["/docs/features/model-fallback", "/docs/gateway-and-routing/model-fallback"], ["/docs/features/model-variants", "/docs/gateway-and-routing/model-variants"], - ["/docs/features/presets", "/docs/gateway-and-routing/presets"], + ["/docs/features/presets", "/docs/gateway-and-routing/virtual-model"], ["/docs/features/structured-outputs", "/docs/gateway-and-routing/structured-outputs"], - ["/docs/features/byok", "/docs/gateway-and-routing/byok"], + ["/docs/features/byok", "/docs/gateway-and-routing/bring-your-own-provider"], // concepts/ section dissolved (2026-07 reorg) → pages land next to their features ["/docs/concepts", "/docs/overview/what-is-bitrouter"], ["/docs/concepts/models", "/docs/overview/supported-models#how-model-ids-work"], @@ -86,7 +86,7 @@ pairs.push( ["/docs/cloud", "/docs/overview/quickstart#self-host-or-cloud"], ["/docs/cloud/overview", "/docs/overview/quickstart#self-host-or-cloud"], ["/docs/cloud/get-started", "/docs/overview/quickstart#self-host-or-cloud"], - ["/docs/cloud/byok", "/docs/gateway-and-routing/byok"], + ["/docs/cloud/byok", "/docs/gateway-and-routing/bring-your-own-provider"], ["/docs/cloud/tracing", "/docs/observability/tracing"], ["/docs/cloud/managed-models", "/docs/overview/supported-models"], ["/docs/cloud/workspaces", "/docs/reference/management/listNamespaces"], @@ -128,9 +128,9 @@ pairs.push( ["/docs/models-and-routing/provider-selection", "/docs/gateway-and-routing/provider-selection"], ["/docs/models-and-routing/model-fallback", "/docs/gateway-and-routing/model-fallback"], ["/docs/models-and-routing/model-variants", "/docs/gateway-and-routing/model-variants"], - ["/docs/models-and-routing/presets", "/docs/gateway-and-routing/presets"], + ["/docs/models-and-routing/presets", "/docs/gateway-and-routing/virtual-model"], ["/docs/models-and-routing/structured-outputs", "/docs/gateway-and-routing/structured-outputs"], - ["/docs/models-and-routing/byok", "/docs/gateway-and-routing/byok"], + ["/docs/models-and-routing/byok", "/docs/gateway-and-routing/bring-your-own-provider"], ["/docs/models-and-routing", "/docs/gateway-and-routing/provider-selection"], // tools/agents pages retitled to name their protocol; features/ dissolved — // guardrails moved, namespaces and payment retired (2026-08) @@ -159,6 +159,15 @@ pairs.push( ["/docs/integrations/parallel", "/docs/gateway-and-routing/websearch"], ["/docs/integrations/firecrawl", "/docs/gateway-and-routing/websearch"], ["/docs/integrations/tavily", "/docs/gateway-and-routing/websearch"], + // gateway pages renamed for what they are, not what the field is called + // (2026-08): presets → virtual model, external providers (BYOK) → bring your + // own provider. The API keeps `routing-presets` and `byok`; the docs don't. + ["/docs/gateway-and-routing/presets", "/docs/gateway-and-routing/virtual-model"], + ["/docs/gateway-and-routing/byok", "/docs/gateway-and-routing/bring-your-own-provider"], + // the OpenRouter page was unpublished (2026-08); the aggregator provider block + // it documented is the worked example on the model-sources page. The + // migrate-from-openrouter guide is unaffected. + ["/docs/integrations/openrouter", "/docs/integrations/models"], ); const docsRedirects = pairs.flatMap(([source, destination]) => [ { source, destination, permanent: true }, diff --git a/scripts/generate-supported-tables.mjs b/scripts/generate-supported-tables.mjs index 3be3944..10ccf49 100644 --- a/scripts/generate-supported-tables.mjs +++ b/scripts/generate-supported-tables.mjs @@ -1,4 +1,4 @@ -// Regenerates the catalog table in `content/docs/(guide)/overview/supported-models.md` +// Regenerates the catalog table in `content/docs/(guide)/overview/supported-models.mdx` // from the committed snapshot. Replaces the Rust `dist-helper registry docs` // generator, which lived in bitrouter/bitrouter and was deleted in #742 when the // docs moved to this repo — leaving these tables frozen and unchecked. @@ -69,7 +69,7 @@ function modelRows() { const TARGETS = [ { - file: "content/docs/(guide)/overview/supported-models.md", + file: "content/docs/(guide)/overview/supported-models.mdx", header: ["Model", "Name", "Context", "Modalities", "Open weights", "Input $/M", "Output $/M"], rows: modelRows, },