From 15058de55777b39c745a51b0b1e499c39193f118 Mon Sep 17 00:00:00 2001 From: Bryan Qiu Date: Tue, 4 Aug 2026 21:32:12 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20routing=20=E2=80=94=20the=20task=5Fv1?= =?UTF-8?q?=20router=20and=20the=20built-in=20judge?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The routing page described a single "external routing API" pointed at task_v0, and framed the built-in judge and the external service as alternatives you pick between. The shipped implementation has two routers and one selector that chooses between them per decision, so the page described neither side accurately. Rewrite it around what routing actually does today: - the two routers — the built-in judge from the server's llm: block (the pure-OSS router) and the Databricks AI Gateway task_v1 router — and the per-decision selector: task_v1 answers when it is configured and every model family the decision involves is AI-Gateway-backed, the judge answers otherwise, and a server with neither has no routing - complete copy-pasteable configs for all four postures: routing off, judge only, task_v1, and both, including that the routing: block is auto-derived from a Databricks provider when omitted and that routing: {provider: none} is the only true off-switch - how to verify: /v1/info smart_routing_sources, /v1/hosts gateway_inference, and the per-family credential rules that decide it - what users see: decision chips and their rationale, the Databricks mark keyed on router_source, per-family degradation to the judge rather than hidden surfaces, the --smart-routing flag, and first-message routing in a bare TUI Also corrects two things the old page got wrong regardless of version: llm: and routing: are read only from the file passed to `omni server -c`, not picked up from ~/.omnigent/config.yaml, and provider: none disables the judge too, not just the external router. Co-authored-by: Isaac --- app/docs/build/routing/page.mdx | 350 ++++++++++++++++++++++++-------- lib/og-manifest.json | 2 +- 2 files changed, 266 insertions(+), 86 deletions(-) diff --git a/app/docs/build/routing/page.mdx b/app/docs/build/routing/page.mdx index 4932111..5a3fd21 100644 --- a/app/docs/build/routing/page.mdx +++ b/app/docs/build/routing/page.mdx @@ -2,7 +2,7 @@ import { pageMeta } from "@/lib/og"; export const metadata = pageMeta( "Smart Routing", - "Let a router pick the best harness and model for each session automatically, using the built-in judge or an external routing API.", + "Let a router pick the best harness and model for each session automatically — with the built-in judge, the Databricks AI Gateway task_v1 router, or both.", { eyebrow: "Build", path: "/docs/build/routing", @@ -26,111 +26,290 @@ fit?", and applies the answer for the rest of the session. style={{ width: "100%", height: "auto", borderRadius: "8px", margin: "1rem 0" }} /> -Two routers are available: +## The two routers -- **Built-in judge**: an LLM call using the server's own `llm:` block. - No extra infrastructure. -- **External routing API**: the server delegates the decision to an external - `routes:select` service that you point it at. +Omnigent ships two routers. They are not alternatives you choose between — a +deployment can have either, both, or neither, and the server picks between them +per decision. -## Enable smart routing +- **The built-in judge** — an LLM call using the server's own `llm:` block. It + names models from the candidate menu it is handed, so it can serve any + harness, on any credential. No extra infrastructure, and no dependency on any + vendor: this is the pure open-source router. +- **The Databricks AI Gateway `task_v1` router** — a hosted routing service the + server calls over the [routing API](#external-routing-api) with + `routing.provider: external`. Its picks are AI Gateway catalog ids, so it can + only serve a harness whose inference actually runs through that gateway. -Smart routing is enabled by configuration alone: +### Which one answers -- Configure a server `llm:` block and the server uses the **built-in judge** backed - by it, with no `routing:` block required. -- Or add a `routing:` block with `provider: external` to delegate to an - **external routing API** (see below). +One selector runs per decision: -Start the server with your config file: +> The `task_v1` router answers when it is configured **and** every model family +> the decision involves is AI-Gateway-backed on the target host. Otherwise the +> built-in judge answers. A server with neither configured has no routing. +Gateway backing selects the **source**, not whether the feature exists. A pane +that is off the gateway still routes — the judge serves it — and the decision +records which router answered. + +| Your setup | Which router answers | What you see | +| ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| `task_v1` configured, **every** family gateway-backed | the AI Gateway `task_v1` router | every surface; each decision chip carries a small Databricks mark | +| `task_v1` configured, one family off the gateway | per decision: the backed family keeps `task_v1`; the off-gateway family — and any decision spanning both — goes to the judge | the same surfaces, nothing hidden; only the judge's chips lack the mark | +| only an `llm:` block | the built-in judge | every surface; no chip carries a mark | +| neither | nobody | no Smart Routing surfaces, and the CLI reports that the server has no routing model | + +A host that reports nothing about gateway backing counts as backed — "unknown" +is not "off the gateway", so a deployment whose hosts cannot yet answer is not +silently downgraded. + +## Configure + +Smart routing is enabled by configuration alone — there is no env var and no +feature flag. The examples below are complete `~/.omnigent/config.yaml` files. + +Point the server at the file explicitly: + +``` +omni server -c ~/.omnigent/config.yaml +``` + +The `-c` matters. `providers:` is picked up from `~/.omnigent/config.yaml` +whether or not you pass it, but **`llm:` and `routing:` are read only from the +file given to `-c`**. An `llm:` block in a config the server was never pointed +at is the most common reason a judge silently fails to appear. + +### Out of the box: routing off + +Providers only, with no `llm:` and no `routing:` block. Nothing routes, and the +Smart Routing options stay hidden. + +```yaml +providers: + anthropic: + kind: key + api_key: ${ANTHROPIC_API_KEY} ``` -omni server -c path/to/config.yaml + +### The built-in judge + +Add an `llm:` block to any provider setup. `model` is the only required field. + +```yaml +providers: + anthropic: + kind: key + api_key: ${ANTHROPIC_API_KEY} + +llm: + model: openai/gpt-5.4 + connection: + base_url: https://api.openai.com/v1 + api_key: ${OPENAI_API_KEY} ``` -With a router configured, users see an **Auto** option in the harness picker; -picking it defers harness + model selection to the router on the first message. -With neither an `llm:` block nor an external `routing:` block, routing stays off -and the Auto option is hidden. +The model id takes either form: provider-prefixed (`openai/gpt-5.4`), or a bare +`databricks-…` catalog id, which the server routes through its Databricks +adapter. Credentials come from either a `connection:` mapping or a Databricks +CLI profile — `connection:` wins when both are present: + +```yaml +llm: + model: databricks-claude-haiku-4-5 + profile: my-workspace +``` -## Configure the external routing API +This is the pure open-source path: no gateway, no external service, and the +judge serves every harness on the host. -To delegate routing to an external service, add a top-level `routing:` block to the -server config with `provider: external`: +### The Databricks AI Gateway `task_v1` router + +A `kind: databricks` provider makes both model families gateway-backed; the +`routing:` block names the router to call. ```yaml +providers: + my-workspace: + default: true + kind: databricks + profile: my-workspace + routing: provider: external - base_url: https://gateway.example.com/ai-gateway/routing/v1 - router_name: task_v0 - # Optional: strip a prefix from catalog model ids before sending them to - # the router (and restore it on the answer). Accepts a string or a list. - model_prefix: databricks- - # Optional auth, see "Authentication" below. - api_key: ${ROUTING_API_KEY} + base_url: https:///ai-gateway/routing/v1 + router_name: task_v1 + profile: my-workspace + model_prefix: + - databricks- + - system.ai. ``` -| Field | Required | Description | -| -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -| `provider` | yes | Must be `external` to select the external routing API. Any other value (or omitting the block) falls back to the built-in judge. | -| `base_url` | yes | Base URL of the routing service. The server appends `/routes:select` to it. | -| `router_name` | yes | The routing strategy the gateway should apply, e.g. `task_v0`. Sent as `route_selector.router_name`. | -| `model_prefix` | no | A prefix (or list of prefixes) stripped from catalog model ids before they are sent to the router, and restored on its answer. Example: `databricks-`. | -| `api_key` | no | Static bearer token. `${ENV}` references are expanded. Takes precedence over `profile`. | -| `profile` | no | Databricks CLI profile. The server mints a fresh bearer per call (OAuth refresh), so a long-lived server never sends an expired token. | +Use `profile:` rather than a static `api_key:` where you can — the server mints +a fresh bearer per call, so a long-lived server never sends a token expired by +the ~1 hour OAuth window. + +Two things worth knowing about this block: -If `base_url` or `router_name` is missing, the external provider is skipped and a -warning is logged: routing stays off rather than failing the server. +- **It is optional on a Databricks provider.** Leave the `routing:` key out + entirely and the server derives the same client from your default Databricks + provider, using that workspace's `/ai-gateway/routing/v1`. Declaring it makes + the target explicit. Note the derivation only fires when the key is absent — + a `routing:` block naming any provider other than `external` or `none` leaves + you with the judge alone. +- **Deleting it does not turn routing off.** Because of that derivation, the + only true off-switch is: -### Authentication + ```yaml + routing: + provider: none + ``` -Auth mirrors the `llm:` block, in precedence order: + This one is absolute: it disables the `task_v1` router *and* the built-in + judge, even if an `llm:` block is present. -1. **`api_key`**: an explicit, provider-agnostic bearer token (`${ENV}` expanded). - Sent as `Authorization: Bearer `. -2. **`profile`**: a Databricks CLI profile. The server resolves a fresh token per - request via the Databricks SDK, so tokens that expire (~1h) are refreshed - automatically. -3. **Neither**: requests are sent unauthenticated. +### Both -## External routing API spec +Configure `llm:` and `routing:` together and you get the full posture: the +`task_v1` router on gateway-backed decisions, the built-in judge everywhere +else. This is what keeps Smart Routing available on a host where only one family +is wired to the gateway — the unbacked family degrades to the judge instead of +losing the feature. -The server calls a single endpoint: +`model_prefix` strips a prefix from catalog model ids before they are sent to +the router and restores it on the answer, so a catalog full of +`databricks-claude-…` ids can talk to a router that knows them by their bare +names. + +## Verify your setup + +Ask the server which routers it has: + +``` +curl -s localhost:50151/v1/info | jq '{smart_routing_enabled, smart_routing_sources}' +``` + +`smart_routing_sources` reports the two sides independently — `external` is the +AI Gateway `task_v1` router, `oss` is the built-in judge. + +Then ask whether the host can actually run the gateway router's picks: + +``` +curl -s localhost:50151/v1/hosts | jq '.hosts[0] | {status, gateway_inference}' +``` + +`gateway_inference` is a per-family map, reported by each host at registration. +A family reported `false` cannot run `task_v1`'s picks, so its decisions fall to +the judge. A family missing from the map could not be evaluated, and counts as +backed — the map distinguishes "not backed" from "could not tell". Two families +are evaluated, Claude and Codex; anything else is always unknown. + +What makes a family gateway-backed: + +| Family | Gateway-backed when | Not backed | +| ---------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | +| **Claude** | the launch pins `ANTHROPIC_BASE_URL` **and** delivers its token via `apiKeyHelper` | Bedrock (`ANTHROPIC_BEDROCK_BASE_URL`, no helper), or a Claude Pro/Max subscription login | +| **Codex** | the launch resolves an AI Gateway Codex base URL (a gateway URL ending `/codex/v1`) | a ChatGPT subscription login, or any non-gateway base URL | + +Subscriptions and Bedrock are never gateway-backed. That is the most common +reason a decision comes back from the judge on an otherwise Databricks-backed +server. + +## What you see + +**Decision chips.** Every routing decision renders a chip under the message +showing the harness and model it picked and the router's own rationale — the +rule trace behind the choice ("trivial task → cheapest arm"). Chips appear for +the session's own decision and for each routed subagent spawn. A session's gear +carries one in-session routing control, **Subagent routing**, which decides +whether spawns route or run on the harness default; the session's own routing is +decided once, when it is created. + +**Which router answered.** A chip carries a small Databricks mark when the +`task_v1` router answered. No mark means the built-in judge answered. Pickers +are never branded — only decisions. The exact answer is in the raw verdict JSON +on the routing card, under `router_source`: `"databricks-aigw"` or `"oss-llm"`. + +**Partial credentials degrade, they don't hide.** If only one family is +gateway-backed, every surface stays. The unbacked family's decisions — and any +decision spanning both families — come from the judge, and their chips carry no +mark. The CLI says so once per launch, in one line: _"<harness> is not +AI-Gateway-backed on this host — routing with the built-in router instead."_ +Surfaces disappear only when the server has no judge either, and then the error +is specific: _"not AI-Gateway-backed"_ is a credential problem on your machine, +_"the server has no routing model configured"_ is a server problem. + +**From the CLI.** `--smart-routing` routes a launch: + +``` +omni run --smart-routing -p "review the last commit" +omni run --harness claude-native --smart-routing -p "fix the flaky test" +``` + +Routing happens when the session is created, so `--smart-routing` starts a new +session and cannot be combined with `--resume` or `--continue`. + +**First-message routing in a bare TUI.** Native harnesses that hook their own +first typed message can be launched with no prompt at all: + +``` +omni claude --smart-routing +omni codex --smart-routing +``` + +The TUI opens unrouted. Type your first message and the harness routes it in +place: the prompt is held, a notice reads _"Smart Routing selected <model>; +rerunning your message on it."_, the model switch is applied while nothing is +running, and your message is replayed on it. The switch is scoped to the +session, so your global CLI defaults are never touched, and the replay does not +duplicate the message in the session. Later messages do not re-route — the +decision is per-session. + +## External routing API + +`routing.provider: external` calls a single endpoint: ``` POST /routes:select Content-Type: application/json ``` -The request and response bodies follow the `omnigent.api.routing.v1` schema -(proto3, serialized as JSON with snake_case field names). This schema is versioned -independently of any gateway so the contract can evolve without coupling to a -gateway's release cycle. +Request and response bodies follow the `omnigent.api.routing.v1` schema (proto3, +serialized as JSON with snake_case field names). The schema is versioned +independently of any gateway, so the contract can evolve without coupling to a +gateway's release cycle — `task_v1` is one router served over it, not the +protocol itself. -### Request: `SelectRouteRequest` +### Configuration fields -| Field | Type | Description | -| ----------------- | ----------------- | ---------------------------------------------------------------------------------------------- | -| `route_options` | `RouteOption[]` | Candidate destinations the router may choose from. One entry per (model, harness) pair. | -| `task` | `Task` | The unit of work to route. Carries the user's `prompt` (truncated to 4000 chars). | -| `route_selector` | `RouteSelector` | The routing strategy to apply. Required in practice; a gateway rejects a request that omits it. | -| `session_history` | `SessionHistory` | Prior turns in the session, when available. Routers may use it to keep turns consistent. | +| Field | Required | Description | +| -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | +| `provider` | yes | `external` selects the routing API. `none` disables routing entirely. Any other value leaves only the built-in judge. | +| `base_url` | yes | Base URL of the routing service. The server appends `/routes:select`. | +| `router_name` | yes | The routing strategy to apply, e.g. `task_v1`. Sent as `route_selector.router_name`. | +| `model_prefix` | no | A prefix, or list of prefixes, stripped from catalog model ids before they are sent and restored on the answer. | +| `selection_model` | no | Pins the model the router uses for its own selection call. Sent as `route_selector.config.model`; omitted leaves the router's default. | +| `api_key` | no | Static bearer token. `${ENV}` references are expanded. Takes precedence over `profile`. | +| `profile` | no | Databricks CLI profile. The server mints a fresh bearer per call (OAuth refresh), so a long-lived server never sends an expired token. | -**`RouteOption`** +If `base_url` or `router_name` is missing, the external side is skipped with a +warning rather than failing the server — the built-in judge, if configured, +carries on alone. -| Field | Type | Description | -| --------- | -------- | ----------------------------------------------------------------------------------------------- | -| `model` | `string` | Model id to serve the request, e.g. `gpt-5-5`. | -| `harness` | `string` | Harness that drives the model. May be omitted for a native harness; required for a meta-harness. | +### Request: `SelectRouteRequest` -**`RouteSelector`** +| Field | Type | Description | +| ----------------- | ---------------- | ------------------------------------------------------------------------------------------------- | +| `route_options` | `RouteOption[]` | Candidate destinations the router may choose from. One entry per (model, harness) pair. | +| `task` | `Task` | The unit of work to route. Carries the user's `prompt` (truncated to 4000 chars). | +| `route_selector` | `RouteSelector` | The routing strategy to apply. Required in practice; a gateway rejects a request that omits it. | +| `session_history` | `SessionHistory` | Prior turns in the session, when available. Routers may use it to keep turns consistent. | -| Field | Type | Description | -| ------------- | -------- | ------------------------------------------------------------------------------------------------- | -| `router_name` | `string` | Name of the routing strategy to invoke (from the `router_name` config field). | -| `config` | `Struct` | Optional router-specific configuration, interpreted by the selected router (gateway-defined). | +**`RouteOption`** — `model` (string, e.g. `gpt-5-5`) and `harness` (string; may +be omitted for a native harness, required for a meta-harness). -Example request body: +**`RouteSelector`** — `router_name` (string, from the config field) and an +optional `config` (`Struct`) interpreted by the selected router. ```json { @@ -140,25 +319,19 @@ Example request body: { "model": "gpt-5-4-mini", "harness": "pi" } ], "task": { "prompt": "Refactor the auth module and add tests" }, - "route_selector": { "router_name": "task_v0" } + "route_selector": { "router_name": "task_v1" } } ``` ### Response: `SelectRouteResponse` -| Field | Type | Description | -| ----------------- | ------------------- | -------------------------------------------------------------- | -| `route_selection` | `RouteSelection[]` | The routing decision(s). The server uses the first entry. | -| `rationale` | `string` | Human-readable explanation of why this route was selected. | - -**`RouteSelection`** +| Field | Type | Description | +| ----------------- | ------------------ | ------------------------------------------------------------ | +| `route_selection` | `RouteSelection[]` | The routing decision(s). The server uses the first entry. | +| `rationale` | `string` | Human-readable explanation of why this route was selected. | -| Field | Type | Description | -| -------------- | ------------- | ------------------------------------------------------------------------ | -| `route_option` | `RouteOption` | The chosen destination (model + harness). | -| `params` | `Struct` | Optional router-specific parameters emitted alongside the decision. | - -Example response body: +**`RouteSelection`** — `route_option` (the chosen model + harness) and an +optional `params` (`Struct`) emitted alongside the decision. ```json { @@ -174,5 +347,12 @@ Example response body: The server maps the chosen `model` (and `harness`, when present) back to the matching catalog entry, restoring any stripped `model_prefix`. A model the server did not offer in `route_options` is rejected, and the session falls back to its -default harness. On any error (HTTP 4xx/5xx, unparseable body, empty selection) the -server surfaces the reason and degrades gracefully rather than blocking the session. +default harness. On any error (HTTP 4xx/5xx, unparseable body, empty selection) +the server surfaces the reason and degrades gracefully rather than blocking the +session. + +## Related + +- [Models & Credentials](/docs/build/models) — how each provider and credential + type is wired, including Databricks workspace profiles. +- [Harnesses](/docs/build/harnesses) — the runtimes a router can pick between. diff --git a/lib/og-manifest.json b/lib/og-manifest.json index 8c5d27a..7e6a1e5 100644 --- a/lib/og-manifest.json +++ b/lib/og-manifest.json @@ -31,7 +31,7 @@ }, "/docs/build/routing": { "title": "Smart Routing", - "description": "Let a router pick the best harness and model for each session automatically, using the built-in judge or an external routing API.", + "description": "Let a router pick the best harness and model for each session automatically — with the built-in judge, the Databricks AI Gateway task_v1 router, or both.", "eyebrow": "Build" }, "/docs/build/scheduled-tasks": {