Route sandbox inference through the gateway behind a deployment flag

[mrubens]

Summary

Stacked on #447. When the InferenceGateway feature flag is enabled, task sandboxes stop receiving raw provider API keys for gateway-covered providers (OpenRouter, Anthropic, OpenAI, Google Gemini) and route inference through the gateway with their run-scoped token instead. Off by default; nothing changes for deployments that don't enable it.

How it works

Flag: InferenceGateway feature flag (deployment metadata key inference_gateway), togglable from the admin metadata controls like other deployment-level features. Evaluation fails closed to direct provider keys if flag evaluation is unavailable, so a Redis outage cannot break task inference.

Control plane (resolveEffectiveModelRuntimeEnv gains an inferenceGateway option):

• The dequeue/resume/env-reload path evaluates the flag and, when enabled, withholds gateway-covered provider keys and emits R_INFERENCE_GATEWAY_URL (derived from the sandbox-reachable platform API URL). Raw deployment env vars carrying those key names are also stripped from the merged sandbox env.
• Control-plane inference (routing, titles, summaries) keeps direct keys, since it holds no run token; those callers simply don't pass the option.
• Keys the gateway cannot serve yet (Bedrock, Vertex) and ChatGPT-subscription OAuth still flow.
• If no platform API URL is configured, resolution falls back to direct keys rather than breaking inference.

Worker launch env: BuildWorkerEnvOptions gains inferenceGatewayEnabled; each controller spawn path evaluates the flag before building the worker daemon env, so the daemon's process env no longer carries covered keys when enabled.

OpenCode config: for each selected model whose provider is gateway-covered, the worker rebases the provider onto the gateway (baseURL = gateway URL + provider + SDK base-path suffix, apiKey = {env:ROOMOTE_CLOUD_TOKEN}), following the existing Bedrock Mantle pattern. OpenRouter attribution headers are preserved, and the openai provider is left untouched when a ChatGPT subscription record is present (its OAuth flows through opencode's Codex plugin).

Token transport: the Anthropic and Gemini SDKs send their "API key" through x-api-key / x-goog-api-key rather than Authorization, so the API token middleware now accepts the run token from those headers on the /api/inference surface only. Everywhere else, Authorization remains the single token transport.

Provider definitions (ids, env keys, upstream URLs, path allowlists, SDK base-path suffixes) live in @roomote/types so the gateway and the worker share one source of truth.

Testing

• Resolver tests: option-driven gating, uncovered keys still flowing, missing-URL fallback
• OpenCode config tests: gateway rebasing for anthropic/google/openrouter (attribution headers intact), ChatGPT-subscription skip, no-op without the flag
• Middleware tests: provider-header token extraction scoped to the inference route, Authorization precedence, invalid tokens rejected
• Worker-env test: covered operator keys held back under the flag, Bedrock key still forwarded
• Full types, feature-flags, db, sdk, compute-providers, controller, api, and worker suites pass; typecheck, lint, knip clean
• Verified end-to-end locally (Modal sandbox, OpenRouter): opencode config rebased onto the gateway, no provider key values anywhere in the sandbox filesystem or process envs, task inference streamed through /api/inference/openrouter/v1 and completed normally

Rollout notes

• Suggested order: enable the flag on internal deployments (OpenRouter) first.
• Environment-snapshot setup env still carries provider keys when stored as deployment env vars (setup commands may reference them); the pre-snapshot scrub ([Fix] Snapshot credentials persist after task sleep #445) keeps them out of snapshot images.

[roomote-roomote]

2 issues outstanding. See task

• High packages/types/src/inference-gateway.ts:93-109 Handle Google's supported alternate credential name. getModelProviderEnvKeyCandidates({ providerId: 'google' }) supports both GEMINI_API_KEY and GOOGLE_GENERATIVE_AI_API_KEY, but the gateway definition only resolves and withholds GEMINI_API_KEY. With the flag enabled, a deployment using the alternate key still sends that raw key into the sandbox while the gateway returns "No Google Gemini API key", defeating credential isolation and breaking inference.
• Medium packages/db/src/lib/model-runtime-config.ts:303-307 Derive the gateway URL from the sandbox-reachable API origin. Docker workers rewrite a local TRPC_URL to host.docker.internal in spawn-docker-worker.ts:376, but this resolver builds R_INFERENCE_GATEWAY_URL from the API process's unmodified localhost URL. Flag-enabled Docker tasks therefore call the worker container's own localhost instead of the platform API.
• Medium packages/compute-providers/src/worker-env/base.ts:90-101 Apply the persisted deployment flag when filtering the worker launch environment. Sandbox env resolution enables the gateway from either the process flag or persisted deployment env, but this path checks only process.env.R_INFERENCE_GATEWAY. With a persisted flag and a provider key supplied in the control-plane process env, the task's worker daemon still receives the raw key, contrary to the stated isolation guarantee.
• High packages/sdk/src/server/lib/task-runs/dequeue-helpers.ts:216-219 Use one gateway decision for worker launch and task env construction, or scrub captured launcher keys when the gateway URL is present. This independently re-evaluates the flag after the controller evaluated it for worker spawn; if spawn returns false (including a transient evaluation failure) and dequeue returns true, the worker has already captured raw provider keys, and run-task.ts:597-606 merges them back into the gateway-enabled task despite dequeue redaction.

_{Reviewed 9c489b0}

[mrubens]

Verified end-to-end locally (Modal sandbox, OpenRouter, worker built from this branch): with the flag enabled the sandbox's opencode config rebases openrouter onto the gateway (baseURL + run-token apiKey), ~/.roomote/env.sh and all sandbox process environments contain no provider key values (only the R_MODEL_ENV_KEYS name list), R_INFERENCE_GATEWAY_URL arrives via the dequeue env, and the task completed normally with inference streaming through /api/inference/openrouter/v1.

[mrubens]

Follow-up commit replaces the R_INFERENCE_GATEWAY env var with an InferenceGateway feature flag (deployment metadata key inference_gateway, off by default), togglable from the admin metadata controls. The resolver now takes an explicit inferenceGateway option, the dequeue path and each worker spawn path evaluate the flag through the shared evaluator, and evaluation fails closed to direct keys if Redis is unavailable. The env-var approach is gone entirely.