Add a flag-gated inference gateway so provider keys stay out of task sandboxes#447
Merged
Conversation
Adds /api/inference/:provider/* to the API server. Task sandboxes authenticate with their run-scoped token; the gateway resolves the deployment's provider API key server-side (runtime env first, then persisted deployment env vars), injects it upstream, and streams the response back. Upstream paths are allow-listed per provider so run tokens can only reach inference endpoints. Covers OpenRouter, Anthropic, OpenAI, and Google Gemini. Providers that need request signing (Vertex) or client-managed OAuth refresh (ChatGPT subscriptions) are not proxied yet.
Contributor
|
No code issues found. See task
Reviewed 401bb08 |
Contributor
Author
|
@roomote-roomote address the feedback |
Contributor
|
I'm on it. I started a dedicated PR follow-up task for this request, and I'll keep updates here. See task |
Contributor
|
Addressed all three inference-gateway findings in 593d95d:
Focused API and DB tests passed, along with API/DB type checks and the pre-push lint, typecheck, and knip gates. See task Revert this commit |
roomote-roomote
Bot
changed the title
* Route sandbox inference through the gateway behind a deployment flag When R_INFERENCE_GATEWAY is enabled (control-plane env or persisted deployment env var), task sandboxes no longer receive raw provider API keys for gateway-covered providers (OpenRouter, Anthropic, OpenAI, Google Gemini). Instead: - The dequeue/resume env resolver emits R_INFERENCE_GATEWAY_URL and withholds covered keys from the sandbox env (control-plane inference keeps direct keys since it holds no run token). - The worker daemon launch env skips covered operator keys. - The worker's OpenCode config rebases covered providers onto the gateway with the run token as the API key, preserving OpenRouter attribution headers and leaving ChatGPT-subscription OAuth and Bedrock/Vertex untouched. - The API token middleware accepts the run token from x-api-key / x-goog-api-key on the inference gateway surface only, since the Anthropic and Gemini SDKs send their key through those headers. Provider definitions are shared between the gateway and the worker via @roomote/types so path suffixes and env keys cannot drift. * Gate the inference gateway on a feature flag instead of an env var Replaces the R_INFERENCE_GATEWAY env var with an InferenceGateway feature flag stored in deployment metadata (metadata key inference_gateway), so the gateway can be toggled from the admin controls like other deployment-level features. - resolveEffectiveModelRuntimeEnv takes an explicit inferenceGateway option instead of sniffing env vars; the dequeue path evaluates the flag through the shared evaluator and fails closed (direct keys) if flag evaluation is unavailable. - Worker daemon env builders take inferenceGatewayEnabled via BuildWorkerEnvOptions; each controller spawn path evaluates the flag before building the env. - Control-plane inference is unaffected: callers that do not pass the option keep direct provider keys. --------- Co-authored-by: Matt Rubens <2600+mrubens@users.noreply.github.com>
mrubens
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Extends the gateway registry from four providers to every provider the setup catalog supports with API-key auth: Vercel AI Gateway, Requesty, Baseten, Together AI, Moonshot AI, MiniMax, OpenCode Zen, xAI, and Amazon Bedrock (Mantle). Upstream bases follow models.dev, the registry OpenCode itself resolves providers from. - MiniMax and Bedrock Mantle proxy through their Anthropic-compatible endpoints (x-api-key); the rest are OpenAI-compatible bearer-auth. - Bedrock upstreams are region-templated: the gateway resolves AWS_REGION (deployment env, default us-east-1) per request and validates it before substitution. The region var still flows to sandboxes; only the bearer token is withheld. - Nested-path allowance is now a per-provider registry field instead of a hardcoded Google check, reused for the Vercel AI Gateway protocol. - The worker's gateway rebase needs no changes: it is model-prefix driven, and the Bedrock Mantle merge runs before the gateway merge so its baseURL/apiKey are overridden while its model config is kept. Still direct: Google Vertex (request signing) and ChatGPT subscriptions (client-managed OAuth refresh).
Merges develop and addresses the /code-review findings, consolidating the gateway decision to a single per-run signal. - Gate the /api/inference endpoint on the InferenceGateway flag: it was live on every deployment, so a run token could pull a provider key its sandbox never held. Fails closed on evaluation error. - Reject encoded-slash / dot-segment traversal on nested-path providers (google, vercel) so a run token cannot ride the allowlist to a non-inference upstream endpoint. - Build the sandbox gateway URL worker-side from the container-reachable platform URL instead of dequeue's raw TRPC_URL (fixes a Docker/ self-host outage where the emitted localhost URL was unreachable). - Thread one authoritative served-keys list (R_INFERENCE_GATEWAY_KEYS): the resolver withholds and advertises exactly the configured, gateway-covered keys; dequeue redaction strips exactly that set (so a provider key the deployment set for its own code survives); the worker strips exactly that set from the harness env (defeating daemon-env re-injection on pre-flag sandboxes) and rebases exactly those providers (so a repo-pinned provider is not withheld-without-rebase). - Apply the same gateway withholding to the snapshot env path. - Prefer GOOGLE_GENERATIVE_AI_API_KEY over GEMINI_API_KEY to match the SDK's pre-gateway precedence when both are set. - Share one fail-closed flag evaluator across dequeue, spawn, and the gateway; share the Bedrock region/id constants via @roomote/types. - Fix stale comments that claimed Bedrock keys still flow directly.
Contributor
Author
|
Addressed the /code-review findings in Fixed in code
Rollout requirements (documented, not code-enforced)
|
The helper now calls getRedis(); without Redis (CI) the real connection floods ioredis unhandled-error events and fails the modal/daytona/blaxel spawn suites. Mock it to a fixed value like the other spawn dependencies.
CodeQL flagged the anchored `/\/+$/` in the gateway URL builders as a polynomial regular expression on uncontrolled input. Strip trailing slashes with a bounded char-code loop instead; add unit coverage for the URL builders and key lookups.
Contributor
Author
|
@roomote-roomote review this again |
Contributor
|
I started a PR review task for this request. See task |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an inference gateway at
/api/inference/:provider/*and, behind theInferenceGatewayfeature flag, routes task-sandbox inference through it so provider API keys stay on the control plane. Every key-authenticated provider in the setup catalog is covered: OpenRouter, Anthropic, OpenAI, Google Gemini, Vercel AI Gateway, Requesty, Baseten, Together AI, Moonshot AI, MiniMax, OpenCode Zen, xAI, and Amazon Bedrock (Mantle, with region-templated upstreams resolved from AWS_REGION per request). Google Vertex (request signing) and ChatGPT subscriptions (client-managed OAuth refresh) still flow directly. Sandboxes authenticate with their run-scoped token (ROOMOTE_CLOUD_TOKEN); the gateway resolves the deployment's provider key server-side, injects it upstream, and streams the response back.This extends the existing broker pattern (integration MCP proxy, git credential proxy, comms brokering, OIDC federation) to LLM traffic. The flag is off by default; nothing changes for deployments that don't enable it.
The gateway
task-tokenin the central route-policy manifest; the handler additionally verifies the run token's task run still exists (same pattern as the MCP proxy). The Anthropic and Gemini SDKs send their "API key" throughx-api-key/x-goog-api-keyrather thanAuthorization, so the token middleware accepts the run token from those headers on the/api/inferencesurface only.count_tokens); Google model routes match nested paths because the model ID and action live below/models. Everything else 403s, so a run token can never reach a provider's account, billing, or admin surface (e.g./v1/messages/batchesand OpenRouter/v1/keyare rejected).GOOGLE_GENERATIVE_AI_API_KEYalias for Gemini. MiniMax and Bedrock Mantle proxy through their Anthropic-compatible endpoints; the aggregators are OpenAI-compatible bearer-auth; Vercel uses the@ai-sdk/gatewayprotocol under/v1/ai. Upstream bases follow models.dev, the registry OpenCode itself resolves providers from.The flag-gated wiring
Flag:
InferenceGateway(deployment metadata keyinference_gateway), togglable from the admin metadata controls. Evaluation fails closed to direct provider keys if unavailable, so a Redis outage cannot break task inference.When enabled:
R_INFERENCE_GATEWAY_URLderived from the sandbox-reachable platform API URL. Control-plane inference (routing, titles, summaries) keeps direct keys since it holds no run token.baseURL= gateway URL + provider + SDK base-path suffix,apiKey={env:ROOMOTE_CLOUD_TOKEN}), following the Bedrock Mantle pattern. OpenRouter attribution headers are preserved; theopenaiprovider is untouched when a ChatGPT subscription record is present.AWS_REGIONalso still flows since it is configuration, not a secret.Provider definitions (ids, key env vars, upstream URLs, allowlists, SDK base-path suffixes) live in
@roomote/typesso the gateway and worker share one source of truth.Testing
types,feature-flags,db,sdk,compute-providers,controller,api, andworker; typecheck, lint, knip clean/api/inference/openrouter/v1and completed normallyRollout notes