Skip to content

Remove the InferenceGateway feature flag and enable the gateway everywhere#473

Merged
mrubens merged 2 commits into
developfrom
remove-inference-gateway-flag
Jul 17, 2026
Merged

Remove the InferenceGateway feature flag and enable the gateway everywhere#473
mrubens merged 2 commits into
developfrom
remove-inference-gateway-flag

Conversation

@mrubens

@mrubens mrubens commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The inference gateway has been running behind the InferenceGateway feature flag since it shipped. This removes the flag and makes the gateway the unconditional behavior for every deployment: gateway-covered provider API keys and ChatGPT subscription auth always stay on the control plane, and task sandboxes route inference through /api/inference with their run-scoped token.

Changes

  • feature-flags: removed the InferenceGateway enum entry, its config block, the isInferenceGatewayEnabledForDeployment fail-closed evaluator helper, and the server export.
  • db / sdk (dequeue): the inferenceGateway boolean option is gone from the resolver's public API. resolveEffectiveModelRuntimeEnv is now the control-plane entry point (routing, titles, summaries — no run token, so raw provider keys), and a new resolveSandboxModelRuntimeEnv is the sandbox entry point (gateway-covered keys stay on the control plane, advertised via R_INFERENCE_GATEWAY_KEYS; ChatGPT subscriptions route through the gateway). fetchResolvedRuntimeEnvVars calls the sandbox variant; the shared implementation keeps the distinction as a private parameter.
  • compute-providers / controller: removed the inferenceGatewayEnabled option from BuildWorkerEnvOptions and all five worker-env builders; buildBaseWorkerEnv now always withholds gateway-covered provider keys from the worker daemon env. Deleted the controller's inference-gateway-flag.ts helper and its use in the five spawn-worker paths.
  • api: the inference gateway handler no longer checks the flag before serving; run-token auth remains the gate.
  • web: dropped the flag from mockFeatureFlags.
  • tests: removed flag mocks and the flag-disabled rejection test; flipped worker-env expectations so gateway-covered keys (e.g. OPENROUTER_API_KEY, AI_GATEWAY_API_KEY) are asserted withheld while custom provider keys still flow through.

Notes

  • A Redis outage can no longer flip inference back to direct provider keys; the old fail-closed path is gone entirely, which is strictly safer for key containment.
  • Deployments that still have inference_gateway in their deployment metadata will see it surface as a legacy metadata control in the admin UI; it is ignored and can be deleted.

Validation

  • pnpm check-types, pnpm lint, pnpm knip all pass.
  • Targeted vitest runs pass for feature-flags, compute-providers worker-env, sdk dequeue-helpers, db model-runtime-config, controller spawn workers, api inference gateway, and the worker gateway tests.

@roomote-roomote

roomote-roomote Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

No new code issues found. See task

Reviewed d837861

@mrubens
mrubens merged commit 746fdfd into develop Jul 17, 2026
16 checks passed
@mrubens
mrubens deleted the remove-inference-gateway-flag branch July 17, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant