Skip to content

Remove the InferenceGateway feature flag and enable the gateway everywhere #473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mrubens merged 2 commits into from
Jul 17, 2026
Merged

Remove the InferenceGateway feature flag and enable the gateway everywhere #473

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cb8083f
Remove the InferenceGateway feature flag and enable the gateway every…
mrubens Jul 17, 2026
d837861
Split resolver into sandbox and control-plane entry points instead of…
mrubens Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 0 additions & 25 deletions apps/api/src/handlers/inference/__tests__/inference-gateway.test.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 0 additions & 19 deletions apps/api/src/handlers/inference/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@ import { Hono } from 'hono';

import { formatSingleLineLog } from '@roomote/types';
import { db, eq, taskRuns } from '@roomote/db/server';
import { isInferenceGatewayEnabledForDeployment } from '@roomote/feature-flags/server';
import { getRedis } from '@roomote/redis';

import type { Variables } from '../../types';
import { fetchWithLongLivedStreamDispatcher } from '../long-lived-fetch';
Expand Down Expand Up @@ -111,23 +109,6 @@ inference.on(['POST', 'GET'], '/:provider/*', async (c) => {
);
}

// The gateway is only open when the deployment has opted into it. Without
// this check the endpoint would serve provider keys on every deployment
// regardless of the flag, so a run token could reach a provider key its
// sandbox never held. Fails closed (gateway closed) when evaluation is
// unavailable, matching the withholding side.
const gatewayEnabled = await isInferenceGatewayEnabledForDeployment(
getRedis(),
logPrefix,
);

if (!gatewayEnabled) {
return c.json(
{ error: 'The inference gateway is not enabled for this deployment' },
403,
);
}

const taskRun = await db.query.taskRuns.findFirst({
columns: { id: true },
where: eq(taskRuns.id, auth.runId),
Expand Down
4 changes: 0 additions & 4 deletions apps/controller/src/compute-providers/__tests__/spawn-blaxel-worker.test.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 0 additions & 4 deletions apps/controller/src/compute-providers/__tests__/spawn-daytona-worker.test.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 0 additions & 4 deletions apps/controller/src/compute-providers/__tests__/spawn-modal-worker.test.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 0 additions & 11 deletions apps/controller/src/compute-providers/inference-gateway-flag.ts
View file
Open in desktop

This file was deleted.

2 changes: 0 additions & 2 deletions apps/controller/src/compute-providers/spawn-blaxel-worker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ import {
resolveAuthBypassHeaderName,
resolveAuthBypassValue,
} from '@roomote/compute-providers';
import { isInferenceGatewayEnabledForWorkerEnv } from './inference-gateway-flag';

import { primeEnvironmentOidcForMachine } from '../sandbox-oidc';
import {
Expand Down Expand Up @@ -186,7 +185,6 @@ export async function spawnBlaxelWorker(
args,
env: buildBlaxelWorkerEnv({
authToken,
inferenceGatewayEnabled: await isInferenceGatewayEnabledForWorkerEnv(),
sandboxExpiresAtMs: Date.now() + config.blaxelTimeoutMs,
deploymentSlug: config.deploymentSlug,
environmentId,
Expand Down
2 changes: 0 additions & 2 deletions apps/controller/src/compute-providers/spawn-daytona-worker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ import {
resolveAuthBypassHeaderName,
resolveAuthBypassValue,
} from '@roomote/compute-providers';
import { isInferenceGatewayEnabledForWorkerEnv } from './inference-gateway-flag';

import { primeEnvironmentOidcForMachine } from '../sandbox-oidc';
import {
Expand Down Expand Up @@ -331,7 +330,6 @@ export async function spawnDaytonaWorker(
args,
env: buildDaytonaWorkerEnv({
authToken,
inferenceGatewayEnabled: await isInferenceGatewayEnabledForWorkerEnv(),
sandboxExpiresAtMs: Date.now() + daytonaTimeoutMs,
deploymentSlug,
environmentId,
Expand Down
2 changes: 0 additions & 2 deletions apps/controller/src/compute-providers/spawn-docker-worker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ import {
resolveAuthBypassHeaderName,
resolveAuthBypassValue,
} from '@roomote/compute-providers';
import { isInferenceGatewayEnabledForWorkerEnv } from './inference-gateway-flag';

import {
getNamedPortsForTaskRun,
Expand Down Expand Up @@ -366,7 +365,6 @@ export async function spawnDockerWorker(

const workerEnv = buildDockerWorkerEnv({
authToken,
inferenceGatewayEnabled: await isInferenceGatewayEnabledForWorkerEnv(),
sandboxExpiresAtMs: Date.now() + config.dockerTimeoutMs,
deploymentSlug: config.deploymentSlug,
environmentId: taskRun.payload.environmentId,
Expand Down
2 changes: 0 additions & 2 deletions apps/controller/src/compute-providers/spawn-e2b-worker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ import {
resolveAuthBypassHeaderName,
resolveAuthBypassValue,
} from '@roomote/compute-providers';
import { isInferenceGatewayEnabledForWorkerEnv } from './inference-gateway-flag';

import { primeEnvironmentOidcForMachine } from '../sandbox-oidc';
import {
Expand Down Expand Up @@ -327,7 +326,6 @@ export async function spawnE2bWorker(
args,
env: buildE2bWorkerEnv({
authToken,
inferenceGatewayEnabled: await isInferenceGatewayEnabledForWorkerEnv(),
sandboxExpiresAtMs: Date.now() + e2bTimeoutMs,
deploymentSlug,
environmentId,
Expand Down
2 changes: 0 additions & 2 deletions apps/controller/src/compute-providers/spawn-modal-worker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ import {
resolveAuthBypassHeaderName,
resolveAuthBypassValue,
} from '@roomote/compute-providers';
import { isInferenceGatewayEnabledForWorkerEnv } from './inference-gateway-flag';

import { primeEnvironmentOidcForMachine } from '../sandbox-oidc';
import {
Expand Down Expand Up @@ -410,7 +409,6 @@ export async function spawnModalWorker(
args,
env: buildModalWorkerEnv({
authToken,
inferenceGatewayEnabled: await isInferenceGatewayEnabledForWorkerEnv(),
sandboxExpiresAtMs: Date.now() + modalTimeoutMs,
deploymentSlug,
environmentId,
Expand Down
1 change: 0 additions & 1 deletion apps/web/src/lib/mock-utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ export const mockFeatureFlags: Record<FeatureFlag, boolean> = {
[FeatureFlag.AuthorshipRules]: false,
[FeatureFlag.BackgroundSubagents]: false,
[FeatureFlag.CodeMode]: false,
[FeatureFlag.InferenceGateway]: false,
};

export const mockUserResource: UserResource = {
Expand Down
24 changes: 7 additions & 17 deletions packages/compute-providers/src/worker-env/__tests__/base.test.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 9 additions & 13 deletions packages/compute-providers/src/worker-env/base.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,7 @@ function getOperatorModelProviderEnvKeys(): string[] {
return [...new Set([...DEFAULT_MODEL_PROVIDER_ENV_KEYS, ...configured])];
}

function buildOperatorModelProviderEnv(
inferenceGatewayEnabled: boolean,
): Record<string, string> {
function buildOperatorModelProviderEnv(): Record<string, string> {
const env: Record<string, string> = {};

for (const key of MODEL_ROLE_ENV_VAR_NAMES) {
Expand Down Expand Up @@ -76,21 +74,20 @@ function buildOperatorModelProviderEnv(
env.R_MODEL_ENV_KEYS = process.env.R_MODEL_ENV_KEYS;
}

// When the inference gateway is enabled, gateway-covered provider keys
// (OpenRouter, Anthropic, OpenAI, Gemini, the aggregators, Bedrock) stay off
// the worker daemon process env so they never appear in the sandbox's
// /proc; the per-task dequeue env routes the harness through the gateway
// instead. ChatGPT subscriptions use their dedicated run-token gateway
// route, while disabled-provider credentials such as Vertex are blocked
// from the worker env entirely.
// Gateway-covered provider keys (OpenRouter, Anthropic, OpenAI, Gemini,
// the aggregators, Bedrock) stay off the worker daemon process env so they
// never appear in the sandbox's /proc; the per-task dequeue env routes the
// harness through the inference gateway instead. ChatGPT subscriptions use
// their dedicated run-token gateway route, while disabled-provider
// credentials such as Vertex are blocked from the worker env entirely.
const gatewayCoveredKeys = new Set(INFERENCE_GATEWAY_PROVIDER_ENV_VAR_NAMES);

for (const key of getOperatorModelProviderEnvKeys()) {
if (BLOCKED_WORKER_ENV_KEYS.has(key)) {
continue;
}

if (inferenceGatewayEnabled && gatewayCoveredKeys.has(key)) {
if (gatewayCoveredKeys.has(key)) {
continue;
}

Expand All @@ -108,7 +105,6 @@ export function buildBaseWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled = false,
}: BuildWorkerEnvOptions): Record<string, string> {
const previewProxyBaseUrl = process.env.PREVIEW_PROXY_BASE_URL;

Expand Down Expand Up @@ -157,7 +153,7 @@ export function buildBaseWorkerEnv({
...(process.env.PREVIEW_AUTH_COOKIE_NAME && {
PREVIEW_AUTH_COOKIE_NAME: process.env.PREVIEW_AUTH_COOKIE_NAME,
}),
...buildOperatorModelProviderEnv(inferenceGatewayEnabled),
...buildOperatorModelProviderEnv(),
...filterWorkerExtraEnv(extraEnv),
};
}
2 changes: 0 additions & 2 deletions packages/compute-providers/src/worker-env/blaxel.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ export function buildBlaxelWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled,
deploymentSlug,
environmentId,
image,
Expand All @@ -20,7 +19,6 @@ export function buildBlaxelWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled,
}),
...buildWorkerContextEnv({
provider: 'blaxel',
Expand Down
2 changes: 0 additions & 2 deletions packages/compute-providers/src/worker-env/daytona.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ export function buildDaytonaWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled,
deploymentSlug,
environmentId,
snapshotName,
Expand All @@ -21,7 +20,6 @@ export function buildDaytonaWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled,
}),
...buildWorkerContextEnv({
provider: 'daytona',
Expand Down
2 changes: 0 additions & 2 deletions packages/compute-providers/src/worker-env/docker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ export function buildDockerWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled,
deploymentSlug,
environmentId,
image,
Expand All @@ -20,7 +19,6 @@ export function buildDockerWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled,
}),
...buildWorkerContextEnv({
provider: 'docker',
Expand Down
2 changes: 0 additions & 2 deletions packages/compute-providers/src/worker-env/e2b.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ export function buildE2bWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled,
deploymentSlug,
environmentId,
templateId,
Expand All @@ -21,7 +20,6 @@ export function buildE2bWorkerEnv({
authToken,
sandboxExpiresAtMs,
extraEnv,
inferenceGatewayEnabled,
}),
...buildWorkerContextEnv({
provider: 'e2b',
Expand Down
Loading
Loading