Terraform auto-enables ARCO_DEBUG on the deployed dev service, turning off all token auth in favor of client-sup...

[ethan-tyler]

Severity: Medium · Category: security

Affected code

• infra/terraform/cloud_run.tf:98-101
• crates/arco-api/src/context.rs:72-88
• crates/arco-api/src/routes/tasks.rs:352-366
• crates/arco-api/src/config.rs:770-783
• infra/terraform/environments/arco-testing-dev.tfvars:38-40

Problem

cloud_run.tf sets debug purely from the environment name and the public flag:

      env {
        name  = "ARCO_DEBUG"
        value = var.environment == "dev" && !var.api_public ? "true" : "false"
      }

crates/arco-api/src/config.rs:770-783 maps ("dev", false) => Posture::Dev, so this exact combination also yields posture.is_dev() == true. The API then takes the debug branch everywhere identity is established. crates/arco-api/src/context.rs:72-88:

    let debug_allowed = state.config.debug && state.config.posture.is_dev();
    let (tenant, workspace, user_id, groups) = if debug_allowed {
        let tenant = header_string(headers, "X-Tenant-Id")...
        let workspace = header_string(headers, "X-Workspace-Id")...
        let user_id = user_id_from_headers(headers);
        let groups = groups_from_headers(headers);

No JWT is parsed at all — tenant, workspace, user id and group membership are read verbatim from request headers. crates/arco-api/src/routes/tasks.rs:352-366 does the same for the task-callback surface: with debug_allowed, any non-empty bearer string plus X-Tenant-Id/X-Workspace-Id headers is accepted and decode_task_token is never called.

infra/terraform/environments/arco-testing-dev.tfvars sets environment = "dev" and api_public = false, i.e. exactly the combination that enables this, for the real GCP project arco-testing-20260320. There is no separate variable to turn debug off: setting api_public = true is the only way, and that simultaneously flips ingress to INGRESS_TRAFFIC_ALL (cloud_run.tf:25) and grants allUsers the run.invoker role (iam.tf:309-316).

Why it matters

A deployed cloud service holding multi-tenant catalog and orchestration data runs with authentication and authorization entirely client-asserted. The only remaining controls are Cloud Run ingress (INGRESS_TRAFFIC_INTERNAL_ONLY) and the run.invoker IAM binding — so every principal in the project that can invoke arco-api-dev (including the flow_worker, flow_dispatcher, flow_sweeper service accounts, any GCE VM in the VPC, and any human with roles/run.invoker) gets unauthenticated cross-tenant read/write plus self-asserted group membership for whatever RBAC consumes ctx.groups. The safety property is inverted from what an operator would expect: hardening the deployment (api_public=false) is what switches the bypass on.

Failure scenario

Deploy with arco-testing-dev.tfvars (environment=dev, api_public=false). arco-api-dev boots with ARCO_DEBUG=true and Posture::Dev. A developer or workload with run.invoker on arco-api-dev — e.g. the flow_worker SA, whose only intended right is to post its own callbacks — sends Authorization: Bearer x, X-Tenant-Id: some-other-tenant, X-Workspace-Id: prod-ws, X-Groups: admins. context.rs:74-85 accepts it verbatim, tasks.rs:354-366 skips task-token verification entirely, and the request runs scoped to a tenant the caller was never authorized for. Conversely, an operator who wants token auth in dev must set api_public=true, which publishes the service to the internet and binds allUsers as invoker.

Suggested fix

Make debug an explicit, independently-controlled variable that defaults to false and is never derived from the environment name, e.g. variable "api_debug" { type = bool; default = false } with a validation that rejects api_debug = true for any environment other than a local/emulator target, and set ARCO_DEBUG = var.api_debug ? "true" : "false". Leave it false in infra/terraform/environments/arco-testing-dev.tfvars. Retain the existing hard guard at crates/arco-api/src/server.rs:1061 (!posture.is_dev() && config.debug rejects) but stop having Terraform produce the dev+debug combination for any deployed Cloud Run service.

Verification notes

Existing mitigations found (do not fully close it): Searched for and found these, none of which neutralize the defect: (1) server.rs:1060-1064 does gate debug on posture — if !posture.is_dev() && config.debug { error } — but Posture::Dev is exactly what terraform produces, so the gate passes. (2) tasks.rs:352 requires a bearer token to be present (extract_bearer_token at :341) before the debug branch, but any non-empty string satisfies it; decode_task_token is never called. context.rs's debug branch requires no bearer token at all. (3) Tests exist and assert this behavior is intentional in Dev, not that it is blocked: tasks.rs:1267-1305 test_task_auth_middleware_accepts_debug_headers_in_dev and :1307-1338 test_task_auth_middleware_rejects_debug_headers_outside_dev. So the app-layer gate works as designed; the defect is terraform handing a deployed service the dev posture. (4) I checked whether debug forces ephemeral storage — it does not: main.rs:35-48 selects ObjectStoreBackend whenever ARCO_STORAGE_BUCKET is set, and cloud_run.tf:139-141 always sets it to google_storage_bucket.catalog.name. So the deployed dev API in debug mode reads/writes the real GCS catalog bucket. (5) Cloud Run IAM is a real remaining control: with api_public=false there is no allUsers binding (iam.tf:308-316 is count = var.api_public ? 1 : 0), ingress is INGRESS_TRAFFIC_INTERNAL_ONLY (cloud_run.tf:25), and grep of google_cloud_run_v2_service.api across infra/terraform shows iam.tf:313 as the ONLY IAM reference to the API service. (6) staging.tfvars sets environment="staging" so Posture::Private and ARCO_DEBUG=false — only the dev environment is affected.

Caveat / scope: Two of the claim's supporting statements are factually wrong and should not be repeated. (a) "every principal ... including the flow_worker, flow_dispatcher, flow_sweeper service accounts" gets in — NO. Terraform grants run.invoker on arco-api only via the allUsers binding when api_public=true; there is no per-SA invoker binding on the API service, and no project-level roles/run.invoker anywhere (the only google_project_iam_member grants are roles/cloudtasks.enqueuer at iam.tf:194-197 and :290-294). The exploit population is limited to principals an operator granted run.invoker/editor/owner outside terraform — in practice project admins. (Separately, flow_worker receives ARCO_FLOW_API_URL at cloud_run.tf:926-928 with no invoker binding on the API; that gap belongs to the already-open #218-#249 deployed-UAT cluster, not here.) (b) "setting api_public = true is the only way" to disable debug — overstated twice over: setting environment to "staging"/"prod" also yields ARCO_DEBUG=false without publishing, and api_public=true against the current dev tfvars would not even boot, because validate_config (server.rs:1068-1078, :1102-1111) rejects allowed_cors_origins="" and requires a JWT key while jwt_secret_name/jwt_issuer/jwt_audience are all "" (tfvars:39-42). The accurate framing is: no dedicated debug variable exists, and the in-place knob that disables it is the one that makes the service public. Severity downgraded to Medium because the blast radius is a testing project (arco-testing-20260320) and exploitation requires a Google-authenticated caller who already holds run.invoker plus internal-network origin. I could not confirm which image is actually running on arco-api-dev today — the committed tfvars pins api_image to the cloudrun/container/hello placeholder (tfvars:8), though tools/test_user_acceptance_uat_runner.sh:122-336 shows the dev environment is genuinely live with real arco-flow- run.app services.

Deployed-posture review: ingress = INGRESS_TRAFFIC_INTERNAL_ONLY whenever api_public=false (cloud_run.tf:25); no Terraform resource grants roles/run.invoker on the arco-api service to any principal except allUsers when api_public=true (iam.tf:309-316 is the only binding targeting google_cloud_run_v2_service.api); server.rs:1061 blocks debug for any non-dev posture, so staging.tfvars (environment="staging") gets ARCO_DEBUG=false and Posture::Private with enforced JWT + task tokens; RequestContext.groups is not consumed by any authorization check in arco-api (only audit), so X-Groups spoofing grants nothing today; arco-iceberg/arco-uc header-tenant surfaces are disabled by default (IcebergApiConfig::default().enabled = false, config.rs:421) and never enabled in Terraform; the committed dev tfvars uses placeholder hello images, so the file as checked in does not by itself stand up the Arco API.

Found during a staff-level audit of origin/main @ c3c0867. Mechanism confirmed by an independent adversarial verifier.