Severity: Medium · Category: security
Affected code
infra/terraform/cloud_run.tf:118-121infra/terraform/cloud_run.tf:643-646infra/terraform/cloud_run.tf:816-819infra/terraform/environments/arco-testing-dev.tfvars:22tools/xtask/tests/deploy_script.rs:44-63crates/arco-core/src/task_tokens.rs:72-89
Problem
The HS256 secret that mints and validates worker task-callback tokens is passed to Cloud Run as a literal env value, not a Secret Manager reference:
env {
name = "ARCO_TASK_TOKEN_SECRET"
value = var.task_token_secret
}
and identically for ARCO_FLOW_TASK_TOKEN_SECRET on the dispatcher (lines 643-646) and sweeper (lines 816-819). Every other secret in this stack uses the value_source { secret_key_ref { ... } } form — ARCO_JWT_SECRET (cloud_run.tf:179-190), ARCO_TENANT_SECRET_B64 (467-478), ARCO_FLOW_WORKER_DISPATCH_SECRET (610-621, 798-809) — and main.tf creates google_secret_manager_secret resources for each of those. There is no such resource for the task token secret.
Worse, a concrete value is committed: infra/terraform/environments/arco-testing-dev.tfvars:22 contains task_token_secret = "dev-arco-task-token-secret" (added 2026-06-20 in commit 6ab8d6c). That file is the --env dev var-file consumed by scripts/deploy.sh (deploy.sh:231 resolves infra/terraform/environments/<env>.tfvars) and docs/reports/2026-06-09-real-world-uat-cloud-local-evidence.md:46 shows it being applied against the real project arco-testing-20260320.
crates/arco-core/src/task_tokens.rs:72-89 (TaskTokenConfig::validate) enforces only non-emptiness and TTL bounds — no minimum length or entropy check — so a 26-char guessable literal is accepted. decode_task_token (task_tokens.rs:313) verifies with DecodingKey::from_secret(config.hs256_secret.as_bytes()).
The pattern is actively pinned by a CI test: tools/xtask/tests/deploy_script.rs:44-63 (cloud_run_services_keep_task_token_env) asserts cloud_run_tf.contains("value = var.task_token_secret"), so converting to Secret Manager fails CI.
Why it matters
crates/arco-api/src/routes/tasks.rs:368-381 derives the request's tenant and workspace directly from the decoded task-token claims. Anyone holding the signing secret can mint a token for an arbitrary tenant/workspace and drive the whole /tasks callback surface (task state transitions, ledger event writes) cross-tenant. For the dev/UAT project that secret is public in the repository. For staging/prod the plaintext-env form still leaks the secret to every principal with run.services.get or run.revisions.get (roles/viewer includes these), records it in Terraform state, and surfaces it in gcloud run services describe output and revision diffs — none of which is true for the Secret Manager-referenced secrets in the same file.
Failure scenario
./scripts/deploy.sh --env dev --tfvars-file infra/terraform/environments/arco-testing-dev.tfvars applies ARCO_TASK_TOKEN_SECRET=dev-arco-task-token-secret to arco-api-dev and the flow dispatcher/sweeper. 2. Anyone who can read the repo signs an HS256 JWT with iss=https://arco.dev/task-token, aud=arco-worker-callback (both also committed, lines 23-24) and tenant_id/workspace_id of any tenant. 3. POSTing that bearer token to the API's task callback routes passes decode_task_token and yields a RequestContext scoped to the victim tenant. Separately, in staging/prod, a read-only project viewer runs gcloud run services describe arco-api-prod --format='value(spec.template.spec.containers[0].env)' and recovers the production signing secret in cleartext.
Suggested fix
Create a google_secret_manager_secret for the task token secret (mirroring flow_worker_dispatch_secret in main.tf:145-159), grant roles/secretmanager.secretAccessor to the api and flow_controller SAs, and replace all three env blocks with the value_source { secret_key_ref { secret = ...; version = "latest" } } form. Delete the literal from arco-testing-dev.tfvars and rotate the value in the arco-testing-20260320 project. Update tools/xtask/tests/deploy_script.rs:44-63 to assert the secret_key_ref form instead of value = var.task_token_secret. Add a minimum-entropy check (e.g. >= 32 bytes) to TaskTokenConfig::validate in crates/arco-core/src/task_tokens.rs:72.
Verification notes
Existing mitigations found (do not fully close it): api_public=false in arco-testing-dev.tfvars → INGRESS_TRAFFIC_INTERNAL_ONLY (cloud_run.tf:25) and no allUsers run.invoker binding (iam.tf:309-316, count=0), so the dev API is not internet-reachable. ARCO_DEBUG=true in that same dev posture (cloud_run.tf:98-101 → config.rs:557 → tasks.rs:352) makes task_auth_middleware skip decode_task_token entirely, so the committed secret adds no attack capability in dev. staging.tfvars and dev.tfvars.example set no task_token_secret, so the default "" leaves the config unconfigured and task_tokens.rs:74-77 rejects all callbacks rather than accepting weak ones. variables.tf:157 marks the variable sensitive (hides plan output only, not state or Cloud Run env).
Caveat / scope: I could not confirm any terraform apply actually landed this value against arco-testing-20260320 — that is not determinable from the repo. I also did not audit whether the CI pipeline sets TF_VAR_task_token_secret from a runner secret for non-dev environments; grep found no such reference under .github/, scripts/, or docs/, but absence there is not proof of the deployed posture. The ARCO_DEBUG header-bypass in tasks.rs:352-367 is a separate weakness I observed while checking mitigations; I have not verified whether it is already tracked, so I am not filing it here.
Deployed-posture review: API ingress is INGRESS_TRAFFIC_INTERNAL_ONLY whenever api_public = false (cloud_run.tf:25), which holds in both dev and staging tfvars; the allUsers run.invoker binding (iam.tf:309-316) is count = 0 in that case. Staging never sets task_token_secret, so variables.tf:152-157 leaves it "" and decode_task_token's validate() rejects all callbacks. No prod tfvars exists in the repo. deploy.sh:231 resolves .tfvars and no dev.tfvars exists, so the committed file requires an explicit --tfvars-file.
Found during a staff-level audit of origin/main @ c3c0867. Mechanism confirmed by an independent adversarial verifier.
Severity: Medium · Category: security
Affected code
infra/terraform/cloud_run.tf:118-121infra/terraform/cloud_run.tf:643-646infra/terraform/cloud_run.tf:816-819infra/terraform/environments/arco-testing-dev.tfvars:22tools/xtask/tests/deploy_script.rs:44-63crates/arco-core/src/task_tokens.rs:72-89Problem
The HS256 secret that mints and validates worker task-callback tokens is passed to Cloud Run as a literal env value, not a Secret Manager reference:
and identically for
ARCO_FLOW_TASK_TOKEN_SECRETon the dispatcher (lines 643-646) and sweeper (lines 816-819). Every other secret in this stack uses thevalue_source { secret_key_ref { ... } }form —ARCO_JWT_SECRET(cloud_run.tf:179-190),ARCO_TENANT_SECRET_B64(467-478),ARCO_FLOW_WORKER_DISPATCH_SECRET(610-621, 798-809) — and main.tf createsgoogle_secret_manager_secretresources for each of those. There is no such resource for the task token secret.Worse, a concrete value is committed:
infra/terraform/environments/arco-testing-dev.tfvars:22containstask_token_secret = "dev-arco-task-token-secret"(added 2026-06-20 in commit 6ab8d6c). That file is the--env devvar-file consumed byscripts/deploy.sh(deploy.sh:231 resolvesinfra/terraform/environments/<env>.tfvars) and docs/reports/2026-06-09-real-world-uat-cloud-local-evidence.md:46 shows it being applied against the real projectarco-testing-20260320.crates/arco-core/src/task_tokens.rs:72-89(TaskTokenConfig::validate) enforces only non-emptiness and TTL bounds — no minimum length or entropy check — so a 26-char guessable literal is accepted.decode_task_token(task_tokens.rs:313) verifies withDecodingKey::from_secret(config.hs256_secret.as_bytes()).The pattern is actively pinned by a CI test:
tools/xtask/tests/deploy_script.rs:44-63(cloud_run_services_keep_task_token_env) assertscloud_run_tf.contains("value = var.task_token_secret"), so converting to Secret Manager fails CI.Why it matters
crates/arco-api/src/routes/tasks.rs:368-381 derives the request's tenant and workspace directly from the decoded task-token claims. Anyone holding the signing secret can mint a token for an arbitrary tenant/workspace and drive the whole /tasks callback surface (task state transitions, ledger event writes) cross-tenant. For the dev/UAT project that secret is public in the repository. For staging/prod the plaintext-env form still leaks the secret to every principal with
run.services.getorrun.revisions.get(roles/viewer includes these), records it in Terraform state, and surfaces it ingcloud run services describeoutput and revision diffs — none of which is true for the Secret Manager-referenced secrets in the same file.Failure scenario
./scripts/deploy.sh --env dev --tfvars-file infra/terraform/environments/arco-testing-dev.tfvarsappliesARCO_TASK_TOKEN_SECRET=dev-arco-task-token-secretto arco-api-dev and the flow dispatcher/sweeper. 2. Anyone who can read the repo signs an HS256 JWT withiss=https://arco.dev/task-token,aud=arco-worker-callback(both also committed, lines 23-24) andtenant_id/workspace_idof any tenant. 3. POSTing that bearer token to the API's task callback routes passesdecode_task_tokenand yields a RequestContext scoped to the victim tenant. Separately, in staging/prod, a read-only project viewer runsgcloud run services describe arco-api-prod --format='value(spec.template.spec.containers[0].env)'and recovers the production signing secret in cleartext.Suggested fix
Create a
google_secret_manager_secretfor the task token secret (mirroringflow_worker_dispatch_secretin main.tf:145-159), grantroles/secretmanager.secretAccessorto the api and flow_controller SAs, and replace all three env blocks with thevalue_source { secret_key_ref { secret = ...; version = "latest" } }form. Delete the literal fromarco-testing-dev.tfvarsand rotate the value in the arco-testing-20260320 project. Updatetools/xtask/tests/deploy_script.rs:44-63to assert thesecret_key_refform instead ofvalue = var.task_token_secret. Add a minimum-entropy check (e.g. >= 32 bytes) toTaskTokenConfig::validatein crates/arco-core/src/task_tokens.rs:72.Verification notes
Existing mitigations found (do not fully close it): api_public=false in arco-testing-dev.tfvars → INGRESS_TRAFFIC_INTERNAL_ONLY (cloud_run.tf:25) and no allUsers run.invoker binding (iam.tf:309-316, count=0), so the dev API is not internet-reachable. ARCO_DEBUG=true in that same dev posture (cloud_run.tf:98-101 → config.rs:557 → tasks.rs:352) makes task_auth_middleware skip decode_task_token entirely, so the committed secret adds no attack capability in dev. staging.tfvars and dev.tfvars.example set no task_token_secret, so the default "" leaves the config unconfigured and task_tokens.rs:74-77 rejects all callbacks rather than accepting weak ones. variables.tf:157 marks the variable sensitive (hides plan output only, not state or Cloud Run env).
Caveat / scope: I could not confirm any terraform apply actually landed this value against arco-testing-20260320 — that is not determinable from the repo. I also did not audit whether the CI pipeline sets TF_VAR_task_token_secret from a runner secret for non-dev environments; grep found no such reference under .github/, scripts/, or docs/, but absence there is not proof of the deployed posture. The ARCO_DEBUG header-bypass in tasks.rs:352-367 is a separate weakness I observed while checking mitigations; I have not verified whether it is already tracked, so I am not filing it here.
Deployed-posture review: API ingress is INGRESS_TRAFFIC_INTERNAL_ONLY whenever api_public = false (cloud_run.tf:25), which holds in both dev and staging tfvars; the allUsers run.invoker binding (iam.tf:309-316) is count = 0 in that case. Staging never sets task_token_secret, so variables.tf:152-157 leaves it "" and decode_task_token's validate() rejects all callbacks. No prod tfvars exists in the repo. deploy.sh:231 resolves .tfvars and no dev.tfvars exists, so the committed file requires an explicit --tfvars-file.
Found during a staff-level audit of
origin/main@c3c0867. Mechanism confirmed by an independent adversarial verifier.