Gate 5 sole-writer IAM invariant is covered only by #[ignore]d, feature-gated tests behind a CI step that silent...

[ethan-tyler]

Severity: Medium · Category: testing

Affected code

• crates/arco-integration-tests/tests/iam_smoke.rs:55-105
• .github/workflows/ci.yml:325-352
• .github/workflows/adr-034-gcs-conformance.yml:30-65
• crates/arco-core/tests/storage_backend_conformance.rs:505-523

Problem

I inventoried every #[ignore] in the tree. Two are golden-file generators (crates/arco-catalog/tests/schema_contracts.rs:198, crates/arco-flow/tests/orchestration_schema_contracts.rs:147) and are harmless. The rest gate real invariants that nothing in a normal run executes:

• iam_smoke.rs — test_api_cannot_write_state (55), test_api_cannot_write_state_even_if_path_contains_ledger_segment (105), test_api_can_write_ledger (141), test_api_can_write_locks (187), test_api_can_write_commits (226). Per the module header these verify "API cannot write to state/ prefix (Compactor-only)" — the Gate 5 sole-writer invariant. They are #[ignore]d AND behind --features iam-smoke, so cargo test --workspace does not even compile them.
• storage_backend_conformance.rs:506,516 — gcs_backend_satisfies_storage_conformance / s3_backend_satisfies_storage_conformance. These are the only executions of assert_storage_conformance (the CAS/precondition contract the entire publish protocol rests on) against a real backend. The file itself proves the in-process substitutes are inadequate: object_store_local_backend_is_not_a_cas_conformance_substitute (476-503) asserts LocalFileSystem returns NotImplemented for MatchesVersion.

Both CI gates skip green. ci.yml sets skip=true when secrets.GCP_SA_KEY_API or vars.ARCO_BUCKET_DEV is absent and then runs a step that just echoes "Skipping IAM smoke tests" — the job passes. adr-034-gcs-conformance.yml does the same with auth_mode=skip. No workflow anywhere references s3_backend_satisfies_storage_conformance.

Why it matters

The sole-writer prefix isolation and the real-backend CAS semantics are the two assumptions every other catalog test takes for granted — MemoryBackend and object_store::InMemory are both in-process and cannot reproduce GCS generation-precondition behavior. If either invariant regresses (an IAM binding widened, an ObjectStoreBackend::put precondition mapped wrong for GCS), the entire suite still goes green, because the only tests that would catch it are #[ignore]d and their CI gates are configured to succeed while running nothing.

Failure scenario

A change to ObjectStoreBackend::put mishandles GCS's x-goog-if-generation-match response (e.g. returns WriteResult::Success on a 412, or an empty current_version on precondition failure). Every in-process conformance test passes because MemoryBackend and InMemory implement the precondition natively. gcs_backend_satisfies_storage_conformance — the one test that would catch it — never runs if vars.ARCO_TEST_GCS_BUCKET is unset, and the nightly job still reports success. The regression ships and manifests in production as two concurrent pointer publishes both believing they won.

Suggested fix

Make the skip loud rather than green: replace the echo "Skipping ..." steps in ci.yml:349-352 and adr-034-gcs-conformance.yml:62-65 with exit 1 (or ::error:: + a required-status check), so an unconfigured credential is a visible red gate rather than a silent pass. Add a scheduled job for s3_backend_satisfies_storage_conformance, which currently has no runner at all. For Gate 5, add an in-process negative test that does not need cloud credentials — a StorageBackend decorator that denies writes under state/ and an assertion that the API-side writer never attempts one — so the sole-writer invariant has coverage independent of a deployed environment.

Verification notes

Existing mitigations found (do not fully close it): tools/xtask/tests/terraform_iam.rs and tools/xtask/tests/deploy_script.rs:64-131 run in normal CI and assert anchored startsWith() prefix conditions plus the no-list custom writer role, giving declarative coverage of prefix scoping (though not of any api_* binding's exclusion from state/). infra/terraform/iam_conditions.tf grants state_object_prefix only to compactor_write_state:230, never to the API SA. crates/arco-core/src/storage.rs:409-437 (ObjectStoreBackend::put) has no GCS-specific code path, so the always-running object_store InMemory conformance run at storage_backend_conformance.rs:402 does cover Arco-side put/precondition logic. ci.yml:88/115/145 use --all-features, so the iam-smoke tests are compiled and linted in CI even though they never execute.

Caveat / scope: Two corrections to the original write-up. (1) "cargo test --workspace does not even compile them" is inaccurate for CI: ci.yml:145 runs cargo test --workspace --all-features without excluding arco-integration-tests, so iam-smoke compiles; only execution is blocked by #[ignore]. (2) The stated failure scenario (an ObjectStoreBackend::put regression on GCS preconditions) is largely covered in-process, because put is backend-agnostic; the real uncovered surface is the backend-specific error-shape mapping in is_object_store_write_precondition_failure (storage.rs:325-344). Separately, the #[ignore] attribute for test_api_can_write_locks is at line 188, not 187. The GitHub-state evidence (empty variable/secret lists, nightly logs) was gathered via the gh CLI against the live repo on 2026-07-30 and could change if someone configures those variables.

Found during a staff-level audit of origin/main @ c3c0867. Mechanism confirmed by an independent adversarial verifier.