No production component ever publishes the metastore storage-governance projection, so credential vending can on...

[ethan-tyler]

Severity: Medium · Category: reliability

Affected code

• crates/arco-catalog/src/metastore/publish.rs:240
• crates/arco-catalog/src/metastore/projections.rs:209
• crates/arco-catalog/src/metastore/publish.rs:174-180
• crates/arco-catalog/src/metastore/publish.rs:358-380
• crates/arco-uc/src/routes/credentials.rs:150-154

Problem

publish_metastore_projection_set (publish.rs:240) and its input builder build_projection_set (crates/arco-catalog/src/metastore/projections.rs:209) have no callers anywhere outside crates/arco-catalog/tests/metastore_replay_publication.rs — verified by grepping the whole tree, including crates/arco-api/src/bin/, crates/arco-compactor/src/, and crates/arco-flow. Nothing writes manifests/metastore_projection.pointer.json.

Meanwhile the enforcement read path requires that pointer: PublishedStorageGovernanceCache::load calls load_projection_manifest(storage) first (publish.rs:175), and every credential route goes through it (state.storage_governance_cache.load(&storage), crates/arco-uc/src/routes/credentials.rs:150-154 and 280-285), mapping failure to ServiceUnavailable{ "credential_scope_unavailable:..." } (credentials.rs:378-383).

Even if a publisher existed, validate_storage_governance_manifest_freshness (publish.rs:358-380) demands manifest.ledger_watermark_sequence == latest.sequence && manifest.ledger_watermark == latest.event_id exactly — so a single new metastore event (e.g. one POST /external-locations) invalidates the projection until it is republished, with no background republisher to do so.

Why it matters

The external-location / storage-credential / workspace-binding metadata written by the UC routes into the metastore ledger is never projected into the state that credential vending reads, so the whole storage-governance feature is inert in any deployed environment: POST /temporary-table-credentials and POST /temporary-path-credentials cannot succeed. The cross-tenant/path-scoping logic in credential_vending.rs and storage_governance/mod.rs is therefore untested against real traffic.

Failure scenario

Fresh deployment, unity_catalog.enabled = true. Admin creates a storage credential and an external location via the UC routes (these append MetastoreEvents to ledger/metastore/). A client then calls POST /temporary-path-credentials for a URL under that external location. storage_governance_cache.load -> load_projection_manifest finds no manifests/metastore_projection.pointer.json -> credential_projection_error -> HTTP 503 credential_scope_unavailable. No sequence of API calls can move the system out of this state because no code path publishes the projection.

Suggested fix

Add a metastore projection publisher to a deployed binary — e.g. in arco-compactor alongside the existing tier-2 consolidation loop, or synchronously at the end of MetastoreLedger::append_event's callers in the UC governance routes: replay the ledger, build_projection_set(&state, &registry, &watermark), then publish_metastore_projection_set(&storage, &set, watermark_sequence). Because validate_storage_governance_manifest_freshness requires exact watermark equality, publication must be driven by (or immediately follow) every metastore ledger append, not a periodic timer alone.

Verification notes

Existing mitigations found (do not fully close it): Fail-closed by design (503, never an incorrect credential vend), asserted as intended by crates/arco-uc/tests/credentials_authoritative.rs:93-124. UC facade disabled by default (crates/arco-api/src/config.rs:471-476) and gated at crates/arco-api/src/server.rs:826; ARCO_UNITY_CATALOG_ENABLED is not set anywhere in infra/. Endpoints already declared partial with planned gaps in crates/arco-uc/src/support.rs:206-219 and return no provider token material.

Caveat / scope: The claim's mechanism and line numbers are correct and I could not refute them, but its blast-radius framing ("inert in any deployed environment") overstates reachability: the UC facade is off by default and enabled by no infra config in this tree. The exact-watermark freshness rule (publish.rs:364-365) is real but is a forward-looking concern that only bites once a publisher exists.

Found during a staff-level audit of origin/main @ c3c0867. Mechanism confirmed by an independent adversarial verifier.

[ethan-tyler]

Scope caveat from the deployed-posture reviewer: the mechanism is confirmed — publish_metastore_projection_set and build_projection_set have no src/ callers in arco-api, arco-compactor, or arco-flow, and the read path hard-requires the pointer, so the "503 forever" behaviour is correct as written.

However, the Unity Catalog facade is disabled by default (UnityCatalogApiConfig::default() has enabled = false), so no deployed environment currently reaches the credential-vending routes. This is an incomplete feature rather than a live outage.

Two points that keep it worth tracking: the failure mode is fail-closed (503, never an incorrect credential vend — asserted intentionally by crates/arco-uc/tests/credentials_authoritative.rs:93-124), which is the right direction; but the cross-tenant path-scoping logic in credential_vending.rs and storage_governance/ is currently unreachable code that has never run in a deployed environment, so enabling UC would activate a substantial amount of untested-in-production authorization logic at once.