[Cosmos] Metadata Hedging: cross-region hedging for Collection and PartitionKeyRange metadata reads

[NaluTripician]

Summary

Adds bounded cross-region hedging for the two metadata cache reads that sit on the critical path of nearly every data operation in the Java SDK:

1. Collection Read (RxClientCollectionCache → container properties)
2. PartitionKeyRange ReadFeed (RxPartitionKeyRangeCache → routing map)

When the primary region is slow on one of these metadata reads, the SDK dispatches a single hedged request to another region after a fixed latency threshold (default 1.5s) and returns the first acceptable answer. This trims the long tail on cold starts (first cache population) and on steady-state cache refreshes (e.g. after a 410/split or lazy expiry) when the primary region is degraded.

This is a Java port of .NET Azure/azure-cosmos-dotnet-v3#5999.

The guiding invariant is "the primary is authoritative." A hedge can only make the operation faster; it can never change the outcome the primary would have produced. A slow, failing, or misconfigured secondary region can never surface a spurious result.

Opt-in

• Follows the account's PPAF (Per-Partition Automatic Failover) state by default (there is no preview/GA package split in Java, so the .NET "preview = on" arm collapses to this).
• AZURE_COSMOS_METADATA_HEDGING_ENABLED environment variable (or the Java-native COSMOS_METADATA_HEDGING_ENABLED env var / COSMOS.metadataHedgingEnabled system property) overrides: true force-enables, false is a hard kill-switch.
• Threshold is configurable via COSMOS.metadataHedgingThresholdInMs / COSMOS_METADATA_HEDGING_THRESHOLD_IN_MS.

Design principle: the primary is authoritative

• A fast, definitive primary error (e.g. 404 for a deleted/recreated collection, 409, 412) is returned verbatim — no hedge is fired, so a stale secondary 200 can never "resurrect" a deleted collection.
• Only a regional failure on the primary is worth hedging: 503, 500, 410, 403 + DatabaseAccountNotFound, transport/network failures, and (for PartitionKeyRange) the regional-lag 404 sub-statuses. This classifier is aligned with the SDK's existing cross-region retry semantics (ClientRetryPolicy), not the narrower metadata throttle policy.
• When neither branch yields a good answer, the primary's outcome is rethrown so the caller's retry policy classifies the real failure exactly as it would have without hedging.
• Cross-region auth failures (401 / plain 403) are not regional, so a misconfigured secondary is treated as a losing hedge.

Architecture

Component	Role
MetadataHedgingStrategy (new, package-private)	Eligibility, threshold timer, primary-vs-hedge race (Reactor), outcome classification, per-branch region pinning, telemetry. One instance per client.
Configs	getMetadataHedgingEnabledOverride() (tri-state) + getMetadataHedgingThreshold().
RxDocumentClientImpl	Constructs the strategy once (GlobalEndpointManager + PPAF manager + opt-in + threshold) and injects it into both caches.
RxClientCollectionCache	Routes the Collection Read through the strategy when present; original path unchanged when null.
RxPartitionKeyRangeCache	Races two region-pinned full routing-map reads; each branch reads all its pages from a single region (via excluded-regions pinning) so the continuation chain stays consistent.
MetadataDiagnosticsContext	New METADATA_HEDGE datum (hedgeFired / hedgeWon / winningRegion), emitted only when a hedge fires — no change to existing no-hedge diagnostics baselines.

Each branch runs on its own request.clone() pinned to a single region (via routeToLocation + excluded regions, mirroring the data-plane getEffectiveExcludedRegionsForHedging), so the primary and hedge never race on shared request context and inner cross-region retry is suppressed.

Scope

Hedged: Collection Read; PartitionKeyRange routing-map read.
Not hedged: data-plane operations (already covered by the availability strategy); address refresh, query plan, database account, and all other metadata reads; single-region accounts (< 2 applicable regions is a no-op).

Backward compatibility

• No public API changes. All new types are package-private/internal; the cache constructor additions are overloaded and default to a null strategy (original path).
• No wire change. Client-side latency optimization over existing gateway metadata reads.
• Off unless eligible. Requires ≥ 2 regions + opt-in/PPAF; single-region and non-eligible reads take the original code path untouched.
• Primary authoritative guarantees no change to the result of any operation — only latency can improve.

Testing

Unit — MetadataHedgingStrategyTest (9, passing): the regional-vs-definitive classifier and the race invariants — primary-fast → no hedge; primary-slow → hedge wins; fast definitive 404 → no hedge; primary 503 → hedge wins; both branches fail → primary's error returned; slow definitive 409 beats an in-flight hedge (primary authoritative).

Integration — MetadataHedgingIntegrationTests (multi-region, 2, passing against a live multi-region account): a primary-region RESPONSE_DELAY is injected on the PartitionKeyRange metadata read while a PARTITION_IS_SPLITTING fault forces a routing-map refresh. With hedging enabled the METADATA_HEDGE datum with hedgeFired=true appears; with it disabled, no hedge fires.

Verification: azure-cosmos full mvn install passes (revapi API-compat clean, javadoc, checkstyle 0 violations); the existing RxPartitionKeyRangeCacheTest still passes (no regression from the constructor change).

Notes

• PartitionKeyRange hedging is realized as region-pinned full-read racing (each branch reads all pages from one region) rather than a literal first-page switchOnFirst, because the Java cache materializes all pages via collectList(); this preserves continuation integrity and is lower-risk while achieving the same latency win.
• AAD-token cross-region validity is not exercised by the tests; if the token is region/audience-scoped an AAD hedge would 401 and safely lose (hedging becomes a no-op for AAD paths). This is called out for follow-up.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[NaluTripician]

Local self-review round (skeptic lens) — applied in 8fd848b

Ran a rigorous local self-review (two independent code-review passes) and cross-checked against the .NET reference (docs/metadata-hedging-simple-design.md from #5999). Three findings, all fixed and verified:

1. Early hedge on a fast primary regional failure (blocking). The hedge previously always waited the full threshold, even when the primary returned a fast regional failure (e.g. 503). The .NET design fires the hedge immediately on a regional failure (primary is slow **or** hit a regional failure -> hedge). Reworked executeHedged to gate the hedge on firstWithSignal(threshold-timer, primary-regional-failure); added a locking unit test (primaryFastRegionalFailure_hedgeFiresBeforeThreshold).
2. PartitionKeyRange hedge branches were not actually region-pinned (blocking). nonDocumentReadFeedInternal never propagated options.getExcludedRegions() to the request context, so both branches resolved to the same region and the ReadFeed hedge was effectively same-region. Wired excluded regions through to request.requestContext (guarded to non-empty; verified LocationCache.resolveServiceEndpoint honors it). Other non-document read-feed callers are unaffected.
3. Null activityId on the PKRange METADATA_HEDGE datum (major/observability). Now emits the probe request's activity id, matching the Collection-read path.

Verification: azure-cosmos full mvn install passes (checkstyle 0 violations); all MetadataHedgingStrategyTest unit tests pass (10/10, including the new early-fire test).

Intentional divergences kept (match .NET v1): active loser cancellation is deferred (loser is abandoned/observed) and the PPAF opt-in is evaluated per request (override ?? live PPAF).

[Copilot]

Pull request overview

This PR introduces bounded cross-region “metadata hedging” for Cosmos Java SDK metadata cache reads (collection lookup and partition key range routing-map readfeed) to reduce tail latency when the primary region is slow or regionally failing. The strategy is constructed once per client, injected into the relevant caches, and emits a new diagnostics datum when a hedge actually fires.

Changes:

• Added MetadataHedgingStrategy and wired it into RxClientCollectionCache and RxPartitionKeyRangeCache from RxDocumentClientImpl.
• Added configuration knobs (Configs) and a new diagnostics datum (METADATA_HEDGE) to surface hedge behavior.
• Added unit + multi-region integration coverage and updated the azure-cosmos changelog entry.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java	Constructs and injects the hedging strategy; ensures excluded-regions are honored for non-document readfeeds.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/MetadataDiagnosticsContext.java	Adds MetadataHedgeDiagnostics and a new METADATA_HEDGE metadata type + serialization.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java	Adds tri-state enable override + configurable hedging threshold.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/caches/RxPartitionKeyRangeCache.java	Implements optional region-pinned hedged routing-map reads and emits hedge diagnostics when fired.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/caches/RxClientCollectionCache.java	Routes collection reads through the hedging strategy and emits hedge diagnostics.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/caches/MetadataHedgingStrategy.java	New core hedging implementation: eligibility, region selection, race semantics, and error classification.
sdk/cosmos/azure-cosmos/CHANGELOG.md	Documents the new cross-region metadata hedging feature and opt-in knobs.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/caches/MetadataHedgingStrategyTest.java	Unit tests for classifier + hedging race invariants.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/faultinjection/MetadataHedgingIntegrationTests.java	Multi-region fault-injection validation for PK-range hedging diagnostics behavior.

[NaluTripician]

Good catch. I removed the PartitionKeyRange-specific 404 classifier entirely (both classifyPartitionKeyRangeThrowable and isPartitionKeyRangeRegionalFailure) and now use the single shared classifyThrowable for both metadata reads. This matches the .NET reference (PR #5999), whose IsRegionalFailure has no 404 handling at all and uses one classifier for both paths. Treating any 404 as definitive (never regional) is actually the correct choice here: it upholds the primary-authoritative invariant, because a lagging secondary returning a stale 200 can never override a primary that legitimately returned 404 (e.g. a deleted/recreated collection or a PK-range gone). The previous variant weakened that guarantee, so removing it is both simpler and safer. Fixed in 5c0462b.

[NaluTripician]

Fixed in 5c0462b. The Collection path now reports the winning branch's actual on-the-wire activityId, read from the winning response's x-ms-activity-id header (hedgeResult.getValue().getResponseHeaders()), falling back to the original request id only when the primary lost with no response (definitive-error path) -- exactly the winning branch's response activityId, fallback to original you suggested. I applied the same correctness fix to the PartitionKeyRange path: each branch now captures its real FeedResponse.getActivityId() and the datum reports the winner's, instead of a throwaway probe id that was never sent.

[NaluTripician]

Fixed in 5c0462b. Updated the class-level javadoc to state that PartitionKeyRange hedging races region-pinned full routing-map reads (each branch reads all its pages from one pinned region), not a first-page-only read -- and noted this is a deliberate divergence from the .NET reference that preserves continuation integrity while achieving the same latency win.

[NaluTripician]

Addressed review comments + CI status (5c0462b)

Copilot review comments (all 3 resolved):

1. PKRange 404 classifier - Removed the PKRange-specific 404 regional classifier and now use the single shared classifyThrowable for both metadata reads, matching the .NET reference. A 404 (any sub-status) is always definitive, so a lagging secondary can never override a primary 404 - this strengthens the primary-authoritative invariant rather than adding the 1003/1013 set (which would weaken it).
2. Hedge telemetry activityId - Both paths now report the winning branch's real on-the-wire activityId (Collection: winning response x-ms-activity-id, fallback to original request id; PKRange: winning branch's FeedResponse activityId) instead of a clone/probe id that was never sent.
3. Class javadoc - Corrected to describe region-pinned full routing-map racing (all pages), not first-page-only.

CI failures (both non-code):

• Build Build (buildId 6531297) was the draft-state build auto-canceled/deleted when the PR was marked ready - stale, not a real failure.
• Test macoslatest_18_NotFromSource_TestsOnly (buildId 6531298) was a native JVM SIGBUS crash in libzip.dylib (newEntry) during Scala compilation on the macOS JDK8 agent (Found 0 errors in compilation; The crash happened outside the Java Virtual Machine in native code) - a known macOS-agent infra flake, unrelated to this change. The fresh CI run triggered by 5c0462b should clear it.

Verification: azure-cosmos full mvn install passes (checkstyle 0 violations); MetadataHedgingStrategyTest 10/10 unit tests pass.