Use backoff for 403 topology retries

[tvaron3]

Summary

• replace fixed or immediate retries for standard write-path 403/3 and all 403/1008 responses with exponential backoff and ±25% jitter
• use a cumulative 5-second delay budget with a defensive retry-count cap
• apply the policy to both single- and multi-write accounts; preserve the separate hub-region read 403/3 discovery path
• preserve delayed topology retries across hedging, including mixed-hedge 403/1008 policy/status handling and deadline-bounded sleeps
• make hedged hub-region discovery deterministic across completion orders and limit hedging to the initial topology round
• make account metadata refresh failure-, cancellation-, and concurrency-safe across drivers sharing the runtime cache
• cover exponential growth, delay caps, budget exhaustion, mixed hedges, refresh recovery, hub discovery, and status propagation

Retry policy

Setting	Value
Base delay	1 second
Growth factor	2×
Per-retry cap	15 seconds
Jitter	±25%
Cumulative delay budget	5 seconds
Defensive attempt cap	10

Real-account measurements

Data volume and partition count

A dedicated account was used to create fresh containers with measured physical partition counts and ~3.8 GB per partition. A real region was added and removed while the driver continuously issued point ReadItem calls against the region being removed. No fault injection was used.

Physical partitions	Throughput	Data	Add region	Remove region	Probe reads	Surfaced errors	Transition read
2	10K RU	8 GB	186 s	674 s	1,486	0	286 ms
7	40K RU	27 GB	277 s	677 s	1,475	0	314 ms
10	60K RU	38 GB	246 s	674 s	1,492	0	263 ms

Remove-region time remained flat at ~11.2 minutes across 2–10 partitions and 8–38 GB. In all configurations the driver absorbed region removal transparently: the transition request completed successfully from the remaining region, with no topology error surfaced across 4,453 probe reads.

Reads across every physical partition

A separate single-write, two-region account used a 60K RU container with 10 measured physical ranges. Candidate logical partition keys were mapped by EPK into each range, one probe document was created per range, and the driver round-robin read all 10 probes while East US 2 was removed.

• 8,399 reads total (839–840 per physical partition)
• 211 internal 403/1008 responses distributed across every partition (21–22 each)
• 0 surfaced errors on every partition
• every partition transitioned successfully to West US 3 using two attempts
• transition-read latency: 167–554 ms for 9 partitions; the first partition triggered the shared topology refresh and completed in 1.78 s

This confirms the result was not limited to repeatedly reading one logical partition.

Single-write 403/3 failover A/B

A dedicated single-write, two-region account was exercised with continuous CreateItem operations during real manual failovers.

Policy	Writes	Internal 403/3	Surfaced 403/3	Longest affected operation
Generic 3 immediate retries	389	65	9	1.30 s (failed)
15-second backend budget	280	12	0	8.99 s (succeeded)
10-second backend budget	304	12	0	6.95 s (succeeded)
5-second backend budget	401	12	0	6.35 s (succeeded)

The service convergence window was ~4.8 seconds. The 5-second budget was the lowest tested value that still provided four total attempts (initial + three retries) and absorbed every observed failover error.

Persistent fault behavior

With persistent fault injection against every region, the final 5-second policy terminated predictably:

Signal	Logical rounds	Physical requests	Wall time	Final result
403/1008 read	4	5 (one initial hedge + three sequential retries)	~5–7 s	Original 403/1008 surfaced
403/3 write	4	4	~6.8 s	Original 403/3 surfaced

Validation

• cargo test -p azure_data_cosmos_driver --all-features --lib (2,296 passed; 64 ignored)
• cargo test -p azure_data_cosmos_driver --features __internal_in_memory_emulator,fault_injection --test in_memory_emulator topology_refresh_on_substatus (15 passed)
• cargo test -p azure_data_cosmos_driver --features __internal_in_memory_emulator,fault_injection --test in_memory_emulator hedging (16 passed)
• cargo clippy -p azure_data_cosmos_driver --all-features --tests
• cargo fmt -p azure_data_cosmos_driver

Fixes #4620

[tvaron3]

/azp run rust - cosmos - weekly

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).