Skip to content

Bound Cosmos diagnostics under retry storms#8

Closed
NaluTripician wants to merge 1 commit into
nalutripician/diagnostics-capture-redesignfrom
nalutripician/cosmos-diagnostics-storm-safe
Closed

Bound Cosmos diagnostics under retry storms#8
NaluTripician wants to merge 1 commit into
nalutripician/diagnostics-capture-redesignfrom
nalutripician/cosmos-diagnostics-storm-safe

Conversation

@NaluTripician

Copy link
Copy Markdown
Owner

Make Cosmos diagnostics storm-safe: bounded-size compaction + live fault-injection validation. Follow-up to Azure#4619; stacked on nalutripician/diagnostics-capture-redesign as the base so the diff shows only this change.

Why

With the cheap Threshold gate, a latency/failure spike trips the gate for a large fraction of requests at once, so the expensive diagnostics materialization becomes steady-state exactly when the system is already stressed. And a single operation that retries a hot partition grows its per-attempt list — and its detailed JSON — without bound. This implements the diagnostics contract's bounded-size guarantee and empirically validates the concern.

Deliverable 1 — Retry-storm compaction (always-on model)

  • At operation finalization (DiagnosticsContextBuilder::complete), collapse runs of consecutive near-identical retries (same region / endpoint / status / sub-status / execution-context) into first + last + a count.
  • Bounded by a new configurable DiagnosticsOptions::max_request_diagnostics (with_max_request_diagnostics, env AZURE_COSMOS_DIAGNOSTICS_MAX_REQUESTS, default 512, min 16).
  • An order-robust global key-bucket fallback guarantees the retained count stays within the cap even under a region ping-pong (A→B→A→B), where consecutive run-length collapse alone would not.
  • Lossless for the signal that matters: the summary is computed from the full attempt list before compaction, so the (status, sub-status) histogram, retry/throttle counts, total RU and regions stay exact. request_count() still reports the true total; new retained_request_count() and compaction() (returning CompactionInfo / CompactedRun) expose the bounded count and per-run rollup.
  • Byte-identical when not triggered: detailed JSON gains a top-level compaction object only under a storm (skip_serializing_if). Regions stay normalized-lowercase. Additive and non-breaking.

Deliverable 2 — Threshold-storm validation

  • Deterministic in-crate measurement (storm_materialization_cost_and_size, #[ignore]d): a 100k-retry storm produces ~48 MB / 3.6 s of detailed JSON uncompacted vs 1.8 KB / 421 µs compacted (~27,000× smaller), while the summary still reports the exact retry count.
  • Env + feature-gated live test (tests/live_storm_diagnostics.rs, required-features = ["fault_injection"], #![cfg(feature = "reqwest")]) reads COSMOSDB_MULTI_REGION and injects latency + 429/503/410 storms, measuring gate-fire fraction and per-op materialization cost. Skips gracefully when the account is absent/unreachable; never prints secrets — so CI/playback never runs it.
  • Mitigations recommended back to the contract: structural compaction (this PR), configurable diagnostics depth under stress, sampling + max-events-per-interval cap, lazy/handle materialization.

Guardrails

  • capture_engine untouched / still off by default; no Off mode; CosmosResponse::diagnostics() stays non-optional.
  • Live tests are env + feature gated; no secrets committed.
  • CHANGELOG updated for the additive public API (D1); D2 is internal tests only.

Testing

  • cargo fmt clean; cargo clippy --all-features --all-targets -- -D warnings clean.
  • cargo test -p azure_data_cosmos_driver --all-features: 1999 lib + all integration tests pass (incl. new compaction unit tests, options tests, and the live storm test against the real account).

Add always-on retry-storm compaction to the DiagnosticsContext model plus
live fault-injection validation of threshold-gate materialization cost.
Follow-up to PR Azure#4619.

Deliverable 1 (compaction): at operation finalization, collapse runs of
consecutive near-identical retries (same region/endpoint/status/sub-status/
execution-context) into first + last + a count, bounded by a new
configurable DiagnosticsOptions::max_request_diagnostics cap (env
AZURE_COSMOS_DIAGNOSTICS_MAX_REQUESTS, default 512, min 16). An order-robust
global key-bucket fallback keeps the retained count within the cap even under
a region ping-pong. The summary is computed from the full attempt list before
compaction, so the (status, sub-status) histogram, retry/throttle counts,
total RU and regions stay exact; request_count() still reports the true total,
with new retained_request_count() and compaction() accessors (CompactionInfo /
CompactedRun). Detailed JSON gains a skip-if-absent compaction marker, so
normal-operation output is byte-identical. Additive and non-breaking.

Deliverable 2 (validation): a deterministic in-crate measurement quantifies
the bound (100k-retry storm: ~48 MB / 3.6 s detailed JSON uncompacted vs
1.8 KB / 421 us compacted) and an env+feature-gated live test
(tests/live_storm_diagnostics.rs, reading COSMOSDB_MULTI_REGION) corroborates
the gate-fire behavior and per-op materialization cost with fault injection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the Cosmos label Jul 1, 2026
@NaluTripician

Copy link
Copy Markdown
Owner Author

Superseded by the upstream PR Azure#4683, which targets the real repo (Azure/azure-sdk-for-rust). This fork-internal PR was a placeholder. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant