Bound Cosmos diagnostics under retry storms#8
Closed
NaluTripician wants to merge 1 commit into
Closed
Conversation
Add always-on retry-storm compaction to the DiagnosticsContext model plus live fault-injection validation of threshold-gate materialization cost. Follow-up to PR Azure#4619. Deliverable 1 (compaction): at operation finalization, collapse runs of consecutive near-identical retries (same region/endpoint/status/sub-status/ execution-context) into first + last + a count, bounded by a new configurable DiagnosticsOptions::max_request_diagnostics cap (env AZURE_COSMOS_DIAGNOSTICS_MAX_REQUESTS, default 512, min 16). An order-robust global key-bucket fallback keeps the retained count within the cap even under a region ping-pong. The summary is computed from the full attempt list before compaction, so the (status, sub-status) histogram, retry/throttle counts, total RU and regions stay exact; request_count() still reports the true total, with new retained_request_count() and compaction() accessors (CompactionInfo / CompactedRun). Detailed JSON gains a skip-if-absent compaction marker, so normal-operation output is byte-identical. Additive and non-breaking. Deliverable 2 (validation): a deterministic in-crate measurement quantifies the bound (100k-retry storm: ~48 MB / 3.6 s detailed JSON uncompacted vs 1.8 KB / 421 us compacted) and an env+feature-gated live test (tests/live_storm_diagnostics.rs, reading COSMOSDB_MULTI_REGION) corroborates the gate-fire behavior and per-op materialization cost with fault injection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Owner
Author
|
Superseded by the upstream PR Azure#4683, which targets the real repo (Azure/azure-sdk-for-rust). This fork-internal PR was a placeholder. Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make Cosmos diagnostics storm-safe: bounded-size compaction + live fault-injection validation. Follow-up to Azure#4619; stacked on
nalutripician/diagnostics-capture-redesignas the base so the diff shows only this change.Why
With the cheap
Thresholdgate, a latency/failure spike trips the gate for a large fraction of requests at once, so the expensive diagnostics materialization becomes steady-state exactly when the system is already stressed. And a single operation that retries a hot partition grows its per-attempt list — and its detailed JSON — without bound. This implements the diagnostics contract's bounded-size guarantee and empirically validates the concern.Deliverable 1 — Retry-storm compaction (always-on model)
DiagnosticsContextBuilder::complete), collapse runs of consecutive near-identical retries (same region / endpoint / status / sub-status / execution-context) into first + last + a count.DiagnosticsOptions::max_request_diagnostics(with_max_request_diagnostics, envAZURE_COSMOS_DIAGNOSTICS_MAX_REQUESTS, default512, min16).summaryis computed from the full attempt list before compaction, so the(status, sub-status)histogram, retry/throttle counts, total RU and regions stay exact.request_count()still reports the true total; newretained_request_count()andcompaction()(returningCompactionInfo/CompactedRun) expose the bounded count and per-run rollup.compactionobject only under a storm (skip_serializing_if). Regions stay normalized-lowercase. Additive and non-breaking.Deliverable 2 — Threshold-storm validation
storm_materialization_cost_and_size,#[ignore]d): a 100k-retry storm produces ~48 MB / 3.6 s of detailed JSON uncompacted vs 1.8 KB / 421 µs compacted (~27,000× smaller), while the summary still reports the exact retry count.tests/live_storm_diagnostics.rs,required-features = ["fault_injection"],#![cfg(feature = "reqwest")]) readsCOSMOSDB_MULTI_REGIONand injects latency + 429/503/410 storms, measuring gate-fire fraction and per-op materialization cost. Skips gracefully when the account is absent/unreachable; never prints secrets — so CI/playback never runs it.Guardrails
capture_engineuntouched / still off by default; noOffmode;CosmosResponse::diagnostics()stays non-optional.Testing
cargo fmtclean;cargo clippy --all-features --all-targets -- -D warningsclean.cargo test -p azure_data_cosmos_driver --all-features: 1999 lib + all integration tests pass (incl. new compaction unit tests, options tests, and the live storm test against the real account).