Add Cosmos client-side observability layer

[NaluTripician]

Why

Long-running Cosmos benchmarks and production workloads need observability that is quiet at steady state and rich on error: enough signal to root-cause a short error window in a 10–12h run without paying a steady-state CPU/telemetry tax. Azure's main already ships the always-collected DiagnosticsContext foundation; what was missing is the SDK emission layer that decides what to emit from it, plus bounding of the context under retry storms.

This PR adds that layer. It builds on top of the existing DiagnosticsContext and is additive and off-by-default: the new OpenTelemetry code is behind the otel_metrics / otel_tracing features, and with no handler registered the completion path is a zero-overhead no-op, so merging this cannot change existing behavior.

What

A pluggable DiagnosticsHandler chain on azure_data_cosmos, three built-in handlers, and driver-side support for thresholds and bounded diagnostics. Register handlers via CosmosClientBuilder::with_diagnostics_handler.

• Handler chain — DiagnosticsHandler trait + ordered DiagnosticsHandlerChain, invoked once per operation at completion; empty chain = zero overhead.
• OpenTelemetry metrics (otel_metrics) — CosmosMetricsHandler emits the stable db.client.operation.duration histogram, plus opt-in development-tier metrics.
• Tail-sampled tracing (otel_tracing) — CosmosTracingHandler reconstructs a backdated span tree only for failures / threshold breaches; fast successful reads emit nothing.
• Rate-limited sampling log — SamplingLogHandler logs a compact JSON line for failed/slow operations, capped per window.
• Bounded diagnostics — driver DiagnosticsOptions::max_request_diagnostics caps the per-attempt list under a retry storm while keeping counts and charges exact and surfacing every truncation.
• Op-scope wiring — a CosmosOperationContext (operation name + database + container) is attached at each completion call site so handlers carry real db.* attributes, built lazily to keep the empty-chain path allocation-free.

A per-workstream breakdown and the full verification matrix are in this comment.

Verification

cargo fmt --check, cargo clippy --all-targets, and cargo test are clean on the touched crates across default, otel_metrics, and otel_tracing; cspell is clean. (--all-features skipped locally: it fails on openssl-sys, an unrelated Windows/env issue.)

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds client-side Cosmos observability with pluggable handlers, OpenTelemetry emission, sampled logging, and bounded retry diagnostics.

Changes:

• Adds diagnostics handler registration and operation context APIs.
• Adds feature-gated metrics/tracing handlers and sampled logging.
• Adds diagnostics thresholds, compaction, and exact aggregate counters.

Reviewed changes

Copilot reviewed 34 out of 35 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
sdk/cosmos/azure_data_cosmos/src/options/client.rs	Stores registered handlers.
sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs	Builds backdated span trees.
sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/mod.rs	Exports and tests tracing.
sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/handler.rs	Implements tracing handler.
sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/attributes.rs	Defines tracing attributes.
sdk/cosmos/azure_data_cosmos/src/diagnostics/operation_context.rs	Adds operation identity context.
sdk/cosmos/azure_data_cosmos/src/diagnostics/mod.rs	Exports diagnostics APIs.
sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/options.rs	Configures optional metrics.
sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/mod.rs	Defines metrics module.
sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/instruments.rs	Creates OTel instruments.
sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/handler.rs	Records operation metrics.
sdk/cosmos/azure_data_cosmos/src/diagnostics/metrics/attributes.rs	Defines metric contracts.
sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/rate_limiter.rs	Limits sampled logs.
sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/mod.rs	Exports and tests logging.
sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs	Implements sampled logging.
sdk/cosmos/azure_data_cosmos/src/diagnostics/handler.rs	Adds handler trait and chain.
sdk/cosmos/azure_data_cosmos/src/diagnostics.rs	Replaced by module directory.
sdk/cosmos/azure_data_cosmos/src/clients/mod.rs	Adds singleton completion dispatch.
sdk/cosmos/azure_data_cosmos/src/clients/database_client.rs	Adds database operation identities.
sdk/cosmos/azure_data_cosmos/src/clients/cosmos_client.rs	Adds account operation identity.
sdk/cosmos/azure_data_cosmos/src/clients/cosmos_client_builder.rs	Registers handler chains.
sdk/cosmos/azure_data_cosmos/src/clients/container_client.rs	Adds container operation identities.
sdk/cosmos/azure_data_cosmos/CHANGELOG.md	Documents observability APIs.
sdk/cosmos/azure_data_cosmos/Cargo.toml	Adds OTel features and tests.
sdk/cosmos/azure_data_cosmos_driver/src/options/mod.rs	Exports diagnostics thresholds.
sdk/cosmos/azure_data_cosmos_driver/src/options/diagnostics_thresholds.rs	Defines sampling thresholds.
sdk/cosmos/azure_data_cosmos_driver/src/options/diagnostics_options.rs	Adds request diagnostics cap.
sdk/cosmos/azure_data_cosmos_driver/src/lib.rs	Re-exports thresholds.
sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/mod.rs	Exports compaction metadata.
sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs	Integrates compaction and predicates.
sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/compaction.rs	Implements retry-storm compaction.
sdk/cosmos/azure_data_cosmos_driver/CHANGELOG.md	Documents driver APIs.
sdk/cosmos/azure_data_cosmos_driver/ARCHITECTURE.md	Describes diagnostics bounding.
sdk/cosmos/.cspell.json	Adds observability terminology.
Cargo.lock	Locks OTel dependencies.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated 13 comments.

[Copilot]

Pull request overview

Copilot reviewed 38 out of 38 changed files in this pull request and generated 8 comments.

Comments suppressed due to low confidence (8)

sdk/cosmos/azure_data_cosmos/src/diagnostics/handler.rs:65

• This parameter is not actually the pipeline/trace context described here. Every completion path constructs a fresh Context::new() containing only CosmosOperationContext (clients/mod.rs:139 and feed/mod.rs:67), so custom handlers cannot observe caller/pipeline values or use this argument to correlate telemetry with the caller trace. Either carry the real operation context through execution or describe/pass a dedicated diagnostics context rather than promising trace correlation.

    /// * `cx` - The pipeline/trace [`Context`] associated with the operation,
    ///   available so handlers can correlate emitted telemetry with the caller's
    ///   trace context.

sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/handler.rs:77

• This public predicate can disagree with what this handler actually does. Production contexts lack an operation name, so for a 2-second query_items context this call uses the 1-second fallback and returns true, while handle receives CosmosOperationContext("query_items"), uses the 3-second non-point threshold, and skips the span (the test at tracing/mod.rs:120-142 demonstrates exactly this distinction). Accept the operation context/name here or avoid exposing this as the handler's decision.

    /// Returns whether the given completed context should emit a span, per the
    /// tail-based sampling policy: emit iff the operation failed or crossed a
    /// threshold.
    pub fn should_emit(&self, diagnostics: &DiagnosticsContext) -> bool {
        should_emit_span(diagnostics, &self.thresholds, None)

sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs:94

• This public predicate has the same mismatch as the tracing predicate: it always passes no operation identity, while handle uses the SDK context. A 2-second query therefore returns true here but is not logged by the registered handler because the actual non-point threshold is 3 seconds. Accept the operation identity/name or avoid presenting this as the handler's effective decision.

    /// Returns whether the given completed context should be logged, per the
    /// tail-based sampling gate (before rate limiting).
    pub fn should_log(&self, diagnostics: &DiagnosticsContext) -> bool {
        should_log(diagnostics, &self.thresholds, None)

sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:1945

• When any source was already compacted, this re-compacts only its retained first/last exemplars and discards its exact source rollups. For example, a source run representing 600 attempts reaches compact_requests here as two records, so the aggregate runs reports count 2 and the wrong RU/P50 even though the PR promises exact per-run rollups. Re-bound/merge the source CompactionInfo::runs rather than recomputing rollups from already-compacted exemplars.
  sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:1975
• This assigns all collapsed middle attempts to omitted_request_count, even though their runs are present. In the new 600+400 regression test, both exact source runs are retained and omitted_runs == 0, but this reports 996 omitted requests; CompactionInfo documents this field as attempts represented by omitted runs. Sum the source omission counts instead.
  sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:2157
• After Phase 2 compaction, requests.last() is not guaranteed to be the terminal attempt: global buckets are emitted in first-seen key order and then truncated. Thus a status-less error whose final failed key is dropped can become is_failure() == false, suppressing exactly the failure span/log this fallback was added to preserve. Store the terminal outcome before compaction (or guarantee that the actual terminal attempt is retained) instead of inferring it from the compacted list.
  sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs:142
• A failure inferred from the terminal request has op_failed == true but no operation-level status, so this block adds no error.type even though the span is marked Error below. These status-less failure paths are explicitly supported by DiagnosticsContext::is_failure; emit the terminal status or _OTHER so failed root spans satisfy the error semantic convention.

    if op_failed {
        if let Some(status) = diagnostics.status() {
            root_attrs.push(KeyValue::new(
                attributes::ERROR_TYPE,
                u16::from(status.status_code()).to_string(),

sdk/cosmos/azure_data_cosmos_driver/DIAGNOSTICS-CONTRACT.md:423

• This shipped-contract table overstates the implementation. The root builder names spans only with <operation> (not <operation> <target>), and emit_backdated_span_tree emits neither request timeline events nor a hedge span. Implement these rows or mark them deferred so the contract accurately describes the telemetry users receive.

[analogrelay]

@NaluTripician can you compact the PR description? Feel free to move other content to a doc or comment.

[analogrelay]

Done an initial review, but not yet finished some of the larger files. Posting some of my comments now before I have to step out for a bit. Will plan to get back to it ASAP!

[analogrelay]

Surprised the driver doesn't have this context. Doesn't it have these on CosmosOperation?

[NaluTripician]

It has some of it. CosmosOperation carries the ContainerReference (database + container names) and an OperationType / ResourceType, but two gaps make CosmosOperationContext worthwhile: (1) the handler chain receives the finalized DiagnosticsContext, not the CosmosOperation — the driver's context deliberately doesn't embed the caller-facing identity; and (2) the semconv db.operation.name (read_item vs query_items) is an SDK-level name, finer-grained than the driver's coarse OperationType. The alternative is threading the identity onto the driver's DiagnosticsContext, but that pushes SDK naming concerns into the driver. Happy to go that way if you'd prefer.

[analogrelay]

Should this be a wrapping handler rather than writing directly to tracing? That would be a little more composable IMO. A SamplingLogHandler would hold an Arc<dyn DiagnosticsHandler> and apply the rate limits, only calling the child handler when the rate limits and thresholds allow.

Then, we would have a separate TracingLogHandler which logs to tracing and could theoretically have handlers that emit to other outputs.

I'm totally fine with us have a "default" configuration somewhere that configures SamplingLogHandler wrapping TracingLogHandler so that users don't need to think about it, but I think we should allow them the flexibility.

[NaluTripician]

Agreed this is more composable. Plan: a SamplingLogHandler that holds an Arc<dyn DiagnosticsHandler> and only calls the child when the thresholds + rate limit allow, plus a separate TracingLogHandler that writes to tracing, and a default that wires SamplingLogHandler around TracingLogHandler. It's a public-API reshape, so I'd like to land it as a focused follow-up once we agree on how thresholds vs rate-limiting split across the two — tracking it. (Made RateLimiterConfig public in the meantime, per the sibling comment.)

[analogrelay]

If you're not addressing it in this PR, can you make an issue and parent it to #4822 ?

[analogrelay]

Some duplication here. Maybe just have a top level src/diagnostics/attributes.rs? It's fine if some are dead code when certain features are/are not enabled.

[NaluTripician]

Agreed there's duplication. Happy to hoist a shared src/diagnostics/attributes.rs with #[allow(dead_code)] for the feature-conditional names. Tracking as a follow-up so it lands as one clean move.

[analogrelay]

Again, would like to see these closer to the actual code-under-test unless there are really multiple separate pieces being tested together here.

[NaluTripician]

Same as the logging tests — these span the sampling predicate (handler) and the backdated span-tree builder (span_builder) together, so they're integration-level. Can split the pure predicate checks out if you'd prefer.

[analogrelay]

We should avoid otel in the feature name. In fact, I might suggest splitting this up into separate metrics, tracing, and opentelemetry features. Especially since the tracing one shouldn't need a direct dependency on opentelemetry

[NaluTripician]

Agree on dropping otel. One snag: a feature literally named tracing collides with the non-optional tracing crate dependency, so Cargo rejects it as-is. Options: make the tracing dep optional and gate it, or names like metrics + distributed_tracing, or metrics / tracing sitting on an underlying opentelemetry feature. Which naming do you prefer? Agree the metrics/tracing split shouldn't force an OTel dep on both — I'll wire it up once we pick names.

[analogrelay]

distributed_tracing works for me.

[analogrelay]

Not really useful tests, but I'm not gonna make you remove them :D

[NaluTripician]

😄 thanks — leaving them in.

[NaluTripician]

Cosmos observability layer — full detail

Moved out of the PR description (per review) for reference. This is the per-workstream breakdown and the full verification matrix.

Per-workstream breakdown

• WS2 — handler chain. DiagnosticsHandler trait + ordered DiagnosticsHandlerChain, invoked exactly once per operation at completion. Empty chain = zero overhead. Additive, swappable surface (Cosmos-local now, a candidate for promotion to azure_core later).
• WS3 — OpenTelemetry metrics (otel_metrics). CosmosMetricsHandler emits the stable db.client.operation.duration histogram with semconv attributes, plus opt-in development-tier metrics (request_charge, returned_rows, active_instance.count) via MetricsOptions. No meter provider registered ⇒ a no-op.
• WS4 — tail-sampled tracing (otel_tracing). CosmosTracingHandler reconstructs a backdated span tree (operation span + one child per retained attempt) after the operation completes, so the emit/skip decision is made from the finished context. A fast, successful point read emits nothing; failures and threshold breaches emit a full span tree. Uses the raw opentelemetry SpanBuilder start/end-time API (the azure_core backdating trait additions are the separate core-crate PR Add span backdating to core tracing traits #4784 and are not included here).
• WS5 — rate-limited sampling log. SamplingLogHandler logs a compact to_json_string line through tracing, only for failed/threshold-breaching operations, capped per window (~100/min) with a bounded failure allowance and exactly one "suppressed N" notice per window.
• WS6 — bounded diagnostics under retry storms. Driver DiagnosticsOptions::max_request_diagnostics caps the finalized per-attempt list so a 429/410 storm cannot grow a DiagnosticsContext without bound. Compaction collapses runs of near-identical retries to first+last with exact per-run rollups; request_count() / total_request_charge() stay exact and every truncation is surfaced explicitly via retained_request_count() / compaction().
• WS8 — op-scope wiring. The driver context knows what happened on the wire but not the caller-facing operation identity. The SDK now populates a CosmosOperationContext (operation name + database + container) at each of the ~14 completion call sites and attaches it to the pipeline Context, so the metrics and tracing handlers carry real db.operation.name / db.namespace / db.collection.name in production. It is built lazily (only when a handler is registered), keeping the empty-chain path allocation-free.

Also folds in the small driver additions the emission layer depends on and that upstream lacked: DiagnosticsThresholds and the is_failure / is_completed / is_threshold_violated / operation_name predicates/accessors.

Verification

Ran on the touched crates (--all-features skipped: it fails locally on openssl-sys, an environmental Windows issue unrelated to these changes):

• cargo fmt --check — azure_data_cosmos and azure_data_cosmos_driver, clean.
• cargo clippy --all-targets — driver, and azure_data_cosmos with default, otel_metrics, and otel_tracing — 0 warnings.
• cargo test — driver, and azure_data_cosmos (lib + doctests) with default, otel_metrics, and otel_tracing — all pass, including the metrics in-memory-meter, backdated-span, op-context-fallback, rate-limiter storm, retry-storm compaction, and threshold/predicate tests.
• cspell — clean over all changed files.

[NaluTripician]

@analogrelay done — compacted the description down to Why / What / Verification and moved the per-workstream breakdown plus the full verification matrix into #4789 (comment). Also pushed 74c29b7 addressing the open review threads (region first-contact ordering + compaction survival, tracing completion gate, server-address override, aggregate root-span window, feed zero-overhead guard, and throughput/offers/poller dispatch).

[Copilot]

Pull request overview

Copilot reviewed 40 out of 41 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (7)

sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:2190

• Global-bucket compaction does not preserve chronological order and may omit the terminal attempt, so requests.last() is not reliably the terminal status. For a status-less error after an alternating-key retry storm, the retained list can end in an earlier successful attempt and is_failure() becomes false, suppressing exactly the error telemetry this fallback was added for. Preserve the terminal status from the full list before compaction and use that for the fallback.
  sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:1970
• When compacted source contexts collectively exceed the cap, this recompacts only their retained exemplars and discards their exact source CompactedRun rollups. The new runs counts/charges/durations therefore describe only exemplars, while omitted_request_count = original - retained also counts collapsed attempts that are not from omitted runs, contradicting that field's contract. Merge and re-bound the source rollups rather than rebuilding them from retained records.
  sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs:165
• is_failure() intentionally recognizes status-less failures from the terminal attempt, but this block emits error.type only from the absent operation status. Those sampled root spans are marked Error yet violate this PR's contract that failed spans carry error.type; use the effective terminal status or the same _OTHER fallback as the metrics handler.

    if op_failed {
        if let Some(status) = diagnostics.status() {
            root_attrs.push(KeyValue::new(
                attributes::ERROR_TYPE,
                u16::from(status.status_code()).to_string(),
            ));
        }
    }

sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/handler.rs:77

• This public predicate can disagree with handle: production contexts lack an operation name, so should_emit applies the 1-second point fallback, while handle uses CosmosOperationContext and applies 3 seconds to queries/batches. For example, it reports true for a 2-second query that the handler will skip. Accept an operation name/context (or clearly expose a separate context-free fallback predicate) so this API answers the behavior it advertises.

    /// Returns whether the given completed context should emit a span, per the
    /// tail-based sampling policy: emit iff the operation failed or crossed a
    /// threshold.
    pub fn should_emit(&self, diagnostics: &DiagnosticsContext) -> bool {
        should_emit_span(diagnostics, &self.thresholds, None)

sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/handler.rs:94

• This public predicate does not model the handler's actual decision for non-point operations: it uses the 1-second fallback because production diagnostics lack an operation name, whereas handle reads the operation context and uses the 3-second threshold. Accept operation identity or expose/document this as a context-free fallback to avoid returning the opposite of what the handler will do.

    /// Returns whether the given completed context should be logged, per the
    /// tail-based sampling gate (before rate limiting).
    pub fn should_log(&self, diagnostics: &DiagnosticsContext) -> bool {
        should_log(diagnostics, &self.thresholds, None)

sdk/cosmos/azure_data_cosmos_driver/DIAGNOSTICS-CONTRACT.md:423

• The shipped span builder never reads RequestDiagnostics::events() or hedge_diagnostics() and emits neither timed events nor a distinct hedge span. Because this document presents itself as the contract implemented by this PR, these rows overstate the tracing output; mark them deferred or implement the described emission.
  sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs:89
• The contract requires a database span name containing the operation and target (for example, read_item orders), but this always uses only the operation name even when the container is available in CosmosOperationContext. Include the appropriate collection/namespace target so emitted spans follow the documented semantic convention.

    let op_name_ref = diagnostics
        .operation_name()
        .or_else(|| op.and_then(CosmosOperationContext::operation_name));
    let op_name = op_name_ref
        .unwrap_or(DEFAULT_OPERATION_SPAN_NAME)
        .to_string();

[Copilot]

Pull request overview

Copilot reviewed 40 out of 41 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (7)

sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:1970

• When any source was already compacted, aggregated_requests contains only retained exemplars. Re-running compaction over those records rebuilds runs with exemplar-only counts, charges, and percentiles, discarding the exact source rollups; omitted_request_count then counts all missing exemplars even when no rollup row was omitted. Preserve and merge the source CompactionInfo rollups when re-bounding the aggregate so these public fields remain internally consistent and exact.
  sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:1992
• This branch preserves every source rollup row, so omitted_runs can be zero, but it sets omitted_request_count to all attempts not retained as full records (996 in the existing 600+400 test). That contradicts the field contract that it counts attempts represented by omitted rollup rows. Sum the sources' omitted_request_count values instead.
  sdk/cosmos/azure_data_cosmos/src/feed/iterator.rs:239
• Taking an owned operation context here forces query callers to construct it before the iterator can check whether the handler chain is empty. Container/database call sites therefore clone names and allocate strings even with no registered handler, contradicting the off-by-default allocation-free path. Accept a lazy factory or an optional context created only when diagnostics is non-empty.

        diagnostics: DiagnosticsHandlerChain,
        op_context: CosmosOperationContext,

sdk/cosmos/azure_data_cosmos/src/feed/change_feed_iterator.rs:267

• The change-feed iterator also requires an eagerly-built operation context, so ContainerClient::query_change_feed allocates database/container names even when the diagnostics chain is empty. Make this context lazy or optional so the advertised no-handler path remains allocation-free.

        diagnostics: DiagnosticsHandlerChain,
        op_context: CosmosOperationContext,

sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:2190

• This fallback assumes the last retained record is the terminal attempt, but global-bucket compaction groups records by first-seen key and may truncate later buckets. In a compacted error path with no operation status, requests.last() can therefore be a non-terminal success and suppress the failure span/log—the exact storm scenario this layer targets. Capture the effective terminal status from the full request list before compaction and store/use it on the finalized context.
  sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs:113
• For failures whose operation-level status was not stamped, the root span omits db.response.status_code and sub-status here (and error.type below), even though the driver recognizes the operation as failed. Use an effective final status preserved before compaction for all root status/error attributes.

    if let Some(status) = diagnostics.status() {

sdk/cosmos/azure_data_cosmos_driver/DIAGNOSTICS-CONTRACT.md:423

• This contract promises behavior the implementation does not provide: root names contain only the operation name (not <operation> <target>), request timelines are not emitted as events, and hedges are represented only as ordinary request spans rather than a hedge span. Because this document defines the shipped contract, either implement these span shapes or narrow these rows to the actual emitted tree before release.

[NaluTripician]

Good nuance. The manual backdated span tree needs the root to stay a mutable, owned Span so it can be ended with an explicit end_with_timestamp after its children — which fights Context::with_span / mark_span_as_active (both take ownership and hand back only an immutable SpanRef). with_remote_span_context is the only public way to set an explicit parent SpanContext while keeping the owned root. Parent/child linkage is correct; the only exposure is parent-based samplers that treat local vs remote parents differently. I'd like to leave this as a tracked follow-up rather than restructure the backdating flow here — flag if you'd prefer I take it on in this PR.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 43 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (6)

sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs:148

• The fallback contradicts the comment and the metrics handler: metrics uses the last contacted endpoint, while this root span uses the first. After a retry or regional failover, the same operation therefore reports different server.address values in metrics and tracing. Use the terminal request here as well; child spans still retain each attempt's actual address.

    }
    root_attrs.push(KeyValue::new(
        attributes::AZURE_COSMOSDB_REQUEST_CHARGE,
        diagnostics.total_request_charge().value(),

sdk/cosmos/azure_data_cosmos/src/diagnostics/tracing/span_builder.rs:89

• The emitted root span name omits the target even when container_name is available. This conflicts with the PR's diagnostics contract (<operation> <target>) and the OpenTelemetry database convention ({db.operation.name} {target}), producing read_item instead of e.g. read_item orders. Include the low-cardinality container target in the span name and update the span-name assertions.

    // operation's `duration()` already reaches back to before its first attempt,
    // but an aggregate operation's `duration()` is the SUM of its sub-op
    // durations (see `DiagnosticsContext::aggregate_sub_operations`) and omits

sdk/cosmos/azure_data_cosmos_driver/src/diagnostics/diagnostics_context.rs:1993

• Recompacting aggregated_requests loses exact source rollups because this vector contains only each already-compacted source's retained exemplars. If multiple compacted PATCH sub-operations contribute more than cap exemplars, the new run counts, RU totals, and P50 values are recomputed from those exemplars (often two records per original hundreds-attempt run), so CompactionInfo::runs is no longer exact. Rebound while merging the source CompactionInfo statistics rather than treating retained samples as the full attempts.
  sdk/cosmos/azure_data_cosmos/src/feed/iterator.rs:239
• Requiring a fully built operation context here makes every query call construct it before the iterator knows whether the handler chain is empty. Database/container query call sites consequently clone database/container strings on the default no-handler path, contradicting the PR's allocation-free empty-chain guarantee. Store an optional dispatch bundle or lazy factory and construct the context only when the chain is non-empty.

        options: OperationOptions,
        diagnostics: DiagnosticsHandlerChain,
        op_context: CosmosOperationContext,

sdk/cosmos/azure_data_cosmos/src/feed/change_feed_iterator.rs:267

• As with the query iterator, this by-value context forces query_change_feed to clone the database and container names even when no diagnostics handler is registered. That adds steady-state allocations to the supposedly allocation-free default path. Make diagnostics state optional or lazy so the operation context is only created for a non-empty chain.

        options: OperationOptions,
        diagnostics: DiagnosticsHandlerChain,
        op_context: CosmosOperationContext,

sdk/cosmos/azure_data_cosmos/src/diagnostics/logging/rate_limiter.rs:40

• window accepts Duration::ZERO, but check then rolls the window on every call (elapsed >= 0) and resets emitted, allowing every event through and completely bypassing storm protection. Validate a non-zero window at this public configuration boundary or define zero with a safe explicit behavior.

    /// Maximum number of emissions allowed per [`window`](Self::window).
    pub max_per_window: u32,
    /// Length of a rate-limiting window.
    pub window: Duration,
    /// Number of failures permitted per window *in addition to* the normal cap,
    /// so a bounded number of failures is always logged during a storm.
    pub failure_reserve: u32,

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated 6 comments.