Throwaway OTel retroactive-span diagnostics spike

[NaluTripician]

Throwaway OpenTelemetry retroactive-span feasibility spike

Companion to the diagnostics contract doc PR (#4684). This is the isolated, throwaway spike that backs up §7–§8 of DIAGNOSTICS-CONTRACT.md — kept on its own branch so the contract doc stands alone.

What it proves

A completed DiagnosticsContext can be reconstructed into a backdated OpenTelemetry span tree using the raw opentelemetry SpanBuilder::with_start_time/with_end_time — timestamps the azure_core::tracing abstraction does not expose (it builds spans at "now"). The recorded attempts are laid on an explicit, clearly-backdated (2020) timeline with injected durations, and the operation → attempt parent/child relationships are reconstructed and asserted against an in-memory exporter.

Decoupled from the deferred #4619 capture engine

• In-crate #[cfg(all(test, feature = "otel_spans_spike"))] module so it can use the pub(crate) DiagnosticsContextBuilder.
• Builds the sample context from main types only — crate::diagnostics / driver::routing / options / models — with no dependency on the diagnostics::capture prototype.
• Off by default; pulls opentelemetry + opentelemetry_sdk (in-memory exporter) only behind the otel_spans_spike feature.

Verification

• cargo fmt ✅ · cargo clippy --all-features --all-targets -- -D warnings ✅ clean · the spike test ✅ passes.
• Throwaway/feasibility only; no public API change; no CHANGELOG. Draft (design review).

4 files, +231: the in-crate test module, the otel_spans_spike feature + optional deps, and the module declaration.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[NaluTripician]

Nice throwaway — it cleanly proves the backdated SpanBuilder::with_start_time / with_end_time path works without touching the capture engine, and the in-memory-exporter assertions on parent/child + backdated timestamps are convincing. Two things to keep in mind so this doesn't get carried into real wiring as-is:

Latent coupling to #4683's request_count() change

The test builds one span per element of ctx.requests() (the retained list) but asserts spans.len() == ctx.request_count() + 1. That holds only because compaction isn't on this branch; once #4683 lands, request_count() returns the pre-compaction total while requests() is the retained (bounded) list, so this assertion breaks. If any of this becomes production span emission, count/iterate off the same list.

Contract §7.3 gap

Same "one span per attempt" assumption — under compaction a run collapses to first+last, so a faithful emitter can't produce request_count() spans. Fine for the spike; flagging so it's not baked into the eventual emitter.

Feature / dep scoping

otel_spans_spike = ["dep:opentelemetry", "dep:opentelemetry_sdk"] enables the deps unconditionally while the only consumer is #[cfg(all(test, feature = "otel_spans_spike"))]. A non-test --all-features build pulls opentelemetry + opentelemetry_sdk (with the testing feature) into the graph with no user. Harmless for a throwaway, but worth a note if it ever graduates.

[NaluTripician]

Addressed in 77db3bb:

• request_count() coupling — span count/iteration is now anchored on ctx.requests() (the retained list the spans are built from); the two span-count assertions use
  equests.len() instead of ctx.request_count(), so they stay correct once Bound Cosmos diagnostics under retry storms #4683's compaction makes
  equest_count() the pre-compaction total. Added an explanatory comment at the emit site.
• Contract §7.3 gap — added a note that under compaction a run collapses to first+last (one span + a repeat count), so a faithful emitter cannot produce
  equest_count() spans.
• Feature/dep scoping — extended the otel_spans_spike comment in Cargo.toml to call out that the deps are enabled unconditionally while the only consumer is the test-gated module (a non-test --all-features build pulls them in unused), with a pointer to move them behind a non-test consumer / dev-dependency if this ever graduates.

cargo fmt + cargo clippy --features otel_spans_spike --all-targets clean; the spike test still passes.

[analogrelay]

Shall we throw this away now, @NaluTripician ?

[NaluTripician]

Superseded. This throwaway spike proved backdated OTel spans are feasible from a completed DiagnosticsContext; that technique is now productionized in two places: #4784 adds the additive start_span_at/end_at backdating methods to the core azure_core/typespec_client_core tracing traits, and #4789 (WS4) implements the tail-sampled, backdated Cosmos tracing handler (via the raw opentelemetry SpanBuilder path). Closing the spike.