[FEATURE] Auxiliary LLM calls (summarization, HITL classifier, routing classifier, memory extraction) bypass usage metrics — token spend is invisible or orphaned #3863

SDK Language

Python (TypeScript should be checked for the same pattern as these features are ported)

Problem Statement

Token usage is only systematically tracked for main event-loop model calls: EventLoopMetrics.update_usage is invoked solely from event_loop.py:603,652, which feeds accumulated_usage, per-invocation usage, and the OTel token histograms (telemetry/metrics.py:355-394).

Every auxiliary LLM call the SDK makes on the user's behalf sidesteps this, each in a different way:

1. Conversation summarization (default path) — generate_summary calls model.stream() directly and explicitly bypasses "lock, metrics, traces, tool loop" (agent/conversation_manager/compression/context_compression.py:156), then discards the usage from the stop event (_, result_message, _, _, line 186). No metrics, no span — the tokens vanish. With a user-supplied summarization_agent, usage lands on that agent's own metrics and is never merged into the parent's.
2. HITL LLM risk classifier — builds a throwaway inner Agent per classified tool call (vended_interventions/hitl/classifier.py:117-124). Tokens reach the global OTel histograms/traces (unattributed), but the inner agent's EventLoopMetrics are discarded, so the parent agent's accumulated_usage undercounts.
3. Model routing InputComplexityStrategy (PR feat(routing): add configurable classifier strategy #3846, in flight) — _classify calls classifier_model.structured_output() raw on the Model: no EventLoopMetrics, no OTel histograms, no span. Its default classifier lazily creates a Bedrock Haiku 4.5 model, so this is recurring per-routing-decision spend that is completely invisible.
4. Memory ModelExtractor — the closest to correct: wraps the call in a model-invoke span and ends it with usage (memory/extraction/model_extractor.py:87-111), so usage is visible in traces — but still absent from EventLoopMetrics and the OTel token histograms.

Consequences:

• result.metrics.accumulated_usage does not mean "what this agent invocation spent" — it silently excludes summarization and HITL classification, and would exclude routing classification once feat(routing): add configurable classifier strategy #3846 lands.
• Any cost feature built on accumulated usage ([FEATURE] Add LiteLLM Cost Tracking to Agent Results #1216, [FEATURE] Expose total request cost (LLM + tool/KB/DB calls) in AgentResult.metadata — avoid manual per-token pricing #1428, PR feat(litellm): add cost tracking to AgentResult #3212) inherits the same blind spots.
• There is no shared "instrumented aux model call" helper, so each new feature re-decides its telemetry story and they have already diverged four ways.

Proposed Solution

Introduce one shared, instrumented entry point for SDK-internal model calls (roughly the stream_messages + start_model_invoke_span/end_model_invoke_span pattern ModelExtractor already uses), and route all four call sites through it. Open design questions:

• Should aux usage roll up into the owning agent's EventLoopMetrics.accumulated_usage (tagged as auxiliary, e.g. per-source breakdown) or be recorded separately? Rolling it up changes observed numbers for existing users; keeping it separate preserves today's semantics but needs a new surface.
• At minimum, always emit the OTel model-invoke span with usage attributes and record the token histograms with a source/component attribute (summarization, hitl_classifier, routing_classifier, memory_extraction) so spend is attributable.

Use Case

Anyone doing cost accounting or capacity planning per agent/invocation: today a summarizing conversation manager, HITL LLM classifier, or (soon) complexity-routing classifier makes their observed token counts diverge from their provider bill, with no way to see where the difference comes from short of provider-side logging.

Alternatives Solutions

• Document that aux calls are out-of-band spend and leave them uninstrumented — cheap, but makes accurate SDK-side cost reporting permanently impossible.
• Instrument only via OTel spans (extend the ModelExtractor pattern to the other three sites) without touching EventLoopMetrics — non-breaking, attributable in tracing backends, but invisible to users who read result.metrics.

Additional Context

• Related but distinct: [FEATURE] Add LiteLLM Cost Tracking to Agent Results #1216 / [FEATURE] Expose total request cost (LLM + tool/KB/DB calls) in AgentResult.metadata — avoid manual per-token pricing #1428 (cost on AgentResult — main-loop only), PR feat(litellm): add cost tracking to AgentResult #3212 (LiteLLM cost tracking, open), [FEATURE] Track metadata and cost-aware model routing #3860 (cost-aware routing strategy metadata), [FEATURE] Per-tool token counts #1503 (per-tool token counts), [BUG] OTLP Telemetry attaches token usage metrics (gen_ai.usage.*) to container spans, causing duplicate token counts in OTEL backends (Opik, Langfuse) #3602 (where gen_ai.usage.* attaches on spans).
• File/line references are at current main (2026-08-17); PR feat(routing): add configurable classifier strategy #3846 references are at its head dcd0d742.