Summary
After the actix-web → axum migration (cb87687), OTel span parenting breaks in orchestration mode. Two distinct symptoms observed:
mcp.tool_call spans orphaned — created as root spans in separate traces instead of nesting under execute_tool
agent.turn spans intermittently missing from traces
Both symptoms are absent on a backup of the pre-axum branch (backup/artifact-infra-local, commit 65e14cc) with identical configs, confirmed via side-by-side testing.
Expected hierarchy
agent.stream → orchestration → orchestration.worker → agent.turn → execute_tool → mcp.tool_call
Actual behavior
mcp.tool_call appears with parentId: null and a different traceId from the orchestration trace
execute_tool spans exist in the orchestration trace but have no mcp.tool_call children
- Intermittent: some traces correctly nest
mcp.tool_call, most don't
agent.turn occasionally absent from traces
Likely area
Span propagation across tokio::spawn in WrappedTool::call() (crates/aura/src/tool_wrapper.rs:437-441). The Span::current() capture and instrument() propagation worked under actix-web's current_thread runtime but behaves differently with axum's multi-threaded runtime.
Related
Summary
After the actix-web → axum migration (
cb87687), OTel span parenting breaks in orchestration mode. Two distinct symptoms observed:mcp.tool_callspans orphaned — created as root spans in separate traces instead of nesting underexecute_toolagent.turnspans intermittently missing from tracesBoth symptoms are absent on a backup of the pre-axum branch (
backup/artifact-infra-local, commit65e14cc) with identical configs, confirmed via side-by-side testing.Expected hierarchy
Actual behavior
mcp.tool_callappears withparentId: nulland a differenttraceIdfrom the orchestration traceexecute_toolspans exist in the orchestration trace but have nomcp.tool_callchildrenmcp.tool_call, most don'tagent.turnoccasionally absent from tracesLikely area
Span propagation across
tokio::spawninWrappedTool::call()(crates/aura/src/tool_wrapper.rs:437-441). TheSpan::current()capture andinstrument()propagation worked under actix-web'scurrent_threadruntime but behaves differently with axum's multi-threaded runtime.Related
force_flushdeadlocking the tokio runtime is a separate symptom of the same migration. The proposed fix in fix(otel): wrap force_flush in spawn_blocking to avoid deadlock #102 (spawn_blocking) resolves the deadlock but the span orphaning persists independently.