Agent observability object model

[anirudha]

agent observability object model

This document defines the requirements for the OpenSearch Agent Observability Dashboard built on top of the existing tri-mode plugin scaffold. The dashboard provides LLM observability capabilities backed by OpenSearch indices, covering tracing, evaluation, monitoring, dataset management, and experiment tracking for LLM and multi-agent workflows.

All naming follows OpenTelemetry semantic conventions and the emerging GenAI semantic conventions (PR #2563, Issue #3398). Traces contain spans. Spans carry gen_ai.* attributes. Evaluations are modeled as evaluation spans with gen_ai.evaluation.* attributes linked to the spans they evaluate. Experiments map to test.suite.* and test cases map to test.case.*, reusing the existing OTel test namespace rather than inventing GenAI-specific experiment attributes.

Requirements are organized into foundation (Phase 1 MVP) and feature layers, with priority annotations (P0/P1).

Glossary

• Trace: An OpenTelemetry trace — the top-level entity representing a complete LLM request or agent workflow execution. Contains traceId, name, userId, sessionId, input, output, metadata, tags, release, version, environment, startTime, endTime, latency, totalCost, totalTokens, and public flag
• Span: An OpenTelemetry span — a single unit of work within a Trace. Spans have a spanKind that classifies them: llm (LLM generation call), tool (tool/function invocation), agent (agent orchestration step), chain (sequential pipeline step), retrieval (RAG retrieval), embedding (embedding generation), or generic (default). Spans carry GenAI semantic convention attributes including gen_ai.system, gen_ai.request.model, gen_ai.response.model, gen_ai.operation.name, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.request.temperature, gen_ai.request.max_tokens, gen_ai.response.finish_reasons, plus input, output, cost, level, and metadata. Spans form parent-child trees via parentSpanId
• Evaluation Span: An OpenTelemetry span with gen_ai.operation.name: "evaluation" that represents a quality assessment of another span. Carries gen_ai.evaluation.name, gen_ai.evaluation.score, gen_ai.evaluation.label, and gen_ai.evaluation.reasoning attributes. Uses OTel span links to reference the evaluated span
• Evaluation Result Event: A gen_ai.evaluation.result span event attached to a span being evaluated, carrying gen_ai.evaluation.name, gen_ai.evaluation.score.value, and gen_ai.evaluation.explanation. An alternative to evaluation spans for inline evaluation recording
• Test Suite (Experiment): Maps to test.suite.* OTel attributes. A batched evaluation run that groups multiple test case evaluations. Identified by test.suite.name and test.suite.run.id (resource-level). test.suite.run.status represents overall experiment outcome
• Test Case (Dataset Row): Maps to test.case.* OTel attributes. An individual evaluation input within a test suite. Identified by test.case.id and test.case.name (span-level). test.case.result.status represents pass/fail of individual evaluations
• Dataset: Referenced via gen_ai.dataset.id, gen_ai.dataset.name, gen_ai.dataset.version attributes on test case spans. A named collection of test cases used for benchmarking. This is an attribute-level reference, not a separate entity type in OTel
• Session: A grouping entity that aggregates multiple Traces sharing the same sessionId, representing multi-turn conversations or workflows. Maps to gen_ai.conversation.id in OTel conventions
• TracedUser: An aggregated user entity derived from userId fields on Traces, with computed metrics such as total token usage, trace count, and cost
• OpenSearch_Client_Service: (might not be needed) A shared service in the core layer that manages connections to OpenSearch and executes search and aggregation queries
• OpenSearch_Config: (might not be needed) A configuration object specifying OpenSearch connection parameters including host, port, authentication credentials, SSL settings, and index name prefix
• Index_Mapping: An OpenSearch index mapping definition that specifies field types, analyzers, and structure for a given index
• Dashboard_Page: The root React page component rendered by the plugin, displaying the LLM Observability header, a left navigation sidebar, and a content area for the active view
• Route_Handler: A framework-agnostic function in server/routes/handlers.ts that processes API requests without depending on Express or Hapi directly
• Mock_Mode: A configuration flag enabling seeded stub data responses without a live OpenSearch backend
• Core_Layer: The shared business logic module (core/) with no platform dependencies, consumed by all three distribution modes
• Span_Tree: An interactive hierarchical visualization of spans within a trace, showing parent-child relationships and execution order

OTel Attribute Reference

OTel Convention	Attribute Namespace	Status
Trace	Standard OTel	Stable
Span with gen_ai.* attributes	gen_ai.operation.name, gen_ai.system, gen_ai.request.model, etc.	Merged (GenAI semconv)
Evaluation Span / gen_ai.evaluation.result event	gen_ai.evaluation.name, gen_ai.evaluation.score, gen_ai.evaluation.label, gen_ai.evaluation.reasoning	Merged (PR #2563)
Test Suite	test.suite.name, test.suite.run.id, test.suite.run.status	Proposed extension (Issue #3398)
Dataset reference	gen_ai.dataset.id, gen_ai.dataset.name, gen_ai.dataset.version	Proposed (Issue #3398)
Test Case	test.case.id, test.case.name, test.case.result.status	Proposed extension (Issue #3398)
Session / Conversation	gen_ai.conversation.id	Merged (GenAI semconv)

Note: test.suite.run.id and test.case.id are proposed extensions to the existing OTel test.* namespace. They are generic and useful beyond GenAI (CI/CD test reporting, etc.). Our implementation adopts these ahead of formal standardization.