Skip to content

Commit b8945a4

Browse files
authored
fix(obs-and-eval): added new lines to display lists correctly + reworded part of the evaluation.md (#299)
Signed-off-by: Jacques Samain <jsamain@cisco.com>
1 parent a018036 commit b8945a4

2 files changed

Lines changed: 6 additions & 6 deletions

File tree

docs/obs-and-eval/evaluation.md

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,14 @@ The Observe SDK emits raw telemetry (traces, metrics, and optional logs) to an O
77
The API Layer abstracts the underlying database so that we are not locked into a single storage technology. While ClickHouse is the initial implementation, the interface is intentionally narrow (session lookup, span search, metric fetch, write‑back of computed artifacts) to keep alternative backends (e.g., PostgreSQL, BigQuery, Elastic, Parquet lake) easily pluggable.
88

99
Capabilities:
10+
1011
- Retrieve raw session traces (for replay, graph reconstruction, diffing).
1112
- Fetch primary metrics emitted directly by the SDK (e.g., latency histograms, token usage counters).
1213
- Provide bounded, pagination‑friendly queries for the MCE (avoids heavy ad‑hoc joins inside the engine).
1314
- Enforce schema normalization/translation so upstream differences (SDK versions, framework variants) do not leak into evaluator logic.
1415

1516
Design goals:
17+
1618
- **Decoupling:** Evaluation logic never embeds vendor‑specific SQL.
1719
- **Portability:** Swap storage by implementing a small provider contract.
1820
- **Consistency:** Uniform response envelopes (metadata + data + pagination cursor).
@@ -21,18 +23,15 @@ Design goals:
2123
## Metrics Computation Engine (MCE)
2224

2325
The Metrics Computation Engine computes **derived** metrics at multiple aggregation levels:
26+
2427
- **Span level:** e.g., answer relevance for a single LLM call or tool invocation success rate.
2528
- **Session level:** overall MAS completion status, agent collaboration success rate, delegation accuracy.
2629
- **Population level:** metrics computed over a set of sessions, like graph determinism score.
2730

2831
The MCE uses a plugin architecture, making it straightforward to add new metrics.
29-
It can run either as:
30-
- A stand‑alone service exposed through a REST API, or
31-
- An embeddable SDK integrated into a third‑party data or analytics platform.
32+
It was designed to be running either as a stand-alone service, exposed through a REST API; or as an embeddable SDK, integrated into a third-party data or analytics platform.
3233

33-
Computed metrics can be:
34-
- Stored back into the database via the API Layer for longitudinal analysis, and/or
35-
- Returned directly to the caller for immediate consumption.
34+
The metrics computed by the MCE can either be stored back into the database via the API Layer for longitudinal analysis, and/or returned directly to the caller for immediate consumpltion.
3635

3736
## Detailed Usage
3837

docs/obs-and-eval/observe-sdk.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ For more information, see the [schema directory in the observe repository](https
1313
We provide a framework‑agnostic, OTel‑compliant observability SDK for multi‑agent systems. Each agent in the MAS can be instrumented by applying lightweight decorators to key functions or by using native OpenTelemetry primitives directly. The SDK exports metrics and traces using the [OpenTelemetry (OTel)](https://opentelemetry.io/) standard.
1414

1515
OTel is central to effective MAS observability as it provides:
16+
1617
- **Unified telemetry collection:** A single, vendor‑neutral API/SDK for metrics, traces, and (optionally) logs across heterogeneous agent components.
1718
- **Consistent instrumentation & standardization:** Semantic conventions and stable APIs ensure uniform telemetry shape, enabling backend flexibility without re‑instrumentation.
1819
- **Distributed tracing for complex workflows:** OTel’s tracing links agent spans, LLM calls, and tool interactions into an end‑to‑end execution graph—critical for understanding non‑deterministic decision paths and debugging across agent boundaries.

0 commit comments

Comments
 (0)