From a37adbc5d6603d91812e18f25cba89b865f1c8ca Mon Sep 17 00:00:00 2001 From: Neco Date: Sat, 20 Jun 2026 11:16:22 +0800 Subject: [PATCH] Add ax to evals and observability Signed-off-by: Neco --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 96562b7..bc448f1 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,7 @@ Generic agent tooling is out of scope unless the page directly covers harness de - [OpenTelemetry Semantic Conventions for Generative AI Systems](https://opentelemetry.io/docs/specs/semconv/gen-ai/) - Standard span, metric, event, and attribute conventions for instrumenting LLM and agent workflows so harness traces stay portable across observability backends. - [AgentOps](https://github.com/AgentOps-AI/agentops) - Open-source Python SDK for agent monitoring, session replay, cost tracking, benchmarking, and tracing across common LLM and agent frameworks. - [agenttrace](https://github.com/luoyuctl/agenttrace) - Local-first TUI/CLI for auditing AI coding-agent session traces, health gates, cost spikes, tool failures, latency gaps, and attempt-to-attempt diffs. +- [ax](https://github.com/Necmttn/ax) - Local-first telemetry and memory graph for auditing AI coding-agent sessions, cost, skill and tool usage, and OTLP events across Claude Code, Codex, OpenCode, Cursor, and Pi. - [Learning to Verify AI-Generated Code](https://openhands.dev/blog/20260305-learning-to-verify-ai-generated-code) - OpenHands' overview of a layered verification stack using trajectory critics trained on production traces for reranking, early stopping, and review-time quality control. - [Demystifying Evals for AI Agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) - Anthropic's guidance on what to measure when agents have many possible trajectories to success or failure. - [Quantifying infrastructure noise in agentic coding evals](https://www.anthropic.com/engineering/infrastructure-noise) - Anthropic on how runtime configuration can move coding benchmark scores by more than many leaderboard gaps.