oxagen-evals

A public benchmark harness that measures whether the Oxagen ontology layer — typed code graph, agent memory, and MCP server — changes a coding agent's performance on real software tasks. The harness runs the same coding agent twice per task: once with vanilla local tools (Read, Edit, Bash, Grep) and once with the same tools plus the Oxagen MCP server connected to a workspace where the task's repo has been pre-ingested. Then it diffs the runs.

We publish this so anyone can fork it, point it at their own codebase, and answer the same question for themselves. Numerical honesty is the brand: this repo does not bake in claims about how much faster Oxagen makes an agent. It runs the experiment and reports the data.

What gets measured

For every (task, config) pair the harness records:

Field	What it captures
success	Did the agent's final answer pass the task's deterministic check?
tool_calls	Total tool invocations across the run
unique_tools	Set of distinct tool names used
tokens_in / tokens_out	Reported by the Anthropic API
cache_creation_tokens	Tokens written to the prompt cache
cache_read_tokens	Tokens served from the cache (the cost-savings story)
wall_seconds	Start to finish
turns	Number of assistant turns
first_correct_file_turn	Turn where the agent first opened the answer file (navigation)

Aggregates land in a markdown report under reports/<run_id>.md with a delta column comparing oxagen and baseline. Raw per-(task, config) JSON metrics land in results/<run_id>/.

Quickstart

You need uv and Python 3.11+.

# Install
uv sync

# Inspect what the lite suite would run, no API calls made
uv run oxagen-evals run --suite=lite --config=both --dry-run

# Run the lite suite on baseline only (no Oxagen account needed)
ANTHROPIC_API_KEY=sk-ant-... \
  uv run oxagen-evals run --suite=lite --config=baseline

# Run both configs (requires an Oxagen workspace + API key)
ANTHROPIC_API_KEY=sk-ant-... \
OXAGEN_API_KEY=oxa_live_... \
OXAGEN_WORKSPACE_ID=00000000-0000-0000-0000-000000000000 \
  uv run oxagen-evals run --suite=lite --config=both

The oxagen config is skipped automatically (with a recorded skipped: env_missing row in the metrics) if OXAGEN_API_KEY and OXAGEN_WORKSPACE_ID are not set, so the suite never crashes from a missing account.

Environment variables

Variable	Required for	Purpose
ANTHROPIC_API_KEY	All non-dry runs	Anthropic Messages API access
OXAGEN_API_KEY	oxagen config	Bearer token sent to the production Oxagen MCP server
OXAGEN_WORKSPACE_ID	oxagen config	Workspace id (also encoded server-side in the api key)
OXAGEN_EVAL_MODEL	Optional	Override the default model (claude-haiku-4-5-20251001)
OXAGEN_MCP_URL	Optional	Override MCP endpoint (default https://mcp.oxagen.ai/sse)

Methodology

• Same model, same prompt, same temperature. Both configs call claude-haiku-4-5-20251001 with temperature=0 and an identical system prompt. The only thing that changes between runs is the set of tools the model can call.
• Baseline is local-tools-only. The agent has Read, Edit, Bash, Grep — implemented in this repo and sandboxed to the task's fixture directory. No MCP servers attached. This mirrors what a stock coding agent gets out of the box.
• oxagen is local tools + the Oxagen MCP server. Same four local tools, plus every tool the MCP server exposes (entity search, graph traversal, semantic recall, etc.).
• Deterministic checking. Every task has a check(answer) method that returns pass/fail by exact comparison — file paths, function names, exact strings. No LLM-as-judge.
• Prompt caching is on by default. The system prompt and the task prompt are marked as cacheable so cache-read tokens dominate after the first run. The cache hit rate is reported in the metrics.
• One model in, one number out. Switching the model is a config flag, not a code change — useful for "does Opus close the gap that Haiku shows?" comparisons.

What this does not measure

• Agent quality across providers. This harness is Anthropic-only. The Oxagen MCP server speaks the Model Context Protocol so it works with any MCP-aware client; testing parity is out of scope here.
• Real-world repository scale. The shipped fixture is ~50 files. Use --fixture=<path-to-your-repo> to point at something larger.
• Cost. Tokens are reported but pricing is not — pricing changes faster than this README does. Multiply by your account's rate card.

Adding a new task

Create a Python file under oxagen_evals/tasks/<category>/ that subclasses Task. A minimal example:

# oxagen_evals/tasks/navigation/find_helper.py
from pathlib import Path

from oxagen_evals.tasks.base import CheckResult, Task


class FindHelperTask(Task):
    """Locate the helper module that owns `formatTimestamp`."""

    id = "navigation.find_helper"
    suite = "lite"
    fixture = "tiny_repo"
    answer_path = "src/utils/format.ts"

    def prompt(self, task_dir: Path) -> str:
        return (
            "In the repo rooted at this directory, find the file that "
            "defines the function `formatTimestamp`. Reply with the "
            "relative path only."
        )

    def check(self, answer: str) -> CheckResult:
        normalised = (answer or "").strip().strip("`'\"")
        ok = normalised == self.answer_path
        return CheckResult(ok=ok, detail=f"got={normalised!r}")

Register it in oxagen_evals/tasks/__init__.py (auto-discovery is based on subclass enumeration, so import the module there). Add the seeded answer to oxagen_evals/fixtures/<fixture>/ANSWERS.md (the agent never sees this file).

Reproducing the published results

git clone https://github.com/oxagenai/oxagen-evals
cd oxagen-evals
uv sync
ANTHROPIC_API_KEY=sk-ant-... \
OXAGEN_API_KEY=oxa_live_... \
OXAGEN_WORKSPACE_ID=$WS \
  uv run oxagen-evals run --suite=lite --config=both --out=results/published

Compare reports/published.md against the results page on oxagen.ai. If your numbers disagree with ours, file an issue with the diff — that's what the repo is for.

Layout

oxagen_evals/
  cli.py            CLI entry — `oxagen-evals run ...`
  runner.py         One (task, config) → MetricsRecord
  metrics.py        Dataclass + JSON serializer
  report.py         Metrics → markdown + CSV
  configs/          baseline.yaml, oxagen.yaml
  tasks/            Task definitions, organised by category
  fixtures/         Self-contained eval repos (with ANSWERS.md keys)
  scripts/          ingest_fixture.py and other operator helpers
results/            Raw per-run JSON output (gitignored)
reports/            Rendered markdown reports (gitignored)

License

MIT — see LICENSE.

Status

This is v0.1. The runner is functional, the lite task suite has five tasks, and the fixture repo is ~50 files. The Oxagen-side ingestion helper (scripts/ingest_fixture.py) is best-effort — the exact ingest-API contract may shift as the platform matures, so the script logs and degrades rather than crashing if the upload endpoint rejects it. File issues, PR new tasks, fork freely.