UT Agent Evaluation Harness

Executive Summary

This repository provides a practical evaluation harness for a black-box UT generation agent built on GitHub Copilot, VS Code, .github prompts, custom agents, and skills.

The goal is not to replace Copilot or introduce a third-party generation pipeline. Instead, this repository evaluates the patch produced by the existing agent. The agent is treated as a black box: it receives a repository snapshot and a task prompt, modifies test files inside the workspace, and the harness captures the resulting Git diff. The evaluator then checks whether the generated tests are valid, executable, non-flaky, and effective.

The system is designed to answer five engineering questions:

1. Can the current UT agent produce tests that can be committed without manual repair?
2. When it fails, is the failure caused by imports, fixtures, mocks, weak assertions, flaky behavior, or insufficient context?
3. Does a graph-enhanced prompt / skill outperform the current agent under the same benchmark cases?
4. Are improvements due to code graph structure, or simply because the prompt contains more context?
5. Is the accuracy gain worth the additional operational complexity and cost?

The recommended evaluation unit is a case manifest. Each case defines a repository snapshot, target files, visible context, allowed modification paths, task prompt, setup commands, test commands, coverage commands, mutation commands, and optional hidden oracle checks. The evaluator produces structured JSON and CSV reports for A/B comparison across agent versions.

Intended Workflow

case manifest
    ↓
prepare clean workspace
    ↓
copy selected .github agent / prompt / skill version
    ↓
run Copilot / VS Code agent manually or semi-automatically
    ↓
capture git diff
    ↓
run evaluation commands
    ↓
classify failures and generate reports

Repository Layout

.
├── README.md
├── pyproject.toml
├── cases/
│   ├── README.md
│   └── examples/
│       ├── python-pytest-case.yaml
│       └── java-junit-case.yaml
├── agent_versions/
│   ├── README.md
│   ├── current/.github/
│   └── graph/.github/
├── templates/
│   └── github-copilot/
├── src/ut_agent_eval/
│   ├── cli.py
│   ├── case.py
│   ├── workspace.py
│   ├── patch.py
│   ├── evaluator.py
│   ├── failure_classifier.py
│   └── report.py
├── docs/
│   ├── architecture.md
│   ├── evaluation-methodology.md
│   ├── metrics.md
│   └── dataset-strategy.md
└── runs/

Quick Start

Install locally:

python -m pip install -e .

Prepare a clean workspace from a case manifest:

ut-agent-eval prepare \
  --case cases/examples/python-pytest-case.yaml \
  --agent-version agent_versions/current \
  --run-id demo-current-001

Open the generated workspace in VS Code, run your Copilot UT agent using the task file generated by the harness, then capture the patch:

ut-agent-eval capture --run-id demo-current-001

Evaluate the generated tests:

ut-agent-eval evaluate --run-id demo-current-001

Compare multiple runs:

ut-agent-eval compare --runs-dir runs --output reports/aggregate.csv

Core Metric

The primary metric is Valid Effective Test Rate:

Valid Effective Test =
  patch applies
  AND only modifies allowed paths
  AND generated tests pass
  AND full suite passes
  AND not flaky
  AND (
       improves coverage
       OR kills relevant mutants
       OR fails on buggy version and passes on fixed version
       OR passes hidden oracle checks
      )

Coverage alone is not enough. A test that executes code but has no meaningful assertions should not be considered successful.

A/B Experiment Design

Recommended agent groups:

Group	Description
A	Current Copilot UT agent
B	Current agent + vector context
C	Current agent + code graph context
D	Current agent + vector + code graph hybrid

Each benchmark case should be run against each agent version under the same repository snapshot, task prompt, allowed paths, and evaluation commands.

License / Data Note

This repository is intended as an internal evaluation harness. Public datasets can be adapted into the case manifest format, but commercial evaluation should prioritize internal PR-based, bug-based, and coverage-gap benchmark cases to reduce licensing and data-contamination risks.