Skip to content

Move eval module out of src/sre_agent/ into top-level evals/ #177

@osw282

Description

@osw282

Summary

The eval module currently lives at src/sre_agent/eval/, which means it gets bundled into the distributed wheel alongside production code (cli, config, core). Evaluation/benchmark code is a dev-time concern and should not ship with the package.

Proposed Changes

  1. Move src/sre_agent/eval/evals/ at the project root (alongside tests/, docs/, etc.)
  2. Remove the eval entry points from [project.scripts] in pyproject.toml:
    • sre-agent-run-tool-call-eval
    • sre-agent-run-diagnosis-quality-eval
  3. Update invocation to run evals as standalone scripts (e.g. python -m evals.tool_call.run) or via a Makefile/task runner target.
  4. Update any CI/docs that reference the old paths or entry points.

Motivation

  • Smaller production package — eval code, fixtures, and eval-only dependencies don't belong in the shipped wheel.
  • Separation of concerns — keeps src/sre_agent/ focused on the agent itself.
  • Convention — matches the common pattern of top-level tests/, evals/, benchmarks/ directories.

Metadata

Metadata

Assignees

Type

No type

Projects

Status

In Review

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions