-
Notifications
You must be signed in to change notification settings - Fork 34
Move eval module out of src/sre_agent/ into top-level evals/ #177
Copy link
Copy link
Open
Labels
Description
Summary
The eval module currently lives at src/sre_agent/eval/, which means it gets bundled into the distributed wheel alongside production code (cli, config, core). Evaluation/benchmark code is a dev-time concern and should not ship with the package.
Proposed Changes
- Move
src/sre_agent/eval/→evals/at the project root (alongsidetests/,docs/, etc.) - Remove the eval entry points from
[project.scripts]inpyproject.toml:sre-agent-run-tool-call-evalsre-agent-run-diagnosis-quality-eval
- Update invocation to run evals as standalone scripts (e.g.
python -m evals.tool_call.run) or via a Makefile/task runner target. - Update any CI/docs that reference the old paths or entry points.
Motivation
- Smaller production package — eval code, fixtures, and eval-only dependencies don't belong in the shipped wheel.
- Separation of concerns — keeps
src/sre_agent/focused on the agent itself. - Convention — matches the common pattern of top-level
tests/,evals/,benchmarks/directories.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
In Review