feat(accuracy): HellaSwag DeepEval-backed benchmark + ExactMatch grader (AIP-877)#923

Merged

debermudez merged 10 commits into

mainfrom

dbermudez/aip-877-implement-hellaswag-benchmark-loader

May 22, 2026

Commits on May 22, 2026

feat(accuracy): HellaSwag benchmark + ExactMatch grader, DeepEval-backed (AIP-877)
debermudez
committed
fix(accuracy): validate HellaSwag tasks before downloading dataset (AIP-877)
debermudez
committed
fix(accuracy): append DeepEval confinement instruction to HellaSwag prompt (AIP-877)
debermudez
committed
fix(accuracy): reject 'all' mixed with other HellaSwag task selectors (AIP-877)
debermudez
committed
fix(accuracy): cap deepeval at <5.0.0 (AIP-877)
debermudez
committed
fix(accuracy): log when --accuracy-enable-cot is ignored for HellaSwag (AIP-877)
debermudez
committed
docs(accuracy): document HellaSwag dataset revision policy (AIP-877)
debermudez
committed
refactor(accuracy): hoist HellaSwag task lookup to module scope (AIP-877)
debermudez
committed
perf(accuracy): bucket HellaSwag validation rows by activity_label (AIP-877)
debermudez
committed
Merge branch 'main' into dbermudez/aip-877-implement-hellaswag-benchmark-loader
debermudez
authored