Surface provider error bodies and stop one dead sampler losing all metrics - #16
Open
dan-s-w wants to merge 1 commit into
Open
Surface provider error bodies and stop one dead sampler losing all metrics#16dan-s-w wants to merge 1 commit into
dan-s-w wants to merge 1 commit into
Conversation
…trics
Two failures that were harder to diagnose than they needed to be.
1. API error bodies were discarded
raise_for_status() reports only the status line, so a rejected request
surfaced as "400, message='Bad Request'" and nothing else. Providers
put the actual reason in the response body. Diagnosing the recent
Perplexity payload bug meant re-issuing the request by hand outside
the harness to read it.
Non-2xx responses now include the body (capped at 2000 chars):
400, message='Bad Request: {"error":{"message":"invalid request
body: json: unknown field \"max_tokens\"", ...
2. One fully failed sampler destroyed the whole summary
write_metrics raised as soon as any sampler had zero successful rows,
so six healthy samplers plus one dead one produced no scorecard at
all. It now logs and skips the dead sampler.
Also adds a failed_count column. Failures are excluded from the accuracy
denominator, so without it a sampler that errored on half the dataset is
indistinguishable from one that answered all of it. The accuracy
calculation itself is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eddy-nassif
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two failures that were harder to diagnose than they needed to be. Both surfaced while debugging AIEV-275.
1. API error bodies were discarded
raise_for_status()reports only the status line, so a rejected request surfaced as:That's everything the harness told us. The actual reason was sitting in the response body the whole time — diagnosing it meant re-issuing the request by hand outside the harness.
Non-2xx responses now include the body, capped at 2000 chars. Same request, after this change:
An hour of probing becomes one line in the log. The GET/POST branches were also collapsed, since they only differed in
json=vsparams=.2. One fully failed sampler destroyed the whole summary
write_metricsraisedValueErroras soon as any sampler had zero successful rows. Six healthy samplers plus one dead one produced no scorecard at all — the healthy results were computed and then thrown away.It now logs the dead sampler and skips it. If nothing succeeded anywhere, it warns and writes nothing rather than raising on an empty frame.
3.
failed_countcolumnFailures are excluded from the accuracy denominator, so a sampler that errored on half the dataset currently looks identical to one that answered all of it.
failed_countsits next toproblem_countso that's visible at a glance.The accuracy calculation itself is unchanged. Whether failures should count against a provider is a methodology question, and this PR deliberately doesn't answer it — it just stops the situation being invisible.
Tests
tests/test_eval_results_analyzer.py— no network. Covers the reported counts, the one-dead-sampler regression, and the everything-failed case.🤖 Generated with Claude Code