Skip to content

Surface provider error bodies and stop one dead sampler losing all metrics - #16

Open
dan-s-w wants to merge 1 commit into
mainfrom
feat/surface-api-errors
Open

Surface provider error bodies and stop one dead sampler losing all metrics#16
dan-s-w wants to merge 1 commit into
mainfrom
feat/surface-api-errors

Conversation

@dan-s-w

@dan-s-w dan-s-w commented Aug 14, 2026

Copy link
Copy Markdown

Two failures that were harder to diagnose than they needed to be. Both surfaced while debugging AIEV-275.

1. API error bodies were discarded

raise_for_status() reports only the status line, so a rejected request surfaced as:

perplexity_finance_historical_lookup failed with error 400, message='Bad Request'

That's everything the harness told us. The actual reason was sitting in the response body the whole time — diagnosing it meant re-issuing the request by hand outside the harness.

Non-2xx responses now include the body, capped at 2000 chars. Same request, after this change:

400, message='Bad Request: {"error":{"message":"invalid request body: json:
unknown field \"max_tokens\"","type":"invalid_request","code":400,
"param":"max_tokens"}}', url='https://api.perplexity.ai/v1/agent'

An hour of probing becomes one line in the log. The GET/POST branches were also collapsed, since they only differed in json= vs params=.

2. One fully failed sampler destroyed the whole summary

write_metrics raised ValueError as soon as any sampler had zero successful rows. Six healthy samplers plus one dead one produced no scorecard at all — the healthy results were computed and then thrown away.

It now logs the dead sampler and skips it. If nothing succeeded anywhere, it warns and writes nothing rather than raising on an empty frame.

3. failed_count column

Failures are excluded from the accuracy denominator, so a sampler that errored on half the dataset currently looks identical to one that answered all of it. failed_count sits next to problem_count so that's visible at a glance.

The accuracy calculation itself is unchanged. Whether failures should count against a provider is a methodology question, and this PR deliberately doesn't answer it — it just stops the situation being invisible.

Tests

tests/test_eval_results_analyzer.py — no network. Covers the reported counts, the one-dead-sampler regression, and the everything-failed case.

🤖 Generated with Claude Code

…trics

Two failures that were harder to diagnose than they needed to be.

1. API error bodies were discarded

   raise_for_status() reports only the status line, so a rejected request
   surfaced as "400, message='Bad Request'" and nothing else. Providers
   put the actual reason in the response body. Diagnosing the recent
   Perplexity payload bug meant re-issuing the request by hand outside
   the harness to read it.

   Non-2xx responses now include the body (capped at 2000 chars):

     400, message='Bad Request: {"error":{"message":"invalid request
     body: json: unknown field \"max_tokens\"", ...

2. One fully failed sampler destroyed the whole summary

   write_metrics raised as soon as any sampler had zero successful rows,
   so six healthy samplers plus one dead one produced no scorecard at
   all. It now logs and skips the dead sampler.

Also adds a failed_count column. Failures are excluded from the accuracy
denominator, so without it a sampler that errored on half the dataset is
indistinguishable from one that answered all of it. The accuracy
calculation itself is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants