Skip to content

fix(eval): drop temperature for claude-opus-4-7 - #1

Closed
ghost wants to merge 1 commit into
mainfrom
unknown repository
Closed

fix(eval): drop temperature for claude-opus-4-7#1
ghost wants to merge 1 commit into
mainfrom
unknown repository

Conversation

@ghost

@ghost ghost commented Jul 26, 2026

Copy link
Copy Markdown

What this PR adds

Bug fix to the prompt-caching eval harness: removes a parameter that the Anthropic API now rejects, which currently prevents the script from running.

Skill checklist (for new skills)

N/A — this is a bug fix to eval/, not a skill change.

Lint output

N/A — no files under skills/ are touched by this PR.

Index regeneration

N/A — no skill files changed, so README.md and the index are unaffected.

Notes for review

run_eval.py passes temperature=0 to claude-opus-4-7, which the Anthropic API rejects:

{"type":"error","error":{"type":"invalid_request_error","message":"`temperature` is deprecated for this model."}}

The script fails before completing a single call. Removing the parameter fixes it.

Worth flagging: temperature=0 was presumably there for reproducible eval runs. Opus 4.7 doesn't accept any value other than 1, so determinism via temperature isn't available on this model. I left a comment in the code explaining why the param is gone rather than removing it silently.

For reference, I tested the sampling params across the current Anthropic lineup. Rejected on opus-5, sonnet-5, fable-5, opus-4-8 and opus-4-7. Still accepted on sonnet-4-6, opus-4-6, opus-4-5, haiku-4-5 and sonnet-4-5. So this only affects the 4-7-and-newer calls.

@ghost
ghost requested a review from cobusgreyling as a code owner July 26, 2026 17:11
Repository owner closed this by deleting the head repository Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants