[Evaluation] Use openai SDK types for BYO judge chat-completion output#48172
Open
shrutiyer wants to merge 3 commits into
Open
[Evaluation] Use openai SDK types for BYO judge chat-completion output#48172shrutiyer wants to merge 3 commits into
shrutiyer wants to merge 3 commits into
Conversation
Address PR review (posaninagendra): re-shape the Responses API result into the real openai chat-completion types instead of hand-written stand-ins. Replace the _Usage/_ChatMessage/_Choice/_ChatCompletion classes with _to_usage (-> openai CompletionUsage) and _to_chat_completion (-> openai ChatCompletion with Choice / ChatCompletionMessage). This keeps the shim in sync with the SDK schema (same rationale as the EasyInputMessageParam input change) and, as a bonus, fixes a latent gap: the prompty formatter calls response.model_dump() on the tool-call path, which the hand-written _ChatCompletion did not provide. _finish_reason now returns the chat-completions finish_reason Literal (clamping any unexpected Responses incomplete reason to 'stop') so Choice validation cannot fail. Runtime behavior is unchanged for the judge path; 29 byo unit tests pass and the 18x4 admin-connected matrix is 72/72 OK. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b11202e6-6a52-4777-bed1-9c48730a63c3
github-actions
Bot
added
the
Evaluation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Replaces handwritten BYO judge response adapters with OpenAI SDK response types.
Changes:
- Adds typed conversion helpers for usage and chat completions.
- Normalizes Responses API finish reasons.
- Updates BYO judge unit tests for the new helpers.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
_byo_judge.py |
Constructs OpenAI SDK completion objects. |
test_byo_judge.py |
Updates adapter and finish-reason tests. |
posaninagendra
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #48002 (merged). Addresses a review comment (@posaninagendra): "Instead of handwriting the objects, could we use OpenAI SDK entities?"
The BYO admin-connected judge shim re-shapes each Foundry Responses API result into the
chat.completionscontract the prompty judge path consumes. This PR replaces the hand-written stand-in classes with the realopenaiSDK types so the shim stays in sync with the SDK schema (same rationale as theEasyInputMessageParaminput-side change already merged in #48002).Changes
_Usage/_ChatMessage/_Choice/_ChatCompletionwith:_to_usage(...)->openai.types.CompletionUsage_to_chat_completion(...)->openai.types.chat.ChatCompletion(built fromChoice+ChatCompletionMessage)_finish_reasonnow returns the chat-completionsfinish_reasonLiteraland clamps any unexpected Responsesincomplete_details.reasontostopsoChoicevalidation cannot fail.Why
openairestructures the chat-completion shape, this is caught at type-check time instead of silently breaking (the reviewer's point).response.model_dump()on the tool-call path; the hand-written_ChatCompletiondid not provide it, so a realChatCompletioncloses that gap.openaiis already a hard dependency (openai>=1.108.0), so importing these types is safe.Testing
test_byo_judge.pyunit tests pass;blackclean.