Route BYO admin-connected judge models through the project Responses API (prompty judge path)

[shrutiyer]

Adds support for admin-connected (BYO) judge models — referenced as "connection/deployment" and served only through the Foundry project Responses API — in the prompty-based LLM-as-a-judge evaluators (coherence, relevance, fluency, groundedness, retrieval, similarity, tool/task judges, ...). Chat Completions cannot route these models (404 DeploymentNotFound); the platform resolves the connection + auth (API key / managed identity / OAuth2) when the call goes through the project Responses API.

Changes

• azure/ai/evaluation/_byo_judge.py (new): AsyncByoProjectResponsesClient, an async chat.completions-compatible shim that routes judge calls to the project Responses API (AIProjectClient(...).get_openai_client().responses.create). Maps chat messages to Responses input items and curated sampling params, and translates response_format: {type: json_object|json_schema} to the Responses text: {format: {...}} param (restores enforced JSON mode). Adapts the Responses result (input/output_tokens) back to a chat-completions shape (prompt/completion_tokens, choices[0].message, ...). is_byo_model_config detects a BYO config (byo_model + project_endpoint).
• azure/ai/evaluation/_legacy/prompty/_prompty.py: AsyncPrompty.__call__ branches on is_byo_model_config(...) and uses the shim instead of AsyncAzureOpenAI, so every prompty judge works with an admin-connected model with no evaluator-code change.
• azure/ai/evaluation/_common/utils.py: validate_model_config passes BYO configs through (they intentionally omit azure_endpoint/azure_deployment).

Caller contract: model_config = {"byo_model": "<connection>/<deployment>", "project_endpoint": ".../api/projects/<project>"} plus a credential; the platform applies the connection auth.

Tests

• tests/unittests/test_byo_judge.py: 15 unit tests (helpers, _Usage token adapter, async shim routing, validate_model_config passthrough, and response_format -> text.format mapping). All pass.

Validation

Ran the real prompty judges against live admin connections using this branch's source: Coherence/Relevance plus an 18-evaluator x 4-connection matrix (API key, managed identity, ModelGateway API key, OAuth2) -> all returned real scores; an instrumented trace confirmed text.format is sent on the wire and enforced-JSON parsing succeeds.

All SDK Contribution checklist

• Does not introduce breaking changes (additive; non-BYO AzureOpenAI/OpenAI configs are unaffected).
• CHANGELOG update pending.
• Pull request includes test coverage.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Copilot-Session: b11202e6-6a52-4777-bed1-9c48730a63c3

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds BYO judge-model routing through the Foundry project Responses API.

Changes:

• Adds an async chat-completions compatibility shim.
• Routes prompty evaluators through the shim for BYO configurations.
• Adds validation passthrough and unit coverage.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
_byo_judge.py	Implements Responses API routing and response adaptation.
_prompty.py	Selects BYO routing for applicable configurations.
_common/utils.py	Allows BYO model configurations through validation.
test_byo_judge.py	Tests routing, mapping, usage, and validation behavior.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_byo_judge.py:176

• This adds a new evaluator capability, but the package CHANGELOG has no corresponding entry, and the PR checklist still marks that update as pending. Add this BYO judge-model support under the next release's Features Added section so the shipped behavior is discoverable.

class AsyncByoProjectResponsesClient:
    """Async OpenAI-compatible shim that routes ``chat.completions.create()`` to the Foundry project
    Responses API for an admin-connected (BYO) model.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/utils.py:226

• model_config now includes _BYOModelConfiguration, but the immediately called parse_model_config_type still accepts only the two existing configuration types. This leaves the internal type contract inconsistent and causes an incompatible-argument diagnostic when this package’s type checking is enabled. Widen the helper’s annotation as well; its existing guards already make BYO a no-op.

    model_config: Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration, _BYOModelConfiguration],

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_byo_judge.py:175

• This adds user-visible evaluator support, but sdk/evaluation/azure-ai-evaluation/CHANGELOG.md has no BYO/admin-connected model entry, and the PR checklist still marks the CHANGELOG update as pending. Add the feature under the next release before merging so the package release notes describe the new supported model path.

class AsyncByoProjectResponsesClient:
    """Async OpenAI-compatible shim that routes ``chat.completions.create()`` to the Foundry project
    Responses API for an admin-connected (BYO) model.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/utils.py:226

• The newly widened union is passed directly to parse_model_config_type, but that helper still accepts only the Azure OpenAI/OpenAI union (utils.py:216-218). This introduces an incompatible-argument error in mypy whenever construct_prompty_model_config is checked. Widen the helper's annotation as well.

    model_config: Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration, _BYOModelConfiguration],

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_byo_judge.py:215

• This import succeeds with the package's current development dependency azure-ai-projects<=1.0.0b10, so the missing-package error does not help, but that version cannot support this call: the projects changelog says project-endpoint construction starts in 1.0.0b11 and get_openai_client() is configured for project Responses operations only from 2.0.0b1 (sdk/ai/azure-ai-projects/CHANGELOG.md:485-500,381-395). The BYO path will therefore fail in the repository-supported environment. Require/validate a Responses-capable azure-ai-projects version and update the development dependency and installation guidance accordingly.

                from azure.ai.projects.aio import AIProjectClient
            except ImportError as ex:
                from azure.ai.evaluation._legacy._adapters._errors import MissingRequiredPackage

                raise MissingRequiredPackage(
                    message="Please install the 'azure-ai-projects' package to use admin-connected "
                    "(BYO) judge models."

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_byo_judge.py:4

• This adds a user-visible evaluator capability, but the package CHANGELOG still has no entry for BYO judge routing, and the PR checklist explicitly leaves the CHANGELOG update pending. Add the release note before merging so the new model-config behavior and optional projects-package requirement are discoverable.

"""Use admin-connected (BYO) models as LLM-as-a-judge evaluator models.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_byo_judge.py:4

• This adds a new evaluator capability, but CHANGELOG.md still has no corresponding entry (its latest section is 1.18.1), and the PR checklist also marks the changelog as pending. Add this BYO judge-model support under the next release’s Features Added section before merging.

"""Use admin-connected (BYO) models as LLM-as-a-judge evaluator models.