[Do not merge][Debug] Preview for voiceagent

eng/tools/azure-sdk-tools/azpysdk/samples.py

@@ -84,6 +84,12 @@
         "hello_world_sample_entra_id_and_bleu.py",
     ],
     "azure-ai-ml": ["ml_samples_authentication_sovereign_cloud.py"],
+    "azure-ai-projects": [
+        # These interactively read from stdin via input(), which raises EOFError when this
+        # runner executes the file non-interactively.
+        "sample_voice_agent_live_text_conversation.py",
+        "sample_voice_agent_live_text_conversation_async.py",
+    ],
     "azure-eventgrid": [
         "__init__.py",
         "consume_cloud_events_from_eventhub.py",

scripts/devops_tasks/test_run_samples.py

@@ -88,6 +88,12 @@
     "azure-ai-ml": [
         "ml_samples_authentication_sovereign_cloud.py"
     ],
+    "azure-ai-projects": [
+        # These interactively read from stdin via input(), which raises EOFError when this
+        # runner executes the file non-interactively.
+        "sample_voice_agent_live_text_conversation.py",
+        "sample_voice_agent_live_text_conversation_async.py",
+    ],
     "azure-eventgrid": [
         "__init__.py",
         "consume_cloud_events_from_eventhub.py",

sdk/ai/azure-ai-projects/.env.template

@@ -23,6 +23,14 @@ AZURE_AI_PROJECTS_CONSOLE_LOGGING=
 FOUNDRY_PROJECT_ENDPOINT=
 FOUNDRY_PROJECT_API_KEY=
 FOUNDRY_MODEL_NAME=
+# Read by the recorded voice-agent CRUD tests only (tests/test_base.py), not by any sample.
+FOUNDRY_VOICE_MODEL_NAME=
+# Read by the samples under samples/agents/voice/ (model deployment name, agent name, model type,
+# and a conversation ID for the read-conversation samples). Distinct from FOUNDRY_VOICE_MODEL_NAME above.
+FOUNDRY_VOICE_MODEL=
+FOUNDRY_VOICE_MODEL_TYPE=
+FOUNDRY_VOICE_AGENT_NAME=
+FOUNDRY_VOICE_CONVERSATION_ID=
 FOUNDRY_AGENT_NAME=
 FOUNDRY_AGENT_CONTAINER_IMAGE=
 CONVERSATION_ID=

sdk/ai/azure-ai-projects/CHANGELOG.md

@@ -4,13 +4,36 @@
 
 ### Features Added
 
-* Added supporting AgentInsight* models and enums covering monitors, runs, generated insights, proposed fixes, highlighted and linked traces, costs, token usage, severity, status, and run triggers.
-* Added Microsoft 365 agent publishing.
+* Added voice agents, unified with the rest of the Agents API as a new `kind="voice"` on `AgentDefinition`:
+  * Define a voice agent with `VoiceAgentDefinition`, configuring its model (`VoiceModelType`), audio input/output (`VoiceAgentAudioConfig`, `VoiceAgentAudioInputConfig`, `VoiceAgentAudioOutputConfig`), turn detection (`VoiceAgentTurnDetectionConfig` and its `VoiceAgentServerVadTurnDetection` / `VoiceAgentAzureSemanticVadTurnDetection` / `VoiceAgentAzureSemanticVadEnTurnDetection` / `VoiceAgentAzureSemanticVadMultilingualTurnDetection` variants), greeting (`VoiceAgentGreetingConfig` and its `VoiceAgentTemplateGreetingConfig` / `VoiceAgentLlmGeneratedGreetingConfig` variants), tools (`VoiceAgentTool`, `VoiceAgentFunctionTool`, `VoiceAgentMcpTool`, `VoiceAgentSystemTool`, `VoiceAgentToolboxTool`), and avatar (`VoiceAgentAvatarConfig`). Manage it like any other agent through `project_client.agents` (`create_version`, `get`, `list`, `disable`/`enable`, `delete`).
+  * Added guided authoring via `project_client.agents.generate_agent(GenerateVoiceAgentRequest(kind=AgentKind.VOICE, ...))`, which returns a service-generated starter definition that can be edited afterward through the standard `create_version`/`update` flow.
+  * Added a new `client.realtime` / `async_client.realtime` entry point for realtime speech-to-speech streaming. Use `with client.realtime.connect(agent_name=...) as connection:` to open a WebSocket connection, `connection.send(...)` to send strongly-typed client events (or use the `connection.response`, `connection.conversation.item`, and `connection.session` helpers), and iterate over `connection` to receive strongly-typed server events (`RealtimeServerEvent*`). Conversation items exchanged with `connection.conversation.item.create(...)` are `RealtimeConversationItemMessageSystem`, `RealtimeConversationItemMessageUser`, `RealtimeConversationItemMessageAssistant`, `RealtimeConversationItemFunctionCall`, `RealtimeConversationItemFunctionCallOutput`, `RealtimeMCPApprovalResponse`, or a raw `Mapping[str, Any]`. The new types `Realtime`, `RealtimeConnection`, and `RealtimeConnectionManager` (and their async equivalents `AsyncRealtime`, `AsyncRealtimeConnection`, `AsyncRealtimeConnectionManager`) are exported from `azure.ai.projects` / `azure.ai.projects.aio`. Requires the optional `websockets` package for the sync client, or `aiohttp` for the async client.
+  * Added the `beta.agent_endpoint_conversations` operation group for reading back persisted voice-agent conversation transcripts and audio, for agents created with `store=True`.
+  * Added the underlying `RealtimeConversationItem*`, `RealtimeMCP*`, `RealtimeResponseUsage`, and related realtime event/session models used by the voice agent WebSocket protocol.
+* Added 11 new samples under `samples/agents/voice/`, covering basic agent lifecycle, guided generation, live audio and text conversations (sync and async), function tools, versioning, and reading back conversation transcripts and audio.
+* Extended voice agents with telephony, WebRTC, and sub-agent consultation:
+  * Added telephony bindings so a voice agent can receive calls through Teams Phone or Twilio. `project_client.agents.create_telephony_binding`/`get_telephony_binding`/`update_telephony_binding`/`delete_telephony_binding`/`list_telephony_bindings` manage the binding (`TelephonyBinding` and its `TeamsPhoneExtensionTelephonyBinding`/`TwilioTelephonyBinding` variants), and `list_telephony_calls`/`get_telephony_call`/`transfer_telephony_call`/`end_telephony_call`/`get_telephony_transfer_targets`/`replace_telephony_transfer_targets` manage in-progress and historical calls (`TelephonyCallRecord`, `TelephonyCallSummary`, `TelephonyCallTrace`, `TelephonyTransferTarget` and its `PSTNTelephonyTransferDestination`/`SipTelephonyTransferDestination`/`TeamsTelephonyTransferDestination` variants).
+  * Added an optional WebRTC transport for realtime voice sessions (`VoiceAgentTransport.WEBRTC`), where only SDP signaling travels over the WebSocket connection while media flows peer-to-peer. The new `VoiceAgentClientEventRtcCallSdpCreate`, `VoiceAgentServerEventRtcCallSdpCreated`, and `VoiceAgentServerEventRtcCallError` events carry the signaling exchange.
+  * Added the `agent_endpoint_conversations.get_agent_conversation_item_generated_audio`/`get_agent_conversation_item_generated_audio_content` methods for reading back a conversation item's *generated* audio, a subordinate artifact that can differ from what the listener heard when playback was interrupted, returning `VoiceGeneratedItemAudioResponse`. This is a new top-level operation group, distinct from `beta.agent_endpoint_conversations`.
+  * Added sub-agent consultation, letting a voice agent consult sibling Foundry text agents as background specialists mid-conversation, through the new `subagent_config` property on `VoiceAgentDefinition` (`VoiceAgentSubAgentConfig`, `VoiceAgentSubAgent`, `VoiceAgentSubagentResponsePolicy`), and the new `session.subagent.started`/`session.subagent.completed`/`session.subagent.aborted` realtime server events.
+  * Added an optional `conversation_engine` property on `VoiceAgentDefinition` (`VoiceConversationEngine`, `VoiceHostedAgentConversationEngine`) to delegate a voice agent's conversation handling to another hosted agent instead of configuring a model directly.
+* Added Microsoft 365 agent publishing:
+  * `project_client.agents.publish_to_microsoft365(agent_name, publish_scope=...)` publishes a Foundry agent to Microsoft 365 / Microsoft Teams and returns a `Microsoft365PublishResult`.
+  * `project_client.agents.get_microsoft365_publish_defaults(agent_name)` returns default and previously-published values (`Microsoft365PublishDefaults`) used to pre-populate a publish request.
+  * `project_client.agents.get_microsoft365_package(agent_name)` downloads the Microsoft 365 app package for an agent.
+  * Added the supporting `Microsoft365PublishScope`, `Microsoft365PermissionScopes`, `ActivityProtocolAccessBoundary`, `PublishApprovalStatus`, and `DigitalWorkerType` enums.
+* Added the `beta.agent_insight_monitors` operation group for creating and managing Agent Insights monitors and their runs (`create`/`get`/`update`/`delete`/`list`/`reset`, `begin_create_run`/`get_run`/`cancel_run`/`list_runs`, `get_insight`/`update_insight`/`list_insights`), along with the supporting `AgentInsightMonitor`, `AgentInsightMonitorCreate`, `AgentInsightMonitorUpdate`, `AgentInsightMonitorListItem`, `AgentInsightRun`, `AgentInsight`, and related models.
+* Added an optional `authorization` argument to `.beta.routines.create_or_update`, with `RoutineAuthorization` and `RoutineDispatchIdentity` for selecting the agent or routine creator identity.
 * Added optional Hosted Agent session defaults through `HostedAgentDefinition.session_configuration` and `SessionConfiguration`, including idle-timeout configuration.
-* Added the optional `authorization` argument to `.beta.routines.create_or_update`, with `RoutineAuthorization` and `RoutineDispatchIdentity` for selecting the agent or routine creator identity.
 * Added `ShellToolboxTool` and supporting container environment and network policy models, with the new `ToolboxToolType.SHELL` enum member.
 * Added `WebIQPreviewTool` and `WebIQPreviewToolboxTool`, with new `ToolType.WEB_IQ_PREVIEW` and `ToolboxToolType.WEB_IQ_PREVIEW` enum members.
 
+### Dependency update
+
+* Added an optional dependency on `websockets` (sync `client.realtime`) and `aiohttp` (async `async_client.realtime`), required only when using the new voice agent realtime streaming APIs.
+
+### Bugs Fixed
+
 
 ### Sample updates