Skip to content

Commit 5307dfe

Browse files
knitvogerknitvo
andauthored
[agentserver] Add typed Voice event relay (#48544)
* [agentserver] Add typed Voice event relay * [agentserver] Add Voice connection termination signal * [agentserver] Resolve Voice review findings * [agentserver] Harden Voice transport edge cases * [agentserver] Harden Voice transport ownership * [agentserver] Fix Voice cancellation and disconnect handling * fox * test * fix * Validate stateless voice codec fields * Preserve Voice shutdown cancellation on Python 3.10 * Restore WebSocket route tests and CSpell terms * teardown race * [agentserver] Document Voice transport helper * Deep-freeze SessionStart caller context --------- Co-authored-by: wujin <wujin@microsoft.com>
1 parent 6528080 commit 5307dfe

22 files changed

Lines changed: 8314 additions & 7 deletions
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
changeKind: feature
3+
packages:
4+
- azure-ai-agentserver-invocations
5+
---
6+
7+
Added the experimental `azure.ai.agentserver.invocations.voice` typed event relay, a synchronous connection-termination signal for developer-owned task cancellation, and the `basic_voice_agent` full-duplex sample.

sdk/agentserver/azure-ai-agentserver-invocations/MANIFEST.in

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
include *.md
22
include LICENSE
33
recursive-include tests *.py
4-
recursive-include samples *.py *.md
4+
recursive-include samples *.py *.md *.txt *.yaml
55
include azure/__init__.py
66
include azure/ai/__init__.py
77
include azure/ai/agentserver/__init__.py

sdk/agentserver/azure-ai-agentserver-invocations/README.md

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ The `azure-ai-agentserver-invocations` package provides the invocation protocol
44

55
- **HTTP** (`invocations` protocol) — `POST /invocations`, `GET /invocations/{id}`, `POST /invocations/{id}/cancel`, `GET /invocations/docs/openapi.json`, `GET /invocations/docs/asyncapi.{json,yaml}`.
66
- **WebSocket** (`invocations_ws` protocol) — full-duplex streaming at `/invocations_ws`, registered with `@app.ws_handler`.
7+
- **Voice Live Bridge** — an experimental typed event relay in the `azure.ai.agentserver.invocations.voice` submodule, layered on `invocations_ws`.
78

89
## Getting started
910

@@ -309,6 +310,102 @@ The handler receives a Starlette [`WebSocket`][starlette-ws] and returns `None`.
309310

310311
[starlette-ws]: https://www.starlette.io/websockets/
311312

313+
## Typed Voice Live Bridge submodule (preview)
314+
315+
`VoiceAgentServerHost` provides typed `on_<event>` decorators over the existing
316+
`invocations_ws` transport. Each callback receives an immutable event and a
317+
send-only `Session`:
318+
319+
```python
320+
from azure.ai.agentserver.invocations.voice import (
321+
ResponseCreated,
322+
ResponseDone,
323+
ResponseOutputTextDone,
324+
Session,
325+
SessionReady,
326+
SessionRejected,
327+
SessionStart,
328+
UserMessage,
329+
VoiceAgentServerHost,
330+
new_item_id,
331+
new_response_id,
332+
)
333+
334+
app = VoiceAgentServerHost()
335+
336+
337+
@app.on_session_start
338+
async def on_session_start(session: Session, event: SessionStart) -> None:
339+
if event.protocol_version != "1.0":
340+
await session.send(
341+
SessionRejected(code="protocol_mismatch", retriable=False)
342+
)
343+
return
344+
# Restore durable application state here when event.reconnect is true.
345+
await session.send(SessionReady())
346+
347+
348+
@app.on_user_message
349+
async def on_user_message(session: Session, event: UserMessage) -> None:
350+
response_id = new_response_id()
351+
item_id = new_item_id()
352+
await session.send(
353+
ResponseCreated(response_id=response_id, in_reply_to=(event.item_id,))
354+
)
355+
await session.send(
356+
ResponseOutputTextDone(
357+
response_id=response_id,
358+
item_id=item_id,
359+
text="Hello from the hosted text agent.",
360+
)
361+
)
362+
await session.send(ResponseDone(response_id=response_id))
363+
```
364+
365+
The submodule is deliberately a thin typed event relay. It decodes one inbound frame,
366+
dispatches the corresponding callback, encodes explicit outbound messages, and
367+
serializes concurrent WebSocket writes. It does **not** own pending responses,
368+
terminal arbitration, timeout/cancel operations, generation tasks, history, or
369+
reconnect state.
370+
371+
When the peer or proxy closes the WebSocket, `@app.on_disconnect` receives a
372+
local `SessionDisconnected` event. This callback represents only the observed
373+
peer disconnect.
374+
375+
`@app.on_connection_terminating` is the common cleanup signal for every
376+
in-process exit from the connection handler, including peer disconnect, local
377+
protocol close, callback failure, transport failure, and task cancellation. It
378+
is synchronous so applications can promptly call `Task.cancel()` or set their
379+
own stop signals without making WebSocket teardown wait for asynchronous
380+
cleanup. The callback must be non-blocking and must not send frames. The SDK
381+
invokes it once as each connection handler unwinds, and applications must keep
382+
their signaling idempotent. The SDK does not retain, join, or guarantee
383+
completion of application-owned tasks.
384+
385+
For the Voice WebSocket relay, shutdown cancellation remains cancellation while
386+
the SDK is awaiting WebSocket accept or receive, even if the ASGI transport
387+
returns normally or translates the cancellation into a standard exception. This
388+
guarantee requires the transport operation to eventually settle after receiving
389+
cancellation; a transport that suppresses cancellation and never returns is
390+
outside the contract.
391+
392+
After repeated cancellation requests, the Voice endpoint is guaranteed to remain
393+
cancelled. The exact nested `asyncio.CancelledError` instance or message selected
394+
from a transport-defined exception graph is unspecified.
395+
396+
Voice callback cancellation is cooperative. A callback that catches
397+
`asyncio.CancelledError` must re-raise it after its own cleanup. If application
398+
code catches cancellation and returns normally, recovery is outside the SDK
399+
contract; the SDK does not forcibly terminate or retain that callback.
400+
401+
For full-duplex streaming, the agent creates and owns a generation task, returns
402+
from `on_user_message`, and cancels that task from `on_barge_in`,
403+
`on_response_cancelled`, `on_response_timeout`, or
404+
`on_connection_terminating`. Each task remains responsible for its own
405+
asynchronous resource cleanup. See the complete
406+
[`basic_voice_agent`](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/agentserver/azure-ai-agentserver-invocations/samples/basic_voice_agent)
407+
sample.
408+
312409
### Reference: configuration
313410

314411
| Environment variable | Default | Description |
@@ -340,6 +437,7 @@ Visit the [Samples](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/
340437
| [async_invoke_agent](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/agentserver/azure-ai-agentserver-invocations/samples/async_invoke_agent/) | Long-running operations with polling and cancellation |
341438
| [ws_invoke_agent](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/agentserver/azure-ai-agentserver-invocations/samples/ws_invoke_agent/) | Combined `POST /invocations` (HTTP) and `/invocations_ws` (WebSocket) host |
342439
| [ws_bidirectional_streaming_agent](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/agentserver/azure-ai-agentserver-invocations/samples/ws_bidirectional_streaming_agent/) | Full-duplex `/invocations_ws` agent: concurrent token streams + mid-flight cancel (relies on the SDK's WS protocol Ping/Pong keep-alive, not application-level heartbeats) |
440+
| [basic_voice_agent](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/agentserver/azure-ai-agentserver-invocations/samples/basic_voice_agent/) | Typed Voice Live Bridge callbacks with developer-owned full-duplex streaming and cancellation |
343441

344442
## Contributing
345443

0 commit comments

Comments
 (0)