[CUA] Expose W365 tools to the model and harden session-expiry recovery#325
Merged
Merged
Conversation
Expose non-duplicate W365 MCP tools to the model as function tools, sanitize conversation history before each model call, and recover from expired sessions on the screenshot and exposed-tool paths by starting a fresh session.
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
desmarest
changed the title
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Windows 365 Computer Use sample orchestrator to (1) expose eligible W365 MCP tools to the model as function tools, and (2) improve resiliency by sanitizing conversation history and hardening recovery from expired/lost W365 sessions.
Changes:
- Exposes non-excluded W365 MCP tools as function tools (while tracking exposed names and appending tool-steering instructions only when applicable).
- Adds conversation-history sanitization to prevent invalid next-turn requests (unpaired calls/outputs, dangling reasoning).
- Refactors session-expiry recovery into shared helpers, including a “start fresh session” path and screenshot retry with recovery.
…oning sanitization Parse non-string function_call arguments via GetRawText instead of throwing, and only emit buffered reasoning before its paired tool call so it can't dangle.
denzelpfeifer
previously approved these changes
Jun 19, 2026
Concurrent RunAsync turns for the same conversation could each take the reuse "initial screenshot" branch and append an input_image to the shared ConversationHistory, producing the Azure OpenAI 400 "Computer tool cannot use multiple image inputs." Add a per-conversation SemaphoreSlim (TurnLock) on ConversationSession and split RunAsync into a thin locking wrapper around a RunTurnAsync core so turns for the same conversation run one at a time.
These W365 tools were grouped with the native-CUA duplicates and excluded from model exposure, but the native computer tool cannot emit them — they are query-only and have no native equivalent, so excluding them removed the capability entirely. Drop them from the native-duplicate set.
MohamedAbdekader
approved these changes
Jun 19, 2026
ajmfehr
approved these changes
Jun 19, 2026
gwharris7
approved these changes
Jun 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Default W365 tool exposure. Non-duplicate W365 MCP tools are exposed to the model as function tools.
IsExcludedFromModelExposurefilters out the session-lifecycle wrappers and the W365 tools that duplicate native CUA actions (click/type_text/press_keys/scroll/move_mouse/drag_mouse/get_cursor_position/get_screen_size). Exposed names are tracked so the loop routes thosefunction_calls through the W365 path viaInvokeExposedW365ToolAsync(errors returned as text, never aborting the turn). AToolSteeringAddendumis appended to the system prompt only when tools are actually exposed, so the non-CUA fast path is never told about tools it doesn't have.Conversation sanitization.
SanitizeConversationHistoryruns before every model call, dropping unpaired function/computer calls, orphan outputs, and dangling reasoning items that would otherwise make the request invalid.Session-expiry recovery.
RecoverAndStartFreshSessionAsyncreleases a stale session and starts a fresh one on the existing MCP client. Both the initial and post-action screenshots run throughCaptureScreenshotWithRecoveryAsync, and the computer-call recovery sites route through a shared retry helper.Active-session reset.
ConversationSession.ClearActiveSessionclears a leftover tracked session id during recovery soStartW365SessionAsyncactually starts a new session instead of short-circuiting on a stale id.