Problem
In live conversations, people sometimes need fast, accurate information they have not memorized, such as a support agent fielding a product question or accessibility scenarios where real-time assistance helps someone participate. Batch, after-the-call processing cannot help in the moment.
Proposed solution
An advanced mode that switches to a real-time streaming transcription path (smaller model, continuous VAD), detects a question or triggers on a hotkey, and dispatches it plus the active custom prompt and a typed context to the agent, streaming the response into the expanded Dynamic Island. Ground answers in a knowledge base via retrieval so responses are accurate. Present suggestions a human adapts, not autopilot. Provide an "answer the last thing" hotkey to reduce false triggers. Support model tiering. Requires custom prompts to be set. Intended for the user's own support, accessibility, and preparation scenarios, not for misrepresenting one's own ability in a live evaluated setting.
Privacy considerations
This is a significant departure from the on-device batch default: it streams call audio continuously to the external agent and pulls from a knowledge base. Make that explicit in settings, require an informed opt-in, and show clearly when the mode is live. Consider all-party awareness implications of real-time processing of a live conversation.
Problem
In live conversations, people sometimes need fast, accurate information they have not memorized, such as a support agent fielding a product question or accessibility scenarios where real-time assistance helps someone participate. Batch, after-the-call processing cannot help in the moment.
Proposed solution
An advanced mode that switches to a real-time streaming transcription path (smaller model, continuous VAD), detects a question or triggers on a hotkey, and dispatches it plus the active custom prompt and a typed context to the agent, streaming the response into the expanded Dynamic Island. Ground answers in a knowledge base via retrieval so responses are accurate. Present suggestions a human adapts, not autopilot. Provide an "answer the last thing" hotkey to reduce false triggers. Support model tiering. Requires custom prompts to be set. Intended for the user's own support, accessibility, and preparation scenarios, not for misrepresenting one's own ability in a live evaluated setting.
Privacy considerations
This is a significant departure from the on-device batch default: it streams call audio continuously to the external agent and pulls from a knowledge base. Make that explicit in settings, require an informed opt-in, and show clearly when the mode is live. Consider all-party awareness implications of real-time processing of a live conversation.