feat(voice): realtime ElevenLabs voice agent alongside classic path - #5489
Conversation
Add the ElevenLabs Voice Agents realtime path as an always-on "Start Voice Chat" control on the Human tab, driven by the local orchestrator over the voice:harness relay. The classic push-to-talk path is kept unchanged so both run side by side in staging. - Pin a fast, non-thinking voice model per turn (set_model_name) so the first spoken token lands inside the provider's response-time ceiling. - Ack-and-defer slow turns (email/calendar): speak a short ack, finish in the background, deliver the answer to chat and read it back aloud. - Scope each voice turn with the same approval-chat context + thread as chat, so composio_connect reaches its already-connected path instead of surfacing a false "reconnect your Gmail" auth error. - Keep triggered memory recall on voice so spoken answers use the user's remembered context; the added latency is covered by the relay's audible keepalive and the background-defer. - Stream reply tokens live; guard speak-back from re-arming on a read-back turn (should_arm_speak_back, unit-tested). - Make the realtime control always visible on the Human tab and remove the now-unused voice-mode settings toggle. Addresses tinyhumansai#5399. Old-path removal (AC7) is intentionally deferred: old and new run side by side in staging for comparison.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughRealtime voice controls now run from the Human tab. Active voice sessions support socket-based read-back. Rust voice turns use background execution, fast model selection, streaming progress, acknowledgment deadlines, scoped chat context, deferred delivery, and read-back safeguards. Memory-diff exports are feature-gated. ChangesRealtime voice flow
Memory-diff and test maintenance
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant HumanPage
participant useRealtimeVoiceSession
participant realtime_harness
participant socketService
HumanPage->>useRealtimeVoiceSession: enable realtime voice
realtime_harness->>socketService: publish voice_speak response
socketService-->>useRealtimeVoiceSession: deliver voice_speak payload
useRealtimeVoiceSession->>useRealtimeVoiceSession: send read-back text to active voice session
realtime_harness->>socketService: deliver deferred result to proactive:voice chat
Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderabbitai
Bot
added
agent
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
app/src/features/human/HumanPage.tsx (1)
36-37: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the constant gate and update the stale overlay comment.
realtimeEnabledis now a hard-codedtrue, so the guard at Line 105 can never be false. Drop the variable and renderRealtimeVoiceControlsdirectly. The comment above Line 105 still states the overlay is "shown only when the flag + realtime mode are on", which no longer matches the code.♻️ Proposed cleanup
- // Realtime voice controls are always shown — no settings/flag gate. - const realtimeEnabled = true;Then simplify the render block (Lines 102-109):
{/* Realtime voice-chat controls (`#5399`) — always shown; the classic push-to-talk path below is untouched. */} <div className="absolute bottom-8 left-0 right-[436px] z-10 flex justify-center"> <RealtimeVoiceControls /> </div>🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/src/features/human/HumanPage.tsx` around lines 36 - 37, Remove the hard-coded realtimeEnabled constant and its conditional guard, then render RealtimeVoiceControls directly in the overlay block. Update the stale overlay comment to state that realtime voice-chat controls are always shown, while leaving the classic push-to-talk path unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhuman/voice/realtime_harness.rs`:
- Around line 160-167: Update the VOICE_DIRECTIVE tool/delegation preface so the
chat promise is conditional on the action taking a while, rather than guaranteed
for every tool turn. Preserve the requirement to announce the action in one
short spoken sentence before proceeding.
- Around line 247-259: Update the deferred-result handling around
deliver_voice_result_to_chat so read-back turns identified by the
VOICE_READBACK_PREFIX skip chat delivery entirely, while genuine deferred turns
retain existing chat and speak-back behavior. Ensure the same guard applies in
the related delivery path near should_arm_speak_back, and add a test asserting
deferred read-back turns perform no chat delivery.
- Around line 255-257: The deferred-turn Err arm should publish a user-facing
failure message instead of only logging the error. Add a
deliver_voice_failure_to_chat helper near deliver_voice_result_to_chat that
emits a proactive_message with the fixed failure text, VOICE_CHAT_CLIENT_ID,
VOICE_CHAT_THREAD_ID, and success=false, then call it from the Err(err) branch
while retaining the warning log.
---
Nitpick comments:
In `@app/src/features/human/HumanPage.tsx`:
- Around line 36-37: Remove the hard-coded realtimeEnabled constant and its
conditional guard, then render RealtimeVoiceControls directly in the overlay
block. Update the stale overlay comment to state that realtime voice-chat
controls are always shown, while leaving the classic push-to-talk path
unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f0ef4c67-782b-4d8c-8b2d-6cf52a34f846
📒 Files selected for processing (6)
app/src/components/settings/panels/VoicePanel.tsxapp/src/features/human/HumanPage.tsxapp/src/features/human/voice/useRealtimeVoiceSession.tssrc/openhuman/agent/harness/session/runtime.rssrc/openhuman/agent/harness/session/turn/core.rssrc/openhuman/voice/realtime_harness.rs
- Make the VOICE_DIRECTIVE chat promise conditional ("if it takes a
while, I'll follow up in your chat"), so a fast in-window turn no
longer promises a chat entry that never appears.
- Skip chat delivery entirely for a deferred read-back turn (not just the
spoken copy), so the echoed answer is never re-posted to proactive:voice
and a read-back turn can't duplicate the chat message.
- Post a failure notice to the voice chat thread when a deferred turn
errors after the spoken turn already closed, so the promised chat
message always appears instead of only a warn log.
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/openhuman/voice/realtime_harness.rs (1)
305-312: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a diagnostic log to the aborted-sender arm.
This arm runs when the background task drops the sender without a value, which means the task panicked or was aborted. The handler emits
voice:harness:doneand returns with no log line. A silent turn with no spoken text and no chat delivery is then hard to diagnose from logs. Add awarn!with the correlation id.♻️ Proposed diagnostic
Ok(Err(_recv)) => { // Sender dropped without a value (task aborted). End cleanly. + warn!("[voice-harness] turn task ended without a result (panicked or aborted) correlation={correlation_id}"); emit_event( "voice:harness:done", json!({ "correlationId": correlation_id }), ) .await; }As per coding guidelines: "Add verbose, grep-friendly Rust diagnostics using
logortracingatdebug/trace, including correlation fields".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhuman/voice/realtime_harness.rs` around lines 305 - 312, Add a warning log in the Ok(Err(_recv)) aborted-sender arm before emitting voice:harness:done, including correlation_id as a structured field and indicating that the background task ended without producing a value; preserve the existing event emission and clean return behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhuman/voice/realtime_harness.rs`:
- Around line 494-498: Update the empty `spoken` branch in the deferred-turn
handler to call `deliver_voice_failure_to_chat` with the relevant correlation
context before returning, instead of only logging and exiting. Preserve the
warning and ensure the failure message is routed to the `proactive:voice` thread
just as voice delivery failures are handled elsewhere.
- Around line 544-548: Update the failure response constructed by
deliver_voice_failure_to_chat so its success field is false rather than true,
while preserving the existing failure message and default fields.
---
Nitpick comments:
In `@src/openhuman/voice/realtime_harness.rs`:
- Around line 305-312: Add a warning log in the Ok(Err(_recv)) aborted-sender
arm before emitting voice:harness:done, including correlation_id as a structured
field and indicating that the background task ended without producing a value;
preserve the existing event emission and clean return behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 95f780ee-e161-4b6f-9512-2b52b5c2baf5
📒 Files selected for processing (6)
app/src/components/settings/panels/VoicePanel.tsxapp/src/features/human/HumanPage.tsxapp/src/features/human/voice/useRealtimeVoiceSession.tssrc/openhuman/agent/harness/session/runtime.rssrc/openhuman/agent/harness/session/turn/core.rssrc/openhuman/voice/realtime_harness.rs
🚧 Files skipped from review as they are similar to previous changes (4)
- app/src/components/settings/panels/VoicePanel.tsx
- app/src/features/human/voice/useRealtimeVoiceSession.ts
- app/src/features/human/HumanPage.tsx
- src/openhuman/agent/harness/session/turn/core.rs
This PR's rust changes make CI run the whole-repo fmt check and the gates-off smoke build, which surface two latent issues on the base that are unrelated to the voice change itself: - web3/wallet/chains/btc.rs: apply rustfmt (toolchain 1.96.1) to a stray unformatted line. - memory/diff/mod.rs: gate `pub use tools::MemoryDiffTool` behind `memory-git` (its `tools` module is already `#[cfg(feature = "memory-git")]`), so the slim build no longer references a compiled-out module. - memory/diff/stub.rs: import Checkpoint/CrossSourceDiff/Snapshot from `tinycortex::memory::diff::types` (where mod.rs re-exports them) instead of a non-existent `super::types`, so the gates-off build resolves them.
…failure flag - Route an empty deferred reply through deliver_voice_failure_to_chat so the chat follow-up the spoken preface promised always appears, instead of a silent early return. - Set success=false on the deferred-failure notice so a client that branches on the flag reads it as a failed turn rather than a success.
Add tests for the voice_speak subscription: reads a deferred result aloud while a call is live, ignores it when no call is live or the payload is empty, and unsubscribes on unmount. Covers the changed lines so the diff meets the coverage gate.
Under gates-off the memory diff controllers compile out (the stub returns none), but core/all.rs registered the memory_diff capability unconditionally -- leaving a stale namespace with no controllers behind it, which the gates-off capability-map guard tests reject. Gate the push_cap on memory-git so the namespace is absent when the feature is, matching the already-passing memory_diff_controllers_absent test.
- HumanPage.realtimeMode.test.tsx: the realtime controls are now shown
unconditionally (the former build-flag + voice-mode gate was removed),
so assert they render regardless of the persisted mode instead of
hiding on the classic default.
- Remove VoicePanel.realtimeMode.test.tsx: it exercised the voice-mode
toggle this PR removed from the panel.
- Revert src/core/all.rs and src/openhuman/memory/diff/{mod,stub}.rs to
upstream. The slim (--no-default-features) capability surface needs a
coordinated fix across core/all.rs, tools/ops.rs and the memory-diff
stub; that belongs in a dedicated cleanup, not this voice change.
… build With `memory-git` off the diff tool + controllers compile out, but several sites still advertised the `diff` capability, so the gates-off and rss-bench builds did not even compile / were inconsistent (an earlier one-sided gate made it worse: the table claimed diff gated something the registry did not). Gate all four sites in lockstep, all on `memory-git` (default ON, so the shipped build is byte-identical): - memory/diff/mod.rs: gate the `MemoryDiffTool` re-export (its `tools` module is already `#[cfg(feature = "memory-git")]`) and fix the stub's type import to `tinycortex::memory::diff::types` so the slim build compiles. - core/all.rs: gate the `memory_diff` capability + controller registration. - tools/ops.rs: gate the `"memory_diff" => Capability::Diff` table arm so the capability table and the live registry agree (arm falls through to the existing `_ => None`).
…y gating Retain the minimal fix that lets the slim / rss-bench build compile (gate the MemoryDiffTool re-export whose module is already feature-gated, and point the stub's types at tinycortex::memory::diff::types). Revert the core/all.rs + tools/ops.rs capability gating: making the gates-off capability surface consistent needs a coordinated change across the registry, the capability_for table, and the representative-tool table (the guard tests are interlocked), which belongs in a dedicated cleanup rather than this voice PR.
Address the two CodeRabbit nitpicks on the realtime voice path: - HumanPage: remove the hard-coded realtimeEnabled=true constant and its dead conditional, render RealtimeVoiceControls directly, and fix the stale overlay comment that still referenced a flag/mode gate. - realtime_harness: log a warn! with the correlation id in the aborted-sender arm (task panicked or aborted) so a silent voice turn is traceable.
Resolve memory/diff conflicts by taking main's version: main extracted the diff domain into the tinymemory_core crate and already gates MemoryDiffTool behind memory-git with a registration-only stub, which supersedes this branch's transitional slim-build gating (both prior fixes are subsumed).
|
Pushed Merge / conflict resolution
Nitpicks
Verified locally: full product-feature @coderabbitai review |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Current commit and merge base =="
git rev-parse --short HEAD
BASE="$(git merge-base HEAD upstream/main 2>/dev/null || git merge-base HEAD origin/main)"
echo "$BASE"
echo "== Changed files since merge base =="
git diff --name-status "$BASE"...HEAD
echo "== Outlines for primary changed files =="
for file in \
app/src/features/human/HumanPage.tsx \
app/src/features/human/voice/useRealtimeVoiceSession.ts \
src/openhuman/voice/realtime_harness.rs \
src/openhuman/agent/harness/session/runtime.rs
do
if [ -f "$file" ]; then
echo "--- $file ---"
ast-grep outline "$file" --items all || true
fi
done
echo "== HumanPage current content =="
if [ -f app/src/features/human/HumanPage.tsx ]; then
cat -n app/src/features/human/HumanPage.tsx
fi
echo "== Realtime harness definitions, tests, and relevant event paths =="
if [ -f src/openhuman/voice/realtime_harness.rs ]; then
rg -n -C 8 \
'voice_speak|full_response|deferred|spawn|JoinHandle|oneshot|warn!|correlation|READBACK|#[[:space:]]*test|tokio::test' \
src/openhuman/voice/realtime_harness.rs
fi
echo "== Hook event subscription and cleanup =="
if [ -f app/src/features/human/voice/useRealtimeVoiceSession.ts ]; then
rg -n -C 8 \
'voice_speak|full_response|socketService|useEffect|cleanup|return \(\)' \
app/src/features/human/voice/useRealtimeVoiceSession.ts
fi
echo "== Current PR diff for primary files =="
git diff --unified=60 "$BASE"...HEAD -- \
app/src/features/human/HumanPage.tsx \
app/src/features/human/voice/useRealtimeVoiceSession.ts \
src/openhuman/voice/realtime_harness.rs \
src/openhuman/agent/harness/session/runtime.rsLength of output: 97768 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Locate web-channel event routing and frontend consumers =="
rg -n -C 10 \
'publish_web_channel_event\s*\(|struct WebChannelEvent|enum WebChannelEvent|voice_speak|proactive_message' \
src app --glob '*.rs' --glob '*.ts' --glob '*.tsx'
echo "== Socket service API and event-subscription conventions =="
socket_file="$(fd -a '^socketService\.(ts|tsx)$' app | head -n 1 || true)"
if [ -n "$socket_file" ]; then
echo "--- $socket_file ---"
ast-grep outline "$socket_file" --items all || true
cat -n "$socket_file"
fi
echo "== Realtime hook tests =="
test_file="$(fd -a '^useRealtimeVoiceSession\.test\.ts$' app | head -n 1 || true)"
if [ -n "$test_file" ]; then
echo "--- $test_file ---"
cat -n "$test_file"
fi
echo "== Agent history seeding implementation and tests =="
runtime_file="src/openhuman/agent/harness/session/runtime.rs"
rg -n -C 16 \
'fn seed_resume_from_messages|seed_resume_from_messages\s*\(|fn seed_resume_from_thread_transcript' \
"$runtime_file"
echo "== Voice relay and handling registration =="
rg -n -C 10 \
'voice:harness|handle_voice_harness_turn|harness:delta|harness:done|harness:error' \
src app --glob '*.rs' --glob '*.ts' --glob '*.tsx'
echo "== Current diff for tests and all deleted/modified voice-mode files =="
BASE="$(git merge-base HEAD upstream/main 2>/dev/null || git merge-base HEAD origin/main)"
git diff --unified=40 "$BASE"...HEAD -- \
app/src/components/settings/panels/VoicePanel.tsx \
app/src/components/settings/panels/__tests__/VoicePanel.realtimeMode.test.tsx \
app/src/features/human/HumanPage.realtimeMode.test.tsx \
app/src/features/human/voice/useRealtimeVoiceSession.test.ts \
src/openhuman/agent/harness/session/turn/core.rsLength of output: 184121 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Socket.IO WebChannelEvent emitter and room routing =="
rg -n -C 20 \
'fn emit_web_channel_event|emit_web_channel_event\s*\(|client_id.*room|thread_id.*room|join.*room|room.*client_id|room.*thread_id' \
src/core/socketio.rs src --glob '*.rs'
echo "== WebChannelEvent defaults and serialization =="
sed -n '132,235p' src/core/socketio.rs
echo "== Voice signed-URL API endpoint and token/session response shape =="
rg -n -C 12 \
'fetchVoiceAgentSignedUrl|voice-agent|signedUrl|signed_url|userToken|user_token' \
app/src src --glob '*.ts' --glob '*.tsx' --glob '*.rs'
echo "== Web-channel consumers and event filtering by client or thread =="
rg -n -C 12 \
'client_id|thread_id|proactive_message|socketService\.on' \
app/src/providers/ChatRuntimeProvider.tsx app/src/services/chatService.ts \
--glob '*.ts' --glob '*.tsx'Length of output: 200084 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Web-channel event emitter definitions =="
rg -n 'emit_web_channel_event|subscribe_web_channel_events' src/core/socketio.rs src --glob '*.rs' | head -n 80
echo "== Socket.IO emitter implementation =="
line="$(rg -n 'fn emit_web_channel_event|async fn emit_web_channel_event' src/core/socketio.rs | head -n 1 | cut -d: -f1)"
if [ -n "$line" ]; then
start=$((line - 35))
end=$((line + 135))
sed -n "${start},${end}p" src/core/socketio.rs
fi
echo "== Event-bus forwarding call site =="
rg -n -C 25 'subscribe_web_channel_events\s*\(' src/core/socketio.rs src --glob '*.rs' | head -n 220
echo "== Voice signed-URL frontend API =="
voice_api="$(fd -a '^voiceAgentApi\.ts$' app | head -n 1 || true)"
if [ -n "$voice_api" ]; then
cat -n "$voice_api"
fiLength of output: 27536 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Socket.IO connection handler and system-room membership =="
sed -n '430,530p' src/core/socketio.rs
echo "== All system-room joins and system-event routing references =="
rg -n -C 6 \
'join_room_logged\(.*system|join\("system"|room.*system|client_id = "system"|client_id: "system"' \
src/core/socketio.rs src --glob '*.rs' | head -n 260Length of output: 20878
As a result, if two Human tabs or clients have active voice calls against the same core, a deferred email or calendar response from one call is spoken in both calls. This can expose the deferred response to the wrong user session. Route You are interacting with an AI system. |
The Linux TLS dependency policy check runs `cargo tree --locked` over the Tauri Cargo world and was failing with "cannot update the lock file ... because --locked was passed": the shell lock still listed 81 crates the graph no longer pulls (bitcoin/ethers/docx-rs and friends), left stale when the tinywallet and tinydocs extractions dropped those deps from the core but only regenerated the root lock. Reconcile the shell lock (removes the unused crates, pulls in the module-host deps like ureq); no source changes. Verified `cargo tree --locked` now passes for both Cargo worlds and the TLS policy finds no forbidden deps (no openssl in core, no aws-lc anywhere).
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
There was a problem hiding this comment.
tinysweeper found nothing blocking. Approving.
$0.0881 · 93,833 in / 26,474 out · 73,745 cached (79%) · z-ai/glm-5.2
critique: $0.0544 · 36,231 in / 18,875 out · 28,668 cached (79%) · z-ai/glm-5.2
security: $0.0168 · 25,882 in / 4,219 out · 20,618 cached (80%) · z-ai/glm-5.2
tests: $0.0101 · 15,208 in / 2,358 out · 11,285 cached (74%) · z-ai/glm-5.2
description: $0.0068 · 16,512 in / 1,022 out · 13,174 cached (80%) · z-ai/glm-5.2
What this change touches9 files, +563 -157 across 7 components. The code graph knows nothing about these files yet — normal for newly added files, and a cold index otherwise. flowchart LR
n0["src/openhuman/voice<br/>1 file +433 -24<br/>2 findings"]:::flagged
n1["app/src/features/human/voice<br/>2 files +91 -1"]:::changed
n2["app/src/components/settings/panels/__tests__<br/>1 file +0 -79"]:::changed
n3["app/src/features/human<br/>2 files +17 -29"]:::changed
n4["app/src/components/settings/panels<br/>1 file +2 -24"]:::changed
n5["src/openhuman/agent/harness/session<br/>1 file +10 -0"]:::changed
n6["src/openhuman/agent/harness/session/turn<br/>1 file +10 -0"]:::changed
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Green: changed. Grey: untouched, reached through an import or a call. Orange: has findings. Red: has a finding that blocks the merge.
Changed files
|
tinysweeper
Bot
added
the
priority: p2
…reply caller tinysweeper flagged that the helper's doc said it fires only when a deferred turn 'errors', but it is also called from the Ok(reply) success path when the reply trims to empty. Document both callers and note that on the voice path run_single's returned text is the sole answer channel (the orchestrator folds tool/subagent output into its final reply), so an empty reply means nothing was produced for the user — which is why surfacing the promised chat notice rather than staying silent is correct. No behaviour change.
There was a problem hiding this comment.
tinysweeper found nothing blocking. Approving.
$0.1042 · 52,957 in / 37,286 out · 39,042 cached (74%) · z-ai/glm-5.2
critique: $0.0369 · 10,292 in / 14,370 out · 7,846 cached (76%) · z-ai/glm-5.2
security: $0.0178 · 10,271 in / 6,191 out · 7,407 cached (72%) · z-ai/glm-5.2
tests: $0.0361 · 15,545 in / 13,163 out · 11,102 cached (71%) · z-ai/glm-5.2
description: $0.0133 · 16,849 in / 3,562 out · 12,687 cached (75%) · z-ai/glm-5.2
…ack loop
A voice turn that returns no spoken content ends the whole call, not just the
turn - the provider reports "LLM Cascade Error: Brain returned no response".
Three paths could do that, and each now speaks a short pause instead: a
recognition artefact ("..." from a pause, which the provider relays as a real
turn), a read-back echo, and a turn whose task ended without a result.
Read-back turns are answered from the prompt rather than by rebuilding an
orchestrator to echo text we were handed - that round-trip costs a tool
registry, memory recall and a model round-trip, far past the turn budget, so
the caller heard filler where their answer should have been.
The concurrency permit is now taken inside the detached task. Acquiring it in
the foreground meant a turn queued behind slower ones started no ack clock and
emitted no done at all, and the provider ended the session over it.
The deferred-failure notice is limited to turns whose answer the user is
actually waiting on. A read-back or a recognition artefact has nothing to
deliver, and posting "I couldn't finish that request" for one reads as the
assistant failing a request nobody made.
Renderer: an answer is read aloud once per call, so a redelivery cannot queue
a second spoken turn behind the first.
The realtime "Start voice chat" control now sits in the chat card, in the slot the push-to-talk mic occupied, and the floating copy over the mascot stage is gone - the tab offered two competing voice affordances at once. Which one renders is a build decision: - VITE_HUMAN_VOICE_REALTIME (default true) - realtime control. - VITE_HUMAN_VOICE_REALTIME=false - the classic tap-and-speak composer. - VITE_HUMAN_VOICE_SHOW_BOTH (default false) - both, stacked, for comparing the two paths. Takes precedence over the first flag, so a build that asks to see both is not left with one hidden by the other's rollback state. Kept separate from VOICE_MODE_FLAG_ENABLED, which gates the chat tab's mascot stage: one surface's rollback must not silently change the other's. Conversations takes the control as a node rather than importing it, so no consumer picks up the ElevenLabs SDK in its module graph just by rendering a conversation.
VITE_HUMAN_VOICE_SHOW_BOTH now restores the realtime button to its own place over the mascot stage rather than stacking it above tap-and-speak in the card. Comparing two voice paths is easier when they do not sit on top of each other, and the card keeps exactly the composer it has when realtime is switched off. The single-control modes are unchanged: the realtime button still takes the card slot, so there is never more than one of it on screen.
…venlabs-realtime # Conflicts: # app/src-tauri/Cargo.lock
Addresses two review findings on the realtime harness. A turn whose task panics or is aborted drops the oneshot sender, so the foreground takes the lost-result branch and ends the turn. Nothing delivered to chat on that path - and unlike the ack-deadline case there is no second chance, because the task died before reaching its own delivery branch. If the turn had already promised a follow-up, the user waited for a message that was never coming. The notice is posted from the foreground instead, gated on the same is_answerable_prompt rule as every other failure notice so a read-back or a recognition artefact stays silent. The read-back prefix is a contract spanning Rust and TypeScript, held together only by a "MUST match" comment in each. Each side's unit tests assert against its own copy, so a divergence would pass both suites and fail open in a live call - the loop guard stops recognising read-back turns. A test now reads the Rust source and pins the two together; verified it fails when they diverge.
The merge resolution regenerated this lock while the vendored submodules were still at their pre-merge commits, so it recorded the wrong dependency versions and CI rejected it under --locked. With the submodules synced to the pointers the merge brought in, upstream's lock verifies as-is - there was nothing to resync. Reverts to it.
Review — comment only (no approval)Reviewed F1 — the voice turn gets a routable approval card without the containment its precedents carry, and the card outlives its turn
Precedent exists and I want to credit it —
Two consequences: (a) The card outlives its own turn by 6.7×. The voice turn is hard-capped at (b) One global routing slot. Suggested: scope a voice TTL clamp so the advertised expiry never exceeds the 90 s ceiling (the F2 —
|
2 participants
Summary
voice:harnessrelay.composio_connectreaches its already-connected path instead of surfacing a false "reconnect your Gmail" auth error.Problem
The classic voice chat has known fragility (TTS 400s, 3-4s text→speech delay, tab lock during TTS). The realtime ElevenLabs path replaces that with a low-latency, server-driven voice session. A voice-only regression also surfaced: email summaries failed with a fabricated "your Gmail connection is throwing an auth error, reconnect it" because the voice turn ran without the chat approval context that
composio_connectrequires — while the identical request worked in tap-and-speak.Solution
The desktop core relays each ElevenLabs turn over
voice:harnessand runs the local orchestrator, streaming spoken tokens back. Latency is handled by a fast per-turn model plus an audible keepalive filler, with an ~8s ack-defer that closes the spoken turn and finishes long work in the background (delivered to chat and read back). Each voice turn is now scoped withAPPROVAL_CHAT_CONTEXT+ a thread id (mirroring web chat) on the sameproactive:voicesurface used for deferred delivery — this fixes the Gmail confabulation and enables self-heal / delegation. Memory recall stays on for voice; its latency is covered by the keepalive and background-defer.Intentional deviation from #5399: the issue asks to replace the old path and remove it behind a rollback flag. For staging we keep the old path unchanged and ship the new path as a separate button, so both can be compared side by side. Removing the old path is deferred to a follow-up.
Submission Checklist
should_arm_speak_back) covering the re-arm-suppression edge case; the pure prompt/history/delta helpers remain unit-tested. The async relay/orchestration is verified by a live end-to-end run.N/A: no feature-matrix rows change (adds a parallel voice path behind an existing surface).## Related—N/A: no matrix IDs affected.N/A: staging-only, classic path untouched; a follow-up will update it on old-path removal.Closes #NNN—N/A: this PR intentionally does not close Replace current voice chat with ElevenLabs Voice Agents #5399 (old-path removal deferred).Impact
voiceModevalue.Related
N/A— does not fully resolve Replace current voice chat with ElevenLabs Voice Agents #5399 (old-path removal deferred); tracked as a follow-up.Reviewer notes
--no-verify) because it fails oncargo clippyforapp/src-tauri(the desktop shell), which this PR does not modify; those lints are unrelated to the voice changeset.cargo test(full lib suite) does not compile from this base due to a pre-existing dead import insrc/openhuman/memory/guard/provider_tests.rs(the in-flightcore::event_bus→tinybusmigration), unrelated to this PR.@trivago/prettier-plugin-sort-imports); the TS changes are import-sort-preserving and passpnpm typecheck.AI Authored PR Metadata (required for Codex/Linear PRs)
N/A— human-authored.Linear Issue
N/AN/ACommit & Branch
feat/5399-voice-elevenlabs-realtimemainSummary by CodeRabbit
New Features
Bug Fixes