Read aloud: speak an agent turn from its footer

[kaspesi]

Every completed agent turn gets a speaker button in its footer, next to copy. Press it and the daemon speaks the turn's closing message — the prose after the last tool call, not the narration in between. Press again to stop.

What it speaks

Only the prose after the turn's last tool call. Copy takes the whole turn; reading a long turn from the top would replay narration the user already watched scroll by.

The turn above contains two ToolSearch calls and one line of prose. Pressing its button synthesizes exactly one 2.03 s segment — the closing line, with the tool calls and the <spoken-input> wrapper above them excluded.

Turns that end on a tool call have nothing to say, so they show no button.

How it works

Piece	What it does
agent-stream/strategy.ts	collectAssistantTurnSpeech — walks the turn backward, breaks on tool_call
read-aloud/turn-read-aloud-button.tsx	The footer button; owns gating, stop intent, failure copy
read-aloud/read-aloud-store.ts	Playback state machine: status, owning turn, owning host
read-aloud/use-read-aloud-route-guard.ts	Stops playback when the route leaves the owning host
read-aloud/use-read-aloud-host.ts	Route-host binding
server/session/voice/read-aloud-controller.ts	Sanitize → split → synthesize with 2-segment prefetch → stream
server/speech/read-aloud-text.ts	Strips markdown and Paseo wrapper tags before synthesis
server/speech/tts-text-splitter.ts	Extracted from tts-manager.ts; shared with voice mode
client/src/daemon-client.ts	startReadAloud() returns a handle with cancel()
protocol/src/messages.ts	speech.tts.read_aloud.request → N .response segments; plus cancel

Uses the existing voice-mode TTS provider (local Kokoro by default, or OpenAI) rather than a second provider stack — that one already has config, env vars, and docs plumbing. Synthesis is segmented with a 2-segment prefetch because local TTS returns raw 24 kHz PCM (~48 KB/s), so a long message in one frame would be megabytes.

Three things carry state that a per-turn button makes non-obvious:

One voice at a time. Playback is a single app-wide slot, so the snapshot names the turn that owns it. Without that, every footer subscribing to the same store would render itself as speaking. Pressing another turn's button supersedes the first.

Playback belongs to a host. The Stop control lives in a turn footer, so navigating to another workspace would unmount it while audio kept playing — unstoppable. The store records the owning host and a guard at the app root stops playback when the route leaves it. Deliberately not a per-button useEffect cleanup: footers unmount during ordinary list virtualization, which would kill playback on scroll.

Cancellation reaches in-flight decodes. A segment crosses two awaits before it registers as active playback. A stop landing in that window would otherwise have nothing to cancel, and the segment would start playing after the user asked for silence. A generation counter closes it.

Scope

Web and Electron only. read-aloud-audio.ts is a playback stub on native, so the button hides where it would be silent. Nothing else blocks iOS/Android — implementing native audio would enable it with no other change.

Bound to the route's host. The text is workspace content, so it is spoken by the host it came from, never another paired daemon. No fallback: a route host that doesn't advertise the capability shows no button, and neither does a route with no host at all (settings, history).

Capability-gated on server_info.features.readAloud (COMPAT(readAloud), v0.2.5). Hosts without it get no button. No fallback path.

Evidence

The GIF is silent, so playback is proven numerically, not by the UI agreeing with itself — AudioBufferSourceNode.start was instrumented to measure what actually reached the output.

Check	Result
Audio reaches the output	segments at 24 kHz, peaks 0.28–0.43, non-silent
Speaks only post-tool-call prose	2 ToolSearch calls + 1 line → exactly one 2.03 s segment
Stop cancels synthesis	0 new segments in the 8 s after stop — the cancel reached the daemon
One owner at a time	press turn B while A plays → B shows Stop, A reverts to idle
Slot released on completion	all buttons idle after playback ends, no explicit stop needed
Route change stops playback	navigate to /sessions mid-read → stop called, 0 new segments
Scrolling does not stop playback	8 aggressive scrolls through virtualization → stop never called
Decode-race cancellation	unit tests fail with the generation check removed, pass with it restored
Shared audio engine unharmed	full src/voice/ suite green — voice mode and realtime voice unaffected

Platform matrix (per docs/qa.md):

Platform	Tested	Notes
Web (browser)	✅	Full flow: play, stop, ownership handoff, route guard, scroll-safety
Desktop macOS (Electron)	✅	Driven over CDP against npm run dev:desktop: buttons render, audio peak 0.392, route guard fires (stop called, 0 new segments over 8 s)
Desktop Windows / Linux	⬜	Not run locally; same web bundle and audio path as macOS Electron, and CI builds all three
iOS	n/a	Button hidden — read-aloud-audio.ts is a playback stub, so there is nothing to show
Android	n/a	Same as iOS

Automated: typecheck, lint, format clean. 229 tests across src/voice/, src/read-aloud/, src/agent-stream/, and src/i18n/, including 4 cases for the speech collector (after last tool call / no tool calls / ends on a tool call → empty / both traversal directions), ownership and host-ownership cases on the store, the decode-race cases, and locale key parity.

Known gaps

1. Segment gaps with local Kokoro. It runs ~1× realtime with a ~6 s cold start, and the splitter emits one segment per sentence — so a short first sentence drains before the next finishes synthesizing. Measured 6.1 s of silence mid-read. Three fixes, none implemented and none chosen: pack short sentences into ~250-char chunks read-aloud-side only (recommended — the splitter is shared with voice mode, which wants a fast short first segment); buffer two segments before starting (gapless, but start moves ~6 s → ~13 s); or use OpenAI TTS, which is far faster than realtime.
2. Markup sanitization is unit-tested only — though the <spoken-input> turn shown above exercises it incidentally against a live daemon.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

The PR adds read-aloud playback for the closing prose of completed agent turns, using the existing daemon TTS stack.

• Collects only assistant prose following the final tool call.
• Streams segmented TTS audio through the client and shared web audio engine.
• Adds ownership, cancellation, capability gating, and route-host lifecycle handling.
• Adds localized controls and coverage for turn collection, playback state, cancellation, server synthesis, and protocol behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/app/src/read-aloud/read-aloud-audio.web.ts	Adds web playback through the shared audio engine and closes the initialization-time cancellation window.
packages/app/src/read-aloud/read-aloud-store.ts	Implements the app-wide ownership, lifecycle, cancellation, and playback-completion state machine.
packages/app/src/read-aloud/use-read-aloud-route-guard.ts	Stops active playback when navigation leaves the host that owns the spoken turn.
packages/app/src/voice/audio-engine.web.ts	Adds generation-based cancellation so audio decoded after a stop cannot reach the output.
packages/server/src/server/session/voice/read-aloud-controller.ts	Sanitizes, segments, synthesizes, and streams read-aloud requests with cancellation and bounded prefetch.
packages/client/src/daemon-client.ts	Adds the segmented read-aloud request API and cancellable client handle.
packages/protocol/src/messages.ts	Defines the validated request, response, and cancellation messages for read-aloud streaming.

Sequence Diagram

sequenceDiagram
  participant U as User
  participant F as Turn footer
  participant S as Read-aloud store
  participant C as Daemon client
  participant D as Daemon TTS
  participant A as Audio engine
  U->>F: Press Read aloud
  F->>S: startReadAloud(turn, host, closing prose)
  S->>C: speech.tts.read_aloud.request
  C->>D: Request synthesis
  loop Segments
    D-->>C: Audio segment
    C-->>S: onSegment
    S->>A: Decode and queue playback
  end
  D-->>C: Stream complete
  C-->>S: onEnd
  S-->>F: Return owner to idle
  alt Stop, supersede, or leave host
    F->>S: stopReadAloud
    S->>C: Cancel synthesis
    S->>A: Clear queue and stop audio
  end

Loading

_{Reviews (15): Last reviewed commit: "fix(read-aloud): drop a segment cancelle..." | Re-trigger Greptile}