Read aloud: speak an agent turn from its footer

docs/refactors/read-aloud-footer-button-plan.md

@@ -0,0 +1,123 @@
+# Read Aloud — pivot from selection bubble to turn footer
+
+## Why
+
+The project owner:
+
+> "i would accept the button in the footer as discussed, just speaking the message
+> after the last tool call should be enough"
+
+The shipped approach anchors a floating speaker button to a live text selection. That
+was web/Electron only — React Native exposes no JS API for the current selection — and
+it carried two open Greptile P1s about selected text crossing daemon boundaries.
+
+A footer button drops all of it: no anchoring, no placement, no selection to leak.
+
+## Scope
+
+Replace, don't add. The selection bubble goes.
+
+| File                                  | Lines | Fate                                             |
+| ------------------------------------- | ----: | ------------------------------------------------ |
+| `use-selection-anchor.web.ts`         |   410 | delete                                           |
+| `read-aloud-selection-bubble.web.tsx` |   398 | delete                                           |
+| `read-aloud-placement.ts`             |   166 | delete                                           |
+| `read-aloud-placement.test.ts`        |   152 | delete                                           |
+| `read-aloud-selection-bubble.tsx`     |    12 | delete (native `return null` stub)               |
+| `read-aloud-store.ts`                 |   185 | keep, one change (below)                         |
+| `read-aloud-audio.web.ts` / `.ts`     |   100 | keep as-is                                       |
+| protocol / server / client            |     — | keep as-is — the daemon contract does not change |
+
+~1,138 lines deleted. Both Greptile P1s live entirely in deleted files, so they are
+resolved by removal rather than by a fix.
+
+## What ships
+
+A speaker button in `AssistantTurnFooter`, next to the copy button
+(`packages/app/src/components/message.tsx:663`). Press to hear the turn's closing
+message; press again to stop.
+
+### 1. The text: stop at the last tool call
+
+`collectAssistantTurnContent` (`agent-stream/strategy.ts:162`) walks the turn backward
+and breaks only on `user_message`. It steps _over_ `tool_call` items, so it returns
+every prose block in the turn — including narration before the first tool.
+
+Add a sibling `collectAssistantTurnSpeech` that also breaks on `tool_call`. It belongs
+in the strategy, not at the call site: traversal direction is
+`config.assistantTurnTraversalStep` (±1), because native renders an inverted list.
+
+This is the only genuinely new logic in the pivot.
+
+Markdown is **not** a problem: `sanitizeTextForReadAloud`
+(`server/speech/read-aloud-text.ts`) already strips fences, HTML-like tags, link URLs,
+and inline markers server-side. Every text path gets cleaned, not just selections.
+
+### 2. The store: name the speaker
+
+`read-aloud-store.ts` is a module-level singleton — one read at a time app-wide. That
+was fine for a single bubble. With a button per turn, every footer subscribes to the
+same snapshot and they would all render "speaking" at once.
+
+Add an owner to the snapshot: `startReadAloud({ client, text, ownerId })`, and
+`ReadAloudSnapshot.ownerId: string | null`. A footer shows the stop state only when
+`snapshot.ownerId === thisTurnId`. Starting a second turn's read supersedes the first,
+which the existing `generation` counter already handles.
+
+Use the assistant item id as `ownerId`.
+
+### 3. Host binding
+
+Keep the invariant the selection version landed: speech goes to the route's host, never
+another paired daemon. `useReadAloudServerId` is currently private to the bubble file —
+lift it into `read-aloud/use-read-aloud-host.ts` and delete the bubble.
+
+`resolvedServerId` already exists in `agent-stream/view.tsx:364` but is **not** passed to
+`TurnFooter`. Either thread it through
+`TurnFooter → CompletedTurnFooterRow → CompletedTurnFooter → AssistantTurnFooter`, or
+keep deriving it from the route. Prefer threading — it is explicit, and the footer
+already takes a `host` prop (note: that prop is a _layout_ grouping, `TurnFooterHost`,
+not a server; do not overload it).
+
+### 4. Platform gating
+
+Native audio is a stub: `read-aloud-audio.ts` exports `isReadAloudAudioSupported = false`
+and `playReadAloudSegment` throws. The button would render on iOS/Android and produce
+silence.
+
+Gate the button on `isReadAloudAudioSupported && hostSupportsReadAloud`. Platform scope
+stays web + Electron, unchanged from today. Native audio is a follow-up — and now a
+reachable one, since nothing about the footer button is web-specific.
+
+The footer is already assistant-only and completed-turn-only (`layout.ts:120` returns
+null while running), and it is not hover-gated, so the button is simply always visible.
+
+## Tests
+
+- `collectAssistantTurnSpeech` — text after the last tool call; a turn with no tool calls;
+  a turn ending in a tool call with no trailing prose (expect empty → button hidden);
+  both traversal directions.
+- `read-aloud-store` — `ownerId` set on start, cleared on stop, superseded on a second
+  start. Extend the existing `read-aloud-store.test.ts`.
+- Delete `read-aloud-placement.test.ts` with its subject.
+
+## Verification
+
+Typecheck, lint, format, the touched test files, then the real app: press the button,
+confirm audio and the idle → speaking → idle transitions.
+
+Assert on the artifact, not the label. The last session's mistake was reading a state
+machine going idle → stop → idle as success when a swallowed error was driving it and no
+audio ever reached the output. Instrument `AudioContext` and check buffer peaks.
+
+## Evidence
+
+Playwright stills of idle / loading / speaking / stopped, plus a screen recording
+converted to GIF, published to the PR. A silent GIF cannot show the feature working —
+pair it with the measured buffer peaks.
+
+## Delivery
+
+New commits on `text-selection-speech-modal`, PR
+[#2675](https://github.com/getpaseo/paseo/pull/2675). The PR description needs a rewrite,
+not an edit: it currently documents the selection feature end to end.

docs/refactors/read-aloud-handoff.md

@@ -0,0 +1,142 @@
+# Read Aloud — Session Handoff
+
+Picking this up cold? Read this, then
+[read-aloud-footer-button-plan.md](./read-aloud-footer-button-plan.md), which records why
+the feature moved off text selection.
+
+- **Branch:** `text-selection-speech-modal`, branched from `d1ce2b77f`. The branch name
+  predates the pivot; the feature is no longer selection-based.
+
+## What the feature is
+
+Every completed agent turn has a speaker button in its footer, next to copy. Press it and
+the daemon speaks the turn's **closing message** — the prose after the last tool call, not
+the narration in between. Press again to stop. Starting a read on another turn supersedes
+the first: playback is one app-wide slot.
+
+It started as a floating button anchored to a live text selection. The project owner asked
+for the footer instead ("just speaking the message after the last tool call should be
+enough"), which deleted ~1,138 lines of anchoring, placement, and bubble code and resolved
+two Greptile P1s about selected text crossing daemon boundaries.
+
+**Web and Electron only, for now** — but for a different reason than before. Selection was
+web-only because RN exposes no selection API; the footer button has no such limit. What
+blocks native now is only that `read-aloud-audio.ts` is a playback stub
+(`isReadAloudAudioSupported = false`). Implementing it with expo-audio makes the button
+work on iOS/Android with no other change. That is the obvious next task.
+
+Uses the **existing voice-mode TTS provider** (local Kokoro by default, or OpenAI). No
+ElevenLabs — the existing provider already has config, env vars, and docs plumbing.
+
+### Architecture
+
+| Piece                                           | What it does                                                                           |
+| ----------------------------------------------- | -------------------------------------------------------------------------------------- |
+| `protocol/src/messages.ts`                      | `speech.tts.read_aloud.request` → N `.response` segments; `…cancel_read_aloud.request` |
+| `server/session/voice/read-aloud-controller.ts` | Sanitize → split → synthesize with 2-segment prefetch → stream                         |
+| `server/speech/read-aloud-text.ts`              | Strips markdown and Paseo wrapper tags before synthesis                                |
+| `server/speech/tts-text-splitter.ts`            | Extracted from `tts-manager.ts`; **shared with voice mode**                            |
+| `client/src/daemon-client.ts`                   | `startReadAloud()` returns a handle with `cancel()`                                    |
+| `app/src/agent-stream/strategy.ts`              | `collectAssistantTurnSpeech` — walks back, stops at the last `tool_call`               |
+| `app/src/read-aloud/turn-read-aloud-button.tsx` | The footer button; owns gating and stop intent                                         |
+| `app/src/read-aloud/read-aloud-store.ts`        | Playback state machine, `ownerId`, generation counter, speed                           |
+| `app/src/read-aloud/use-read-aloud-host.ts`     | Route-host binding — speech never crosses to another paired daemon                     |
+| `app/src/read-aloud/read-aloud-audio.web.ts`    | Reuses the voice-mode `AudioEngine` (playback context only, no mic)                    |
+
+Segmented streaming is not gold-plating: local TTS returns raw 24 kHz PCM (~48 KB/s), so a
+long message in one frame would be megabytes.
+
+**Capability-gated** on `server_info.features.readAloud` (`COMPAT(readAloud)`, v0.2.5).
+Hosts without it get no button. No fallback path.
+
+## Environment gotchas — every one of these cost real time to rediscover
+
+**1. Node 22+ is required for the dev daemon.** In dev the daemon runs from TS source and
+forks the local speech worker with `--experimental-strip-types`, a Node 22+ flag, using
+`process.execPath`. On Node 20 the worker dies instantly (`exit code 9`) and every
+synthesis fails. nvm default here is v20.19.6; v22.20.0 and v23.3.0 are installed.
+
+```bash
+nvm use 22
+```
+
+This is pre-existing and not read-aloud specific — voice mode and dictation break the same
+way.
+
+**2. The daemon does NOT hot-reload.** `packages/server/scripts/dev-runner.ts` has no
+watcher. The Expo app hot-reloads; the daemon does not. **Restart `npm run dev` after any
+server change** or you will test stale code and draw wrong conclusions.
+
+**2b. `nvm use 22` does not work from a non-interactive shell here.** The zsh profile
+installs an nvm lazy-load shim, and outside an interactive shell the shim recurses until zsh
+gives up with `maximum nested function level reached` — `node`, `npm`, and `npx` all fail,
+and prepending to `PATH` does not help because the shell _function_ shadows the binary. Call
+the binary by absolute path instead:
+
+```bash
+~/.nvm/versions/node/v22.20.0/bin/node ~/.nvm/versions/node/v22.20.0/bin/npm run typecheck
+```
+
+**3. `Buffer` is not a browser global.** Every consumer imports it explicitly
+(`voice/voice-runtime.ts:1`). `@types/node` makes a bare `Buffer` reference typecheck while
+failing at runtime in the bundle. This already caused one silent no-audio bug.
+
+## Running it
+
+```bash
+nvm use 22
+npm run dev        # terminal 1 — daemon on 127.0.0.1:6768
+npm run dev:app    # terminal 2 — Expo web on http://localhost:8081
+```
+
+No extra env needed: `dev-app.sh` derives the daemon endpoint from `PASEO_LISTEN`, and
+`dev-daemon.sh` defaults `PASEO_LOCAL_MODELS_DIR` to `~/.paseo/models/local-speech`, which
+already has Kokoro — so no ~1 GB model download.
+
+Runs alongside the user's normal Paseo on **6767** — different port, different `PASEO_HOME`
+(`.dev/paseo-home`). **Never restart the 6767 daemon**; it manages live agents. The dev
+home already has a project and workspace registered from earlier testing.
+
+To try it: open a workspace with an agent that has finished at least one turn, and press
+the speaker button in that turn's footer, next to the copy button.
+
+### Probing the daemon directly
+
+Faster than the UI for server-side questions. Connect to `ws://localhost:6768/ws`, send a
+`hello` frame **first** (session messages before hello are rejected), then wrap requests as
+`{type: "session", message: {…}}`. Run the script from the repo root so `ws` resolves.
+
+## Open questions for the next session
+
+- **Segment gaps.** Local Kokoro runs ~1× realtime with a ~6 s cold start. The splitter
+  emits one segment per sentence, so a short first sentence ("Are you working?" → 1.79 s)
+  drains before the next finishes synthesizing — measured **6.1 s of silence** mid-read.
+  Three options, none implemented, user has not chosen:
+  1. Pack short sentences into ~250-char chunks, **read-aloud only** — do not touch the
+     shared splitter, voice mode wants the fast short first segment. Recommended.
+  2. Buffer two segments before starting. Gapless, but start moves ~6 s → ~13 s. Worse.
+  3. Switch to OpenAI TTS — far faster than realtime, gaps vanish, needs an API key.
+- **Error detail is dropped.** Unknown failure codes render the generic "Couldn't read that
+  aloud"; the daemon's real message sits unused in `failure.message`. Worth surfacing on
+  hover so the next failure is self-diagnosing.
+- **Markup sanitization is unit-tested only.** It has not been exercised against a live
+  daemon. Confirm a turn containing a `<spoken-input>` block or a fenced code block speaks
+  only the prose.
+- **Native audio is unimplemented.** `read-aloud-audio.ts` is a stub, so the button hides
+  on iOS/Android. Nothing else blocks native now that selection is gone.
+
+## Things I got wrong — don't repeat them
+
+- **Claimed "verified end-to-end" when I had only verified the state machine.** The UI went
+  idle → Stop → idle exactly as it would on success, but that transition was driven by a
+  swallowed error, and no audio sample ever reached the output. Two bugs hid in that gap.
+  For anything audio- or layout-related, assert on the real artifact — buffer peaks,
+  `getBoundingClientRect()` — not on labels.
+- **Forced `/opt/homebrew/bin/node` (v24) onto PATH** so my runs worked, which masked the
+  Node 20 worker crash the user hit immediately. Verify in the environment the user
+  actually runs, not one bent to work.
+- **Swallowed errors in a `.catch(() => {})`.** Turned a hard `ReferenceError` into "no
+  error, no sound", the worst possible failure mode. That catch now reports.
+- **Built the hard version first.** Selection anchoring — endpoint rects, clipping
+  ancestors, placement clamping — was ~1,138 lines that the owner replaced with a button in
+  a footer. The expensive part was never the speech; it was the anchoring nobody asked for.

packages/app/src/agent-stream/render-strategy.test.ts

@@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest";
 import type { StreamItem } from "@/types/stream";
 import {
   collectAssistantTurnContentForStreamRenderStrategy,
+  collectAssistantTurnSpeechForStreamRenderStrategy,
   getBottomOffsetForStreamRenderStrategy,
   getFrameChildOrderForStreamRenderStrategy,
   getHistoryLiveBoundaryIndexForStreamRenderStrategy,
@@ -38,6 +39,24 @@ function assistantMessage(id: string, text: string, seed: number): StreamItem {
   };
 }
 
+function toolCall(id: string, seed: number): StreamItem {
+  return {
+    kind: "tool_call",
+    id,
+    timestamp: createTimestamp(seed),
+    payload: {
+      source: "orchestrator",
+      data: {
+        toolCallId: id,
+        toolName: "bash",
+        arguments: "cmd",
+        result: null,
+        status: "completed",
+      },
+    },
+  };
+}
+
 describe("resolveStreamRenderStrategy", () => {
   it("uses forward_stream on web", () => {
     const strategy = resolveStreamRenderStrategy({
@@ -196,6 +215,99 @@ describe("neighbor and traversal semantics", () => {
     ).toBe("assistant-1\n\nassistant-2");
   });
 
+  it("speaks only the prose after the last tool call", () => {
+    const chronological: StreamItem[] = [
+      userMessage("u1", "user-1", 1),
+      assistantMessage("a1", "before the tool", 2),
+      toolCall("t1", 3),
+      assistantMessage("a2", "after the tool", 4),
+    ];
+
+    const forward = resolveStreamRenderStrategy({ platform: "web", isMobileBreakpoint: false });
+    const startIndex = chronological.findIndex((item) => item.id === "a2");
+
+    // Copy takes the whole turn; speech stops at the tool call. Reading a long
+    // turn from the top would replay narration the user already watched.
+    expect(
+      collectAssistantTurnContentForStreamRenderStrategy({
+        strategy: forward,
+        items: chronological,
+        startIndex,
+      }),
+    ).toBe("before the tool\n\nafter the tool");
+    expect(
+      collectAssistantTurnSpeechForStreamRenderStrategy({
+        strategy: forward,
+        items: chronological,
+        startIndex,
+      }),
+    ).toBe("after the tool");
+  });
+
+  it("speaks the whole turn when it has no tool calls", () => {
+    const chronological: StreamItem[] = [
+      userMessage("u1", "user-1", 1),
+      assistantMessage("a1", "assistant-1", 2),
+      assistantMessage("a2", "assistant-2", 3),
+    ];
+
+    const forward = resolveStreamRenderStrategy({ platform: "web", isMobileBreakpoint: false });
+
+    expect(
+      collectAssistantTurnSpeechForStreamRenderStrategy({
+        strategy: forward,
+        items: chronological,
+        startIndex: chronological.findIndex((item) => item.id === "a2"),
+      }),
+    ).toBe("assistant-1\n\nassistant-2");
+  });
+
+  it("speaks nothing when the turn ends on a tool call", () => {
+    const chronological: StreamItem[] = [
+      userMessage("u1", "user-1", 1),
+      assistantMessage("a1", "before the tool", 2),
+      toolCall("t1", 3),
+    ];
+
+    const forward = resolveStreamRenderStrategy({ platform: "web", isMobileBreakpoint: false });
+
+    // The button hides on an empty result rather than synthesizing silence.
+    expect(
+      collectAssistantTurnSpeechForStreamRenderStrategy({
+        strategy: forward,
+        items: chronological,
+        startIndex: chronological.findIndex((item) => item.id === "t1"),
+      }),
+    ).toBe("");
+  });
+
+  it("stops at the last tool call in both traversal directions", () => {
+    const chronological: StreamItem[] = [
+      userMessage("u1", "user-1", 1),
+      assistantMessage("a1", "before the tool", 2),
+      toolCall("t1", 3),
+      assistantMessage("a2", "after the tool", 4),
+    ];
+
+    const inverted = resolveStreamRenderStrategy({
+      platform: "android",
+      isMobileBreakpoint: false,
+    });
+    const invertedItems = orderTailForStreamRenderStrategy({
+      strategy: inverted,
+      streamItems: chronological,
+    });
+
+    // Native renders an inverted list, so the walk runs the other way. Same text.
+    expect(
+      collectAssistantTurnSpeechForStreamRenderStrategy({
+        strategy: inverted,
+        items: invertedItems,
+        startIndex: invertedItems.findIndex((item) => item.id === "a2"),
+      }),
+    ).toBe("after the tool");
+  });
+
   it("returns undefined neighbor when index would be out of bounds", () => {
     const forward = resolveStreamRenderStrategy({
       platform: "web",