Document audio mode for debugging LiveKit agents

README.md

@@ -19,7 +19,7 @@ name, so a task typically loads two or three of them instead of one large docume
 |---|---|---|
 | Know | **reading-livekit-docs** | Looks up current LiveKit facts (APIs, CLI flags, changelogs, pricing) through the Docs MCP server or `lk docs` instead of answering from memory. The other skills load it first. |
 | Build | **building-livekit-agents** | Architecture for low-latency voice agents: keeping context small, splitting monoliths into handoffs and tasks, designing for users who listen rather than read. |
-| Try | **debugging-livekit-agents** | Drives a multi-turn conversation with the agent running locally in text mode, via `lk agent debugger`, and reads the tool calls behind each reply. The default way to check a change. |
+| Try | **debugging-livekit-agents** | Drives a multi-turn conversation with the agent running locally, in text or audio mode, via `lk agent debugger`, and reads the tool calls behind each reply. The default way to check a change. |
 | Test | **testing-livekit-agents** | Turn-level tests in the project's own pytest or Vitest suite: assertions on messages, tool calls and handoffs, LLM judging of intent, tool mocking. |
 | Simulate | **writing-livekit-scenarios** | Creates the scenarios a simulation runs (generate a baseline, refine it, cover the hard cases, organize into sets) and wires the agent to seed state from them and grade its final state. |
 | Ship | **running-livekit-simulations** | Runs simulations in text or audio mode, against a local or deployed agent, in CI before a release, and triages the failures. |

evals/trigger/queries.json

@@ -79,6 +79,18 @@
    "debugging-livekit-agents"
   ]
  },
+ {
+  "q": "callers say their order number and the agent keeps getting it wrong. can you try saying a few order numbers to it out loud and see what it actually hears",
+  "expect": [
+   "debugging-livekit-agents"
+  ]
+ },
+ {
+  "q": "we switched to a speech-to-speech realtime model and now i can't text chat with it anymore. how do i try a couple of turns locally after each edit",
+  "expect": [
+   "debugging-livekit-agents"
+  ]
+ },
  {
   "q": "write {test_runner} tests for {agent} covering the greeting and that it refuses to give medical dosages",
   "expect": [

skills/building-livekit-agents/SKILL.md

@@ -125,7 +125,7 @@ Prompt changes break agent behavior as easily as code changes do, and trying it
 doesn't count as verification.
 
 - **While building**, drive conversations with `debugging-livekit-agents`. It runs your agent
-  locally in text mode, lets you send turns, and shows the tool calls behind each reply.
+  locally, lets you send turns as text or speech, and shows the tool calls behind each reply.
 - **Before you call it done**, write tests with `testing-livekit-agents`. At minimum, cover the core
   behavior the user asked for, tool invocation with correct arguments if there are tools, and one
   failure path.

skills/debugging-livekit-agents/SKILL.md

@@ -1,19 +1,20 @@
 ---
 name: debugging-livekit-agents
-description: 'Drives a multi-turn conversation with a LiveKit agent running locally to see what it does. Use when the user says "test my agent", "try my agent", "does this work", "why did it call that tool", "it says the wrong thing when I ask X", "test this change", or whenever you have edited an agent and need to check how it behaves. Wraps `lk agent debugger`: start the agent in text mode, send turns, read the tool calls, handoffs, errors and logs behind each reply, and restart after an edit. It runs without audio or a LiveKit room at one LLM call per turn, so it is the preferred way for a coding agent to live-test during development, and the default when the user says "test" without naming unit tests or simulations.'
+description: 'Drives a live multi-turn conversation with a LiveKit agent running locally, using `lk agent debugger`. Use when the user says "test my agent", "try my agent", "does this work", or "why did it call that tool", after editing an agent to check how it behaves, or when an agent on a speech-to-speech model or one that mishears users needs checking. The default for a bare "test"; for regression tests use testing-livekit-agents, for graded runs running-livekit-simulations.'
 license: MIT
 metadata:
   author: livekit
 ---
 
 # Debugging a LiveKit agent live
 
-`lk agent debugger` runs the user's agent locally as a background process in text mode and lets
-you drive a conversation one turn at a time. It's built for coding agents: you play the user,
-choose each next line based on the last reply, and inspect what the agent did in between.
+`lk agent debugger` runs the user's agent locally as a background process and lets you drive a
+conversation one turn at a time. It's built for coding agents: you play the user, choose each next
+line based on the last reply, and inspect what the agent did in between.
 
-Speech is off and nothing goes to a LiveKit room, so a turn costs only the agent's own LLM and tool
-calls. That's cheap enough to use constantly while building.
+By default the session is text: speech is off and nothing goes to a LiveKit room, so each turn is
+fast. When the bug lives in the audio path, start the session in audio mode instead (see
+[Audio mode](#audio-mode)).
 
 Before the first use, confirm the command exists and read its help:
 
@@ -63,14 +64,42 @@ time debugging behavior the file no longer has.
 - **Script it if a program is deciding the turns.** There's machine-readable output and meaningful
   exit codes. Check `--help` for the current format instead of assuming field names.
 
+## Audio mode
+
+Text mode skips STT and TTS, so it can't show a name heard as a different spelling, digits the STT
+spells out as words, a turn that ends too early, or anything from a realtime model that only takes
+audio. Where the CLI supports it, starting the session with audio (check `start --help`) speaks
+each `say` with LiveKit Inference TTS into the agent's microphone input, and the agent runs its full
+audio pipeline.
+
+- **Use it when the symptom is about hearing**, or when the agent uses a speech-to-speech model that
+  can't run in text mode. Stay in text mode for logic, prompt, and tool bugs: it's faster, and
+  the transcription noise only gets in the way.
+- **It needs a LiveKit Cloud project.** The spoken turns run on LiveKit Inference, so `lk` must
+  have project credentials. Text mode doesn't need them.
+- **Compare what was heard with what you sent.** Each turn shows the agent's transcript of your
+  line, and the sent text when the words differ. That difference is often the whole bug: the agent
+  answered correctly to what it heard.
+- **Write lines the way a caller would say them.** Spell out numbers and codes when testing how the
+  agent handles spoken input, and test both forms if the agent should accept either.
+- **The mode belongs to the session.** Restart keeps it; stop and start again to switch.
+- **A long silent pause can end a turn early.** A turn ends once the agent has answered and stayed
+  quiet for a moment. If the agent pauses for a long time before finishing (for example, a realtime
+  model waiting on a delegated answer), the rest of its reply shows up in the next turn. Check the
+  chat history before concluding the agent never answered.
+
+Audio mode still can't interrupt the agent mid-reply or tell you how the agent sounds. Use
+`lk agent console` or audio simulations for those.
+
 ## When to use something else
 
 | Situation | Use |
 |---|---|
 | You want to *hear* it, or hand the user something to try | `lk agent console` (mic and speakers, a human at the keyboard) |
 | The same failure keeps coming back | A turn-level regression test (`testing-livekit-agents`) |
 | You need whole-conversation outcomes graded at scale | Simulations (`running-livekit-simulations`) |
-| The bug is about speech: turn-taking, interruptions, transcription | Audio simulations. The debugger is text-only and can't see these |
+| Transcription or endpointing on a turn you're investigating | The debugger in audio mode (above) |
+| Interruptions, or speech behavior graded across many conversations | Audio simulations (`running-livekit-simulations`) |
 
 The debugger is for finding bugs interactively. Once you've found one, write a test for it so a
 later change can't reintroduce it unnoticed.