Skip to content

Commit 8ea86ec

Browse files
committed
feat(audio): cut over clients and backend to protocol v2
1 parent bad85ff commit 8ea86ec

162 files changed

Lines changed: 11671 additions & 11841 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ios-testflight.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ permissions:
55

66
on:
77
push:
8-
branches: [main]
8+
branches: [main, feature/audio-v2-cutover]
99
paths: ['app/**']
1010
workflow_dispatch:
1111
inputs:
@@ -51,9 +51,9 @@ jobs:
5151
run: |
5252
npm run test:wearable-activation
5353
npm run typecheck
54-
npm run test:voice-protocol
55-
npm run test:phone-duplex
54+
npm run test:durable-audio-spool
5655
npm run check:theme
56+
npx --no-install expo-modules-autolinking verify --platform ios --verbose
5757
5858
- name: Write and validate ASC API key
5959
env:

‎AGENTS.md‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,25 @@ Do not assume a remote deployment's checkout is clean or matches `origin` — ch
5050
inspect the live application. Remote deployment, restarts, or other mutations still
5151
require the user's request to change or deploy the running system.
5252

53+
### Verifying live historical data
54+
55+
Before claiming that captured data was not sent, stored, processed, or added to a
56+
timeline:
57+
58+
- Anchor the investigation on the exact user-provided URL or ID. Confirm the live
59+
database and inspect a real document's field names and BSON types; do not guess that
60+
a domain ID is MongoDB `_id` or that a date is stored as a string. Treat an empty
61+
query as inconclusive until the same query shape returns a known positive control.
62+
- Verify each layer separately: ingress/backfill records, canonical `audio_chunks`,
63+
`conversations`, timeline days/episodes, and the UI. Promotion between collections
64+
can leave an ingress collection empty even though canonical data exists.
65+
- For timelines, use the stored `local_date` type and user timezone, follow explicit
66+
conversation/audio references, check time overlap, and distinguish active/open
67+
episodes from provisional or superseded runs.
68+
- State exactly which claim the evidence supports: sent, stored, processed, linked, or
69+
merely visible in one UI route. Report UTC and user-local timestamps when time is
70+
relevant.
71+
5372
### Investigating a slow or unresponsive backend
5473

5574
When *everything* is slow at once — the dashboard hangs, health checks time out, a
@@ -280,7 +299,7 @@ voice evidence and human review assign a real person.
280299

281300
### Key Components
282301
- **Audio Pipeline**: Real-time Opus/PCM → Application-level processing → Deepgram transcription → memory extraction
283-
- **Wyoming Protocol**: WebSocket communication uses Wyoming protocol (JSONL + binary) for structured audio sessions
302+
- **Audio Protocol V2**: `/ws/audio` uses generated control messages and atomic raw-Opus media envelopes
284303
- **Unified Pipeline**: Job-based tracking system for all audio processing (WebSocket and file uploads)
285304
- **Job Tracker**: Tracks pipeline jobs with stage events (audio → transcription → memory) and completion status
286305
- **Task Management**: BackgroundTaskManager tracks all async tasks to prevent orphaned processes
@@ -310,8 +329,12 @@ Optional:
310329
311330
## Data Flow Architecture
312331
313-
1. **Audio Ingestion**: OMI devices stream audio via WebSocket using Wyoming protocol with JWT auth
314-
2. **Wyoming Protocol Session Management**: Clients send audio-start/audio-stop events for session boundaries
332+
For any capture, streaming, playback, wake-word, transcription, or audio-import
333+
change, read `docs/backend/audio-interface-map.md`. It is the boundary inventory and
334+
migration ledger for the generated Chronicle audio contract.
335+
336+
1. **Audio Ingestion**: Devices stream raw Opus through generated audio-v2 envelopes with bearer authentication
337+
2. **Typed Session Management**: Bound start/stop controls define capture sessions and reject stale sockets
315338
3. **Application-Level Processing**: Global queues and processors handle all audio/transcription/memory tasks
316339
4. **Speech-Driven Conversation Creation**: Continuous capture creates no user-facing row until speech is detected; deliberate recordings/uploads are visible immediately
317340
5. **Audio Evidence and Conversations**: Opus audio documents are persisted in `audio_chunks`; `captured_at` remains stable when chunks are split, merged, or trimmed. A conversation is the current semantic/operational claim over those documents, not their temporal identity.
@@ -460,7 +483,7 @@ SPEAKER_SERVICE_URL=http://speaker-recognition:8085
460483
### Common Endpoints
461484
- **GET /health**: Basic application health check
462485
- **GET /readiness**: Service dependency validation
463-
- **WS /ws**: Audio streaming endpoint with codec parameter (Wyoming protocol, supports pcm and opus codecs)
486+
- **WS /ws/audio**: Audio-v2 WebSocket (generated control JSON plus atomic raw-Opus media)
464487
- **GET /api/conversations**: User's conversations with transcripts
465488
- **GET /api/memories/search**: Agentic vault search (retrieval agent over the Markdown vault)
466489
- **POST /auth/jwt/login**: Email-based login (returns JWT token)

‎app/README.md‎

Lines changed: 16 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -108,26 +108,18 @@ Reference in `AndroidManifest.xml`:
108108

109109
## Backend Configuration
110110

111-
### Supported Backends
111+
### Supported Backend
112112

113-
The app connects to any backend that accepts OPUS audio streams:
114-
115-
1. **Simple Backend** (`backends/simple/`)
116-
- Basic audio capture and storage
117-
- Good for testing and development
118-
- WebSocket endpoint: `/ws`
119-
120-
2. **Advanced Backend** (`backends/advanced/`)
121-
- Full transcription and memory features
122-
- Real-time processing with speaker recognition
123-
- WebSocket endpoint: `/ws?codec=pcm`
113+
The app connects to Chronicle's advanced backend using audio protocol v2 at
114+
`/ws/audio` with the `chronicle.audio.v2` WebSocket subprotocol. Capture and
115+
playback media are atomic raw-Opus envelopes generated from the shared contract.
124116

125117
### Connection Setup
126118

127119
#### Local Development
128120
```
129-
Backend URL: ws://[machine-ip]:8000/ws?codec=pcm
130-
Example: ws://192.168.1.100:8000/ws?codec=pcm
121+
Backend URL: ws://[machine-ip]:8000/ws/audio
122+
Example: ws://192.168.1.100:8000/ws/audio
131123
```
132124

133125
#### Public Access (Production)
@@ -138,7 +130,7 @@ Use ngrok or similar tunneling service:
138130
ngrok http 8000
139131

140132
# Use provided URL in app
141-
Backend URL: wss://[ngrok-subdomain].ngrok.io/ws?codec=pcm
133+
Backend URL: wss://[ngrok-subdomain].ngrok.io/ws/audio
142134
```
143135

144136
### Configuration Steps
@@ -147,14 +139,14 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws?codec=pcm
147139
2. **Open the mobile app**
148140
3. **Navigate to Settings**
149141
4. **Enter Backend URL**:
150-
- Local: `ws://[your-ip]:8000/ws?codec=pcm`
151-
- Public: `wss://[your-domain]/ws?codec=pcm`
142+
- Local: `ws://[your-ip]:8000/ws/audio`
143+
- Public: `wss://[your-domain]/ws/audio`
152144
5. **Save configuration**
153145

154146
## Duplex Phone Voice
155147

156148
### Overview
157-
The phone is a protocol-v1 Chronicle voice endpoint. One native engine owns microphone
149+
The phone is an audio-v2 Chronicle voice endpoint. One native engine owns microphone
158150
capture and Chronicle response playback while the session is active. The backend binds
159151
every response to the authenticated socket, audio session, voice session, capture
160152
epoch, and response generation.
@@ -163,7 +155,7 @@ epoch, and response generation.
163155
- **Barge-in**: Verified speakerphone AEC and isolated headphone routes capture while Chronicle speaks
164156
- **Safe fallback**: Unsupported routes report native half duplex instead of claiming simultaneous capture
165157
- **Native cancellation**: Socket loss, route changes, interruptions, and engine resets flush playback immediately
166-
- **Session fencing**: Reconnects create new audio and voice sessions and require short-lived resume proof
158+
- **Session fencing**: Reconnects create an exact physical-connection lease and a new capture binding
167159
- **Cross-platform replacement**: One `chronicle-duplex-audio` interface backs iOS and Android
168160

169161
### Setup & Usage
@@ -179,7 +171,7 @@ epoch, and response generation.
179171
- **iOS**: A native development or release build with microphone permission
180172
- **Android**: Android API 31+ native build with microphone permission
181173
- **Network**: Stable connection to Chronicle backend
182-
- **Backend**: Advanced backend with voice duplex protocol v1 at `/ws?codec=pcm`
174+
- **Backend**: Advanced backend with audio protocol v2 at `/ws/audio`
183175

184176
#### Switching Audio Sources
185177
- **Mutual Exclusion**: Cannot use Bluetooth and phone audio simultaneously
@@ -202,7 +194,7 @@ JavaScript layer does not create a second player or gate microphone buffers.
202194

203195
#### Audio Not Streaming
204196
- **Check Permissions**: Ensure microphone access granted
205-
- **Verify Backend URL**: Confirm `ws://[ip]:8000/ws?codec=pcm` format
197+
- **Verify Backend URL**: Confirm `ws://[ip]:8000/ws/audio` format
206198
- **Network Connection**: Test backend connectivity
207199
- **Authentication**: Verify JWT token is valid
208200

@@ -307,7 +299,7 @@ curl -i -N -H "Connection: Upgrade" \
307299
-H "Upgrade: websocket" \
308300
-H "Sec-WebSocket-Key: test" \
309301
-H "Sec-WebSocket-Version: 13" \
310-
http://[backend-ip]:8000/ws?codec=pcm
302+
http://[backend-ip]:8000/ws/audio
311303
```
312304

313305
## Development
@@ -353,10 +345,9 @@ npx expo build:android
353345
### WebSocket Communication
354346
```javascript
355347
// Connect to backend
356-
const ws = new WebSocket('ws://backend-url:8000/ws?codec=pcm');
348+
const ws = new WebSocket('ws://backend-url:8000/ws/audio', 'chronicle.audio.v2');
357349

358-
// Send audio data
359-
ws.send(audioBuffer);
350+
// Send generated ClientControl JSON and atomic binary MediaEnvelope packets.
360351

361352
// Handle responses
362353
ws.onmessage = (event) => {

‎app/app/theme-preview.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,7 @@ function ThemeShowcase({ name }: { name: ThemeName }) {
177177

178178
<SectionLabel>Form fields</SectionLabel>
179179
<Card>
180-
<TextField label="Backend URL" value="wss://kraken.parrot-census.ts.net/ws" hint="From the QR on the dashboard." />
180+
<TextField label="Backend URL" value="wss://kraken.parrot-census.ts.net/ws/audio" hint="From the QR on the dashboard." />
181181
<TextField label="Email" placeholder="user@example.com" />
182182
<TextField label="Password" value="hunter2" secureTextEntry error="Incorrect password." />
183183
<TextField label="Device name" value="pendant" editable={false} hint="Disabled while streaming." />

0 commit comments

Comments
 (0)