Support 16 kHz voice and repair recurring CI failures

[dimavrem22]

Executive Summary

Enable 16 kHz call audio and repair recurring CI installation and live-test failures.

• Negotiate wideband PCM with streaming conversion in both directions.
• Keep legacy call streams compatible and discard interrupted response audio.
• Use the published SDK in CI and remove a progress-test timing race.

Description

The call WebSocket advertises X-Inkbox-Audio-Format: pcm_s16le_16000. The bridge waits for the start frame before speaking, selects the delivered format, and converts mono PCM between 16 kHz call media and the realtime connection's required 24 kHz format. Streaming resampling retains chunk boundaries, filters downsampling aliases, and flushes each response independently. Legacy 8 kHz call descriptors remain supported; unsupported formats terminate clearly.

CI now installs the pinned published SDK instead of a source link that fails during dependency resolution. Live discovery reads retry connection establishment failures without repeating submissions. The progress scenario stays active long enough to observe two complete periodic updates, and the auxiliary summary has a bounded 20-second response window. The hosted voice test uses the original single caller request, with no reminder or re-ask; action persistence and single-delivery assertions remain unchanged. Post-call execution and correction prompts explicitly preserve caller-specified message bodies verbatim instead of substituting acknowledgements or summaries. Hosted SMS guards now remain active through host retries and intermediate tool steps; a completed step timestamp is not mistaken for final turn completion.

Reason

Raw call audio should preserve wideband speech, and CI should exercise plugin behavior instead of failing in obsolete setup paths or racing task completion against progress delivery.

Decisions

• Audio boundary: Keep 16 kHz call media and resample to the realtime connection's supported 24 kHz PCM instead of relabeling samples.
• Compatibility: Select the format from the start frame before the greeting so existing 8 kHz callers remain usable.
• CI dependencies: Install the already-published pinned SDK, preserving the supported-version and latest-host lanes.
• Turn lifecycle: Require an idle host and terminal finish reason before releasing the hosted send guard; preserve exactly-once enforcement during retries and tool continuations.
• Unassisted follow-up: Require the original caller request to persist an action and trigger post-call SMS delivery without caller reminders.

Testing

• npm test: 891 tests passed, including wideband frequency preservation, anti-alias filtering, fragmented samples, response resets, protocol headers, and call-start ordering.

• npm run lint, npm run typecheck, and npm run build: passed.

• python3 -m unittest tests/live/test_a2a_preflight.py: four tests passed, including bounded connection retry and non-retryable errors.

• Incoming and outgoing realtime calls: expect natural-speed two-way audio and prompt interruption; the outgoing realtime live check also requires a 16 kHz negotiation event correlated to that call. Passed on 6a11507, including the exact-call 16 kHz negotiation assertion and the unchanged two-way speech checks.

• A2A long-running task: expect a prompt receipt, two nonterminal periodic updates, and the correct final result; all five A2A scenarios passed on 6a11507.

• Unassisted full-stack validation: all lanes passed on 6a11507 (mock/real channels, five A2A scenarios, three voice scenarios, external events). The hosted scenario passed in 76 seconds using the original single caller request: one persisted action before hangup, completed post-call settlement, and exactly one accepted SMS containing the requested marker to the current caller. No caller reminders or re-asks. This supersedes the earlier reminder-assisted hosted result; full SMS-body byte equality is not asserted.

• Focused post-call completion, durable SMS guard, send boundary, and host-session checks: 83 passed on 6a11507.

• Guard lifecycle regressions: retry, intermediate tool/unknown finishes, and a timestamp without a terminal finish cannot release the hosted send guard; the original three cases fail against the preceding implementation.

[greptile-apps]

Greptile Summary

The PR adds negotiated wideband call audio, streaming PCM conversion, stricter hosted-turn completion handling, and CI/live-test reliability improvements.

• Negotiates 16 kHz PCM call media while retaining legacy 8 kHz PCMU support.
• Converts call audio to and from the realtime service’s 24 kHz PCM format.
• Keeps hosted SMS guards active through retries and intermediate tool steps.
• Replaces source-linked SDK installation in CI with the pinned published package.
• Adds audio, lifecycle, retry, and live-test coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/gateway/voice/audio.ts	Introduces negotiated call-format selection and bidirectional conversion between call media and realtime PCM.
src/gateway/voice/pcm-resampler.ts	Adds a stateful windowed-sinc PCM16LE resampler with fragmented-sample handling and response-scoped reset behavior.
src/gateway/voice/bridge.ts	Defers realtime greeting startup until the call start frame selects the delivered audio format.
src/gateway/voice/realtime.ts	Switches realtime audio to 24 kHz PCM and integrates conversion, response flushing, and interruption resets.
src/gateway/sessions.ts	Retains hosted side-effect guards until the host is idle and the assistant reports a terminal finish.
.github/workflows/tests.yml	Replaces the source SDK checkout and build with installation of the pinned published SDK.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant Bridge as Call Bridge
    participant Audio as CallAudio
    participant RT as Realtime API
    Caller->>Bridge: start(media_format)
    Bridge->>Audio: select 16 kHz PCM or 8 kHz PCMU
    Bridge->>RT: start greeting
    Caller->>Bridge: negotiated audio frames
    Bridge->>Audio: decode and resample to 24 kHz PCM
    Audio->>RT: input_audio_buffer.append
    RT->>Audio: 24 kHz output audio deltas
    Audio->>Bridge: resample and encode call audio
    Bridge->>Caller: outbound media
    RT->>Audio: audio done
    Audio->>Bridge: flush response tail
    Bridge->>Caller: audio done

Loading

_{Reviews (5): Last reviewed commit: "test: restore unassisted hosted SMS scen..." | Re-trigger Greptile}