repeatAudio() does not produce lip-sync or audio for custom VIDEO avatars in LITE mode

[Yuvraj3905]

Description

session.repeatAudio(pcmString) in LITE mode sends audio chunks correctly via WebSocket but produces no visible lip-sync and no audio output for custom VIDEO avatars. The server fires rapid AVATAR_SPEAK_STARTED / AVATAR_SPEAK_ENDED events per chunk (~20ms each) instead of treating them as a single utterance.

Environment

• SDK version: @heygen/liveavatar-web-sdk@0.0.12
• Avatar type: Custom VIDEO avatar (created from uploaded video)
• Session mode: LITE
• Browser: Chrome 130+ (also tested Firefox)

Steps to Reproduce

1. Create a session token with mode: "LITE" and a custom VIDEO avatar
2. Connect and wait for SESSION_STREAM_READY
3. Generate PCM audio (24kHz, 16-bit mono, binary string) — tested with both OfflineAudioContext resampling and manual linear interpolation from a 22050Hz WAV source
4. Call session.repeatAudio(pcmString)

Expected Behavior

• Avatar lip-syncs to the provided audio
• Audio plays through the LiveKit audio track
• Single AVATAR_SPEAK_STARTED at the beginning, single AVATAR_SPEAK_ENDED at the end

Actual Behavior

• No lip movement on the avatar
• No audio output through the stream
• Rapid cycling of AVATAR_SPEAK_STARTED → AVATAR_SPEAK_ENDED events (one pair per ~20ms chunk, hundreds of pairs for a 3-second audio clip)

Console output:

[useHeyGenAvatar] Avatar lip-sync started
[useHeyGenAvatar] Avatar lip-sync ended
[useHeyGenAvatar] Avatar lip-sync started
[useHeyGenAvatar] Avatar lip-sync ended
... (repeats 100+ times in rapid succession)

Verification Steps Taken

We verified the issue is not in our PCM encoding:

1. Attempt 1 — OfflineAudioContext resampling: Fetched WAV (22050Hz) → decoded with AudioContext.decodeAudioData → resampled to 24kHz via OfflineAudioContext → Float32 to Int16 → binary string → repeatAudio(). Same rapid start/stop, no lip-sync.

2. Attempt 2 — Manual WAV parsing + linear interpolation: Parsed WAV header manually → extracted raw Int16 PCM → resampled 22050→24000 with linear interpolation → binary string → repeatAudio(). Identical result.

3. FULL mode works: The same avatar works perfectly with session.repeat(text) in FULL mode — lip-sync, audio, and speaking lifecycle events all function correctly.

4. LITE mode session connects fine: SESSION_STREAM_READY fires, video stream renders, SESSION_STATE_CHANGED: CONNECTED fires. The session itself is healthy.

PCM Diagnostics

WAV: 22050Hz, 16bit, 1ch, 45678 bytes
Extracted 22000 mono samples, duration: 1.00s
Resampled 22000 (22050Hz) → 24000 (24000Hz)
Binary string: 48000 chars (24000 samples, 1.00s)
Calling repeatAudio with 48000 chars PCM

The PCM data matches the format expected by splitPcm24kStringToChunks() (960 bytes = 20ms at 24kHz mono 16-bit).

Use Case

We want to use an external TTS engine (Vachana AI, Indian English) to provide the voice while using HeyGen's avatar for lip-sync. This is the intended use case of LITE mode + repeatAudio() per the SDK architecture, but it doesn't work for custom VIDEO avatars.

Question

Is repeatAudio() in LITE mode supported for custom VIDEO avatars? If not, are there plans to support it? This would unlock third-party TTS integration which is a significant use case for non-English voices.

[ruben-ondaloop]

I'm seeing the exact same behaviour with @heygen/liveavatar-web-sdk@0.0.17. I'm posting a second independent reproduction in case it helps maintainers triage this.

Setup

• SDK version: @heygen/liveavatar-web-sdk@0.0.17
• Avatar type: custom video avatar
• Session mode: LITE (no elevenlabs_agent_config — true BYO audio path)
• Audio source: ElevenLabs Conversational AI via @elevenlabs/client@0.13.0, connectionType: "websocket". The agent's TTS output format is configured to PCM 24000 Hz in the ElevenLabs dashboard.
• Browser: Chrome 142 (also reproduced on Firefox).

What I feed to repeatAudio

In the EL SDK onAudio(base64) callback, I decode and forward each chunk:

session.repeatAudio(atob(base64Pcm24k))

The very first chunk I send is large (typical 24K-binary length ~233000–253000 bytes for a 5s utterance). Console diagnostics confirm:

[VoiceSession] negotiated formats — output: {format: 'pcm', sampleRate: 24000} input: {format: 'pcm', sampleRate: 16000}
[LiveAvatarRenderer] first audio chunk: { b64Length: 337100, binaryLength: 252824, firstBytes: '253,255,0', streamReady: true }

So: format and sample rate match the SDK contract; SESSION_STREAM_READY has fired before the call; the first byte values are plausible 16-bit signed PCM samples, with small near-silence values.

Symptoms

• The session itself is healthy: SESSION_STREAM_READY fires, the avatar's video stream renders, and attach(videoElement) works.
• After repeatAudio() is called: no lip-sync, no audio output through the LiveKit audio track.
• I added listeners for AVATAR_SPEAK_STARTED / AVATAR_SPEAK_ENDED / SESSION_STATE_UPDATED, and none of them fire for repeatAudio-driven utterances. I did not see the rapid start/end cycling reported in the original issue — in my case there's complete silence on the avatar side.
• ElevenLabs is healthy: I receive the conversation transcript via onMessage and the audio chunks via onAudio correctly. EL's own native playback works fine when I don't silence it. The only thing missing is HeyGen producing the lip-sync.

Verification I ruled out client-side

1. The output format negotiated by the EL SDK is exactly {format: 'pcm', sampleRate: 24000}. Note: WebRTC mode forces pcm_48000 — I explicitly use connectionType: "websocket" to honour the agent's PCM 24K config.
2. I pass voiceChat: false to LiveAvatarSession so the SDK doesn't capture the user's mic; the mic is owned by the EL Conversation. This isolates the LITE → BYO audio path cleanly.
3. The avatar lip-syncs correctly in FULL mode (mode: "FULL", interactivity_type: "CONVERSATIONAL") with the same avatar_id. So the avatar itself is fine; the path that fails is specifically LITE + repeatAudio().
4. I verified _sessionEventSocket.readyState === OPEN at the moment I call repeatAudio().

Use case

I'm integrating ElevenLabs Conversational AI as the conversation engine (ASR + LLM + TTS) and HeyGen as the visual avatar. I previously tried the hosted elevenlabs_agent_config connector, but that introduces audible audio degradation: browser → LiveKit → LiveAvatar worker → ElevenLabs round-trip causes quality loss compared with the native EL signed-URL path.

The whole point of LITE + repeatAudio() for me is to keep the EL audio path lossless and only use HeyGen for lip-sync — exactly the use case the SDK advertises.

What would unblock me

A confirmation that BYO audio via repeatAudio() is supported for custom video avatars, and what I'm doing wrong if so; or a published timeline / workaround if this is a known backend limitation.

[mvtandas]

Server confirms: "invalid base64 audio data" — SDK ↔ server protocol mismatch

Same symptoms as the OP and @ruben-ondaloop: LITE mode, custom avatar, repeatAudio() produces no lip-sync, no audio, and no AVATAR_SPEAK_STARTED / AVATAR_SPEAK_ENDED events. Tested with both 0.0.10 and 0.0.17.

I tapped the SDK's _sessionEventSocket to log every inbound message and the server replies with an explicit error event for each chunk the SDK sends:

{ "type": "error",
  "event_id": "84ece05d-d5b9-49ae-8f42-05af2ca5eb1d",
  "error": {
    "type": "invalid_request_error",
    "message": "invalid base64 audio data",
    "event_id": "cd15757d-f52a-46f9-8ecc-d57bb9939e52"
  } }

One such error is emitted per agent.speak packet — for a 12.5s utterance with 0.0.17 chunking I see ~13 errors, and the SDK silently drops them because handleWebSocketMessage only emits agent.speak_started / agent.speak_ended.

Root cause

The server expects the audio field of agent.speak to be base64-encoded PCM 16-bit 24 kHz mono (matches https://docs.liveavatar.com/docs/lite-mode/events). The SDK, in both 0.0.10 and 0.0.17, sends the raw "binary string" produced by splitPcm24kStringToChunks():

https://github.com/heygen-com/liveavatar-web-sdk/blob/master/lib/index.esm.js — sendCommandEventToWebSocket case AVATAR_SPEAK_AUDIO: audio: audioChunk is the raw 1-byte-per-char string, not btoa(audioChunk).

So both the OP's and @ruben-ondaloop's setups (and ours) are doing exactly what the SDK's TypeScript signature documents (repeatAudio(audio: string) "raw 16-bit signed PCM") — the bug is on the SDK's send path, not in the caller.

Workaround we shipped

Until a fix lands, we wrap _sessionEventSocket.send after session.start() and base64-encode the audio field on the way out:

const sock: WebSocket | undefined = (session as any)._sessionEventSocket;
if (sock && !(sock as any).__base64AudioPatched) {
  const origSend = sock.send.bind(sock);
  sock.send = (data: any) => {
    if (typeof data === 'string') {
      try {
        const parsed = JSON.parse(data);
        if (parsed?.type === 'agent.speak' && typeof parsed.audio === 'string') {
          parsed.audio = btoa(parsed.audio);
          return origSend(JSON.stringify(parsed));
        }
      } catch { /* not our shape */ }
    }
    return origSend(data);
  };
  (sock as any).__base64AudioPatched = true;
}

With this patch in place, lip-sync and audio work end-to-end and AVATAR_SPEAK_STARTED / AVATAR_SPEAK_ENDED fire as expected. I'd be happy to send a PR moving this into sendCommandEventToWebSocket itself if maintainers prefer.

Suggested fix

Either:

1. Have the SDK btoa() the audio chunk inside sendCommandEventToWebSocket before sending, or
2. Update the server to accept the raw-binary-string format the SDK currently sends (and update the docs page accordingly).

The current state is that repeatAudio() is broken end-to-end on the SDK's documented contract.

[mvtandas]

Opened #103 with the SDK-side fix (btoa() the chunk inside sendCommandEventToWebSocket before send) and an extended test asserting the wire format. Verified end-to-end against a real LITE session — AVATAR_SPEAK_STARTED / AVATAR_SPEAK_ENDED fire and the avatar lip-syncs correctly with the patch in place.