saa-js

@attenlabs/saa-js

JavaScript SDK for Attention Labs Selective Auditory Attention (SAA): the addressee layer for voice agents. One decision per utterance about whether speech was meant for your agent, before STT, LLM, or TTS. No wake word, model-agnostic, drop-in for the voice stack you already use.

The mental model is simple: audio in -> addressee gate (the SAA decision) -> only addressed audio out. The attention model runs on Attention Labs' service, so this is a thin Apache-2.0 client: it captures and streams your mic + camera and emits typed events. All inference runs server-side.

Sign up

Get your API key at attentionlabs.ai.

Install

npm install @attenlabs/saa-js

Quick start

import { AttentionClient } from "@attenlabs/saa-js";

const videoEl = document.querySelector("video");

const client = new AttentionClient({
  token: "your-api-key",
});

client.on("prediction", ({ cls, confidence, source, numFaces }) => {
  console.log(`${cls}: ${confidence.toFixed(2)}`);
});

client.on("turnReady", ({ audioBase64, durationSec }) => {
  // Forward the captured turn to your LLM of choice
});

await client.start({ videoElement: videoEl });

Options

Option	Type	Default	Description
token	string	none	Your API key from attentionlabs.ai.
initialThreshold	number	0.7	Confidence threshold for predictions (0-1).
enableAudio	boolean	true	Capture the mic internally. Set false to push audio via feedAudio().
enableVideo	boolean	true	Capture the camera internally. Set false for audio-only or to push frames via feedVideo().
autoReconnect	boolean	true	Reconnect with backoff after an unclean mid-session drop. Set false to surface the drop as an error instead.
serverProfile	string	inferred	Server processor variant. Defaults to "audio_only" when enableVideo: false, else the full processor. Pass "default" to force the full processor without local video.
workletUrl	string	bundled	URL of the audio-capture AudioWorklet module. Override only when self-hosting the worklet.
video.width	number	1920	Capture width.
video.height	number	1080	Capture height.
video.jpegQuality	number	0.5	JPEG quality (0-1).
audio.targetSampleRate	number	16000	Sample rate audio is resampled to before sending.
audio.onAudioFrame	function	none	Called with each captured 16-bit PCM frame (ArrayBuffer).
audio.onWorkletError	function	none	Called when the capture worklet throws (also emitted as an error event).
audio.onContextStateChange	function	none	Called with the AudioContext state string on change (e.g. suspended, interrupted).

Methods

Method	Description
start({ videoElement, mediaStream? })	Start streaming + connect. Calls getUserMedia unless mediaStream is supplied. videoElement is required when video capture is enabled.
stop()	Stop streaming and disconnect.
feedAudio(audio, sampleRate?)	Push externally-captured audio (requires enableAudio: false). Accepts Float32 [-1,1], Int16 PCM, or a raw int16 buffer; re-chunked + resampled to the wire's 16 kHz / 100 ms blocks. See External capture.
feedVideo(jpeg)	Push an externally-captured JPEG frame (requires enableVideo: false). Accepts a Blob, ArrayBuffer, or view.
mute() / unmute()	Pause or resume audio.
markResponding(boolean)	Signal that your app is responding, pauses predictions until finished.
setThreshold(value)	Update the confidence threshold (0-1).
isConnected	Getter — true while the WebSocket is open.
currentThreshold	Getter — the current confidence threshold (0-1).
on(event, listener)	Subscribe to an event. Returns an unsubscribe function.

Events

Event	Payload
connected	none
started	none
warmupComplete	none
prediction	{ cls, rawCls, confidence, source, numFaces, responding }
vad	{ probability, isSpeech }
state	{ state } (one of listening, sending, cancelled, idle)
turnReady	{ audioBase64, audioPcm16, durationSec, frames, context }
config	{ modelClass2Threshold }
stats	{ rttMs, bufferedAmount, sentVideo, skippedVideo, sentAudio, uptimeMs }
interrupt	{ fadeMs, confidence }
interjection	{ reason, audioBase64, audioPcm16, durationSec }
error	{ title, message, detail }
disconnected	{ code, reason, wasClean }
reconnecting	{ attempt, delaySec, lastCode }
reconnected	{ attempts }

warmupComplete fires once the server model has warmed up and is producing real predictions; use it to drop any loading UI. prediction.responding is true while your app is mid-response (see markResponding), and interjection fires when the agent should volunteer after humans go quiet.

If the camera is unavailable when video capture is enabled and audio is enabled, start() continues with an audio-only session (audio_only server profile), emits an error with kind: "environment" and title: "Camera unavailable". The original enableVideo request is restored on the next start(), so a later session retries video.

LLM integration

The SDK captures speech but does not route it to an LLM. SAA is model-agnostic and drop-in: use the turnReady event to forward only device-directed audio to any model, ASR, or voice stack you like.

When your LLM starts responding, call client.mute() and client.markResponding(true). When it finishes, call client.unmute() and client.markResponding(false).

External capture

By default the SDK opens its own mic + camera. To run on capture you already own there are two paths:

Share a MediaStream (the SDK reads it but won't stop its tracks):

const stream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true });
videoEl.srcObject = stream;                 // your app renders it
await client.start({ videoElement: videoEl, mediaStream: stream });
// ... another consumer (e.g. a gaze SDK) reads the same stream / videoEl

Push frames yourself (no getUserMedia at all) for taps, Twilio media, or non-browser sources:

const client = new AttentionClient({ token, enableAudio: false, enableVideo: false });
await client.start();                        // opens the WS, captures nothing
client.feedAudio(pcmChunk);                  // Float32 [-1,1] | Int16 | int16 buffer
client.feedAudio(pcm48k, 48000);             // resampled to 16 kHz
client.feedVideo(jpegBlob);                  // Blob | ArrayBuffer | view

Mix and match: enableVideo: false with internal mic for audio-only, or enableAudio: false + feedAudio() while the SDK still grabs camera frames.

License

Apache-2.0