Voice Chat

Real-time voice in, voice out. Audio bytes flow as WebSocket binary frames between control messages. Everything else is JSON.

Audio formats

Format	Sample rate	Frame duration	Notes
audio/pcm	8000, 16000	n/a	16-bit signed mono. Default.
audio/opus	8000, 16000	20, 40, 60 ms	Lower bandwidth. frameDurationMs is required.

You declare your choice in the handshake's audioConfiguration. See Handshake → audioConfiguration.

The platform converts between PCM and Opus internally — your client picks whichever it implements.

Today's gateway uses 8 kHz for both directions regardless of what the handshake's audioConfiguration.output.sampleRate declares — the Nova Sonic voice path is wired to 8 kHz. The handshake validation accepts 16000, but the speaker stream is currently 8 kHz. Plan for that until the platform's voice model supports higher rates end-to-end.

Steps

1. Connect to the WebSocket and send the audio handshake (see Handshake).
2. Wait for handshake_ack.
3. Send an audio_stream_start control message with metadata.role: "user".
4. Stream raw audio chunks as binary WebSocket frames.
5. Send an audio_stream_end when the user stops talking.
6. Receive the agent's response:
   • audio_stream_start (server, metadata.role: "assistant").
   • Binary audio frames.
   • audio_stream_end.
   • transaction_end.
7. Repeat.

Control messages

audio_stream_start (client → server)

Marks the beginning of a user audio segment. Send before any binary frames.

{
  "type": "audio_stream_start",
  "content_type": "json",
  "content": {},
  "metadata": { "role": "user" }
}

If you send raw audio bytes without first sending audio_stream_start, the server emits an error with code AUDIO_STREAM_NOT_STARTED.

audio_stream_end (client → server)

Marks the end of the user's audio segment. The server then waits for the model to respond.

{
  "type": "audio_stream_end",
  "content_type": "json",
  "content": { "reason": "end_turn" },
  "metadata": { "role": "user" }
}

reason is informational — "end_turn" is the common case.

Server-side equivalents

The server sends the same two control messages around the assistant's audio output, with metadata.role: "assistant". Your client should:

• On audio_stream_start (assistant) — clear the audio buffer; prepare to play.
• On binary frames — play them in order.
• On audio_stream_end (assistant) — mark playback complete; treat any subsequent message as a new turn.

The server's audio_stream_end may carry an extra reason:

reason	Meaning
"end_turn"	Agent finished naturally.
"interrupted"	The user barged in (see below).
"timeout"	Model timed out.

Barge-in (interrupting the agent)

Barge-in is server-detected. When the user starts speaking while the agent is still talking, just keep streaming the user's audio — the platform's voice model detects the overlap and:

1. Cancels the in-flight assistant turn.

2. Sends you a barge_in envelope server → client:

   {
     "type": "barge_in",
     "content_type": "text",
     "content": "interrupted",
     "metadata": {
       "interrupted_at_ms": 1714592834012,
       "sequence_number": 17,
       "timestamp": 1714592834012
     }
   }

3. Use that signal to stop playing whatever assistant audio you've already buffered locally and to mark the previous assistant turn as interrupted in your UI.

You don't need to send anything special on the client side to trigger a barge-in — just keep streaming user audio over the existing audio_stream_start.

Binary audio frame format

• Frame type: WebSocket binary frame (not JSON).
• Payload: Raw bytes of the agreed-upon format.
• PCM: linear 16-bit signed little-endian, mono. At 8 kHz, that's 16,000 bytes per second.
• Opus: encoded frames; one Opus packet per WS frame is a safe pattern. The platform decodes using the frameDurationMs you declared.

There's no envelope, no length prefix — just bytes. The server reads them as they arrive.

User and assistant transcripts during voice

Voice turns also produce text envelopes you can render as captions:

• A user envelope (content_type: "text") with metadata.transcript: true carrying the transcribed user speech.
• One or more assistant envelopes (content_type: "text") carrying the assistant's transcribed reply, with metadata.generation_stage set to either "speculative" (an early hypothesis) or "final" (the persisted text). For caption rendering, prefer final and either ignore or render-then-replace speculative.

Both are emitted alongside the assistant audio frames; you don't need to do anything special to receive them.

Ending a voice session gracefully

To close the voice session cleanly without dropping the WebSocket, send an event message with content "end", "stop", or "close":

{
  "type": "event",
  "content_type": "text",
  "content": "end"
}

The server tears down the voice session (Nova Sonic stream) and the WebSocket closes normally afterward. This is the only client-initiated graceful end besides closing the WebSocket itself.

Mixed text + voice

You can mix text and voice in one conversation. After ending an audio segment, send a user text message, then go back to audio later. Both are recorded as turns in the same conversation.

Recommended client architecture

For browsers / Node:

• Capture: getUserMedia → AudioWorkletNode → resample to 8 kHz → PCM Int16Array → send as binary.
• Playback: receive binary → decode if Opus → schedule into an AudioBufferSourceNode queue.

For ESP firmware: use the on-device Opus encoder, send 20 ms frames as binary WS messages, decode the response back into PCM for the speaker.

Latency budget

A typical voice turn breaks down as:

Phase	Time
Audio capture buffer (one frame)	20–60 ms
Network upload	30–80 ms
Speech recognition + LLM inference	300–1500 ms
Network download (first audio frame)	30–80 ms
Playback start	< 10 ms

End-to-end first-audio latency is usually 0.5–2 seconds depending on network and model.

Tips

• Use Opus on flaky networks. PCM at 16 kHz is ~32 KB/s; Opus is more like 4–8 KB/s.
• Keep your barge-in path responsive. A delay of more than ~150 ms makes interruptions feel sluggish.
• Always pair audio_stream_start with audio_stream_end. Forgetting the end leaves the server waiting.
• Don't send empty audio chunks, but it's fine to send small ones (a few frames at a time).

Full sketch

const ws = new WebSocket(`wss://api.example.com/user/agents/${agentId}/ws?token=${jwt}`);
ws.binaryType = "arraybuffer";

ws.onopen = () => {
  ws.send(JSON.stringify({
    type: "handshake",
    content_type: "json",
    content: {
      conversationType: "audio",
      audioConfiguration: {
        input:  { format: "audio/pcm", sampleRate: 8000 },
        output: { format: "audio/pcm", sampleRate: 8000 },
      },
    },
  }));
};

ws.onmessage = (ev) => {
  if (ev.data instanceof ArrayBuffer) {
    // assistant audio frame — feed to playback
    enqueueAudio(ev.data);
    return;
  }
  const msg = JSON.parse(ev.data);
  if (msg.type === "handshake_ack") startMicCapture();
  else if (msg.type === "audio_stream_start" && msg.metadata?.role === "assistant") prepareSpeaker();
  else if (msg.type === "audio_stream_end" && msg.metadata?.role === "assistant") finishSpeaker();
};

function startMicCapture() {
  ws.send(JSON.stringify({
    type: "audio_stream_start",
    content_type: "json",
    content: {},
    metadata: { role: "user" },
  }));

  micWorklet.onaudio = (pcmInt16) => ws.send(pcmInt16.buffer);

  micWorklet.onsilence = () => {
    ws.send(JSON.stringify({
      type: "audio_stream_end",
      content_type: "json",
      content: { reason: "end_turn" },
      metadata: { role: "user" },
    }));
  };
}