Skip to main content

WebSocket Protocol

The platform exposes a single WebSocket endpoint that handles both text and voice conversations between an end user and one agent. This is the protocol your client (browser, mobile app, ESP firmware, custom integration) speaks.

REST is documented separately — see the API Reference.

Endpoint​

wss://api.<your-domain>/user/agents/{agent_id}/ws

One connection = one conversation with one agent.

Authentication​

Pass the user's Cognito JWT one of two ways:

  • Authorization header — Authorization: Bearer <jwt> (preferred for native clients).
  • token query parameter — wss://.../ws?token=<jwt> (handy in browsers, where you can't set custom WS headers).

The server accepts the WebSocket first, then verifies the token. If verification fails, the server sends an error message and closes with code 1008.

There's no token refresh on the WebSocket itself — when your token expires, you reconnect with a fresh one.

Lifecycle at a glance​

The first message on the socket must be a handshake. If it isn't, the server sends an error and closes the socket. See Errors & Lifecycle → Close codes for what to expect.

Message format​

Every message — both directions — is a JSON envelope:

{
  "type":         "<message_type>",
  "content_type": "<content_type>",
  "content":      <payload>,
  "metadata":     { ... }
}

Binary frames (no JSON) are used only for raw audio during voice chat.

See the Message Reference for every type and content_type.

What you can do over this socket​

ModePages
Text chatText Chat — send user text, receive streamed assistant chunks.
Voice chatVoice Chat — stream PCM/Opus audio, get audio back, with barge-in.
ToolsTool Calls — tool_request / tool_response for client-executed tools.
ResumeReconnect — pick up an existing conversation.
ErrorsErrors & Lifecycle — error codes, close codes, TTL.

Quick example (text)​

const ws = new WebSocket(
  `wss://api.example.com/user/agents/${agentId}/ws?token=${jwt}`
);

ws.onopen = () => {
  // Step 1: send handshake first
  ws.send(JSON.stringify({
    type: "handshake",
    content_type: "json",
    content: { conversationType: "text" },
  }));
};

ws.onmessage = (ev) => {
  // Binary audio frames arrive as ArrayBuffer; JSON arrives as string
  if (typeof ev.data !== "string") return;
  const msg = JSON.parse(ev.data);

  if (msg.type === "handshake_ack") {
    // Step 2: send the first user message after the ack
    ws.send(JSON.stringify({
      type: "user",
      content_type: "text",
      content: "Hello!",
    }));
  } else if (msg.type === "assistant") {
    process.stdout.write(msg.content);     // streamed chunk
  } else if (msg.type === "transaction_end") {
    console.log("\n[turn done]");
  }
};

Conventions​

  • JSON for control, binary for audio bytes. Never send raw audio inside a JSON message — use a binary frame between audio_stream_start and audio_stream_end.
  • Don't expect a single assistant message — expect a stream. The agent emits multiple assistant chunks per turn, finished by a transaction_end.
  • Always handle error and close. Even healthy sessions hit a TTL eventually.
  • One conversation per socket. Don't try to multiplex.

On this page