Message Reference

Every message — both directions — uses the same envelope. This page is the catalog of every type and content_type value, with the content shape for each.

Envelope

{
  "type":         "<message_type>",
  "content_type": "<content_type>",
  "content":      <payload>,
  "metadata":     { ... }
}

Field	Required	Notes
type	yes	One of the message types listed below.
content_type	yes	Tells you how to parse content.
content	yes	The payload. Type depends on content_type.
metadata	no	Free-form key/value bag. The server adds sequence_number and timestamp to most messages.

Content types

content_type	content shape	Used by
text	string	Most text messages
json	object	Control messages, structured payloads
multimodal	{ text, media[] }	User turns with images / files

For voice, raw audio bytes flow as WebSocket binary frames — they're not wrapped in this envelope at all. The envelope is used for control (audio_stream_start, audio_stream_end).

Message types

The full set of valid type values, grouped by purpose.

Lifecycle

handshake (client → server)

The first message. See Handshake.

handshake_ack (server → client)

Reply to a successful handshake. See Handshake.

transaction_end (server → client)

End-of-turn marker.

{
  "type": "transaction_end",
  "content_type": "text",
  "content": "done",
  "metadata": {
    "sequence_number": 42,
    "timestamp": 1714592834012,
    "total_duration_ms": 1872
  }
}

metadata.total_duration_ms is included in text mode (the time from the start of the turn to the end). It's omitted in voice mode.

User turns (client → server)

user

A user message. Either text or multimodal.

// Text
{ "type": "user", "content_type": "text", "content": "Hi!" }

// Multimodal
{
  "type": "user",
  "content_type": "multimodal",
  "content": {
    "text": "What's in this picture?",
    "media": [
      {
        "type": "image",
        "media_id": "med_abc",
        "s3_key": "media/...",
        "content_type": "image/png",
        "filename": "x.png",
        "size_bytes": 12345
      }
    ]
  }
}

media[].type can be "image", "video", or "document". Upload media via REST first to obtain media_id / s3_key.

Assistant output (server → client)

assistant

The assistant's reply. In non-streaming mode, one envelope per turn carries the full reply. In streaming mode, the final assembled reply is also emitted as assistant after the deltas finish — concatenate the assistant_delta chunks for the streamed view.

{ "type": "assistant", "content_type": "text", "content": "Hello, how can I help?" }

In voice mode, assistant envelopes are emitted as transcripts alongside audio frames. They carry metadata.generation_stage: "speculative" | "final" — see Voice Chat → User and assistant transcripts.

assistant_delta

A streaming chunk of the assistant's reply. Emitted when the agent's text model is configured to stream. Concatenate content across deltas until transaction_end.

{ "type": "assistant_delta", "content_type": "text", "content": "Hello, how" }

thinking

A complete extended-thinking block, emitted in non-streaming mode for models that expose chain-of-thought. Render this collapsibly if you support it; otherwise ignore.

{ "type": "thinking", "content_type": "text", "content": "The user is asking..." }

thinking_delta

A streaming chunk of extended thinking. Same rendering guidance as thinking.

{ "type": "thinking_delta", "content_type": "text", "content": "The user is..." }

Tool-call messages

See Tool Calls for end-to-end flow.

tool_call_info (server → client)

Informational notice that a server-side tool was invoked.

{
  "type": "tool_call_info",
  "content_type": "text",
  "content": "weather: {\"city\":\"Berlin\"}"
}

tool_result_info (server → client)

Informational notice that a server-side tool returned.

{
  "type": "tool_result_info",
  "content_type": "text",
  "content": "weather: {\"temp_c\":14}"
}

tool_request (server → client)

The agent asks the client to execute a local tool.

{
  "type": "tool_request",
  "content_type": "json",
  "content": {
    "request_id": "req_abc",
    "tool_name": "set_volume",
    "input": { "level": 30 },
    "server_name": "Living Room MCP"
  }
}

server_name is optional; it's set when the tool was discovered through a specific local MCP server. See Tool Calls.

tool_response (client → server)

Your reply to a tool_request. The request_id must match.

{
  "type": "tool_response",
  "content_type": "json",
  "content": {
    "request_id": "req_abc",
    "result": { "ok": true }
  }
}

Audio (voice mode)

See Voice Chat for end-to-end flow.

audio_stream_start (both directions)

{
  "type": "audio_stream_start",
  "content_type": "json",
  "content": {},
  "metadata": {
    "role": "user" | "assistant",
    "sample_rate": 8000,
    "encoding": "pcm"
  }
}

When the server sends audio_stream_start for the assistant, it also sets metadata.sample_rate and metadata.encoding. Today both are hard-coded to 8000 and "pcm" regardless of what was negotiated in the handshake — see the caveat in Voice Chat.

audio_stream_end (both directions)

{
  "type": "audio_stream_end",
  "content_type": "json",
  "content": { "reason": "end_turn" | "interrupted" | "timeout" },
  "metadata": { "role": "user" | "assistant" }
}

barge_in (server → client)

The platform's voice model detected user speech overlapping with assistant audio and cancelled the assistant turn. Use this to stop local playback. See Voice Chat → Barge-in.

{
  "type": "barge_in",
  "content_type": "text",
  "content": "interrupted",
  "metadata": {
    "interrupted_at_ms": 1714592834012,
    "sequence_number": 17,
    "timestamp": 1714592834012
  }
}

event (client → server)

Voice-mode-only control message that lets the client end the session gracefully. Content is "end", "stop", or "close".

{ "type": "event", "content_type": "text", "content": "end" }

Usage and quotas (server → client)

The shapes of these envelopes differ slightly between text and voice modes.

usage_info

Per-turn usage update. Emitted in text mode only.

In text mode, content_type is "text" and content is a JSON-encoded string — clients should JSON.parse(content) to read the fields:

{
  "type": "usage_info",
  "content_type": "text",
  "content": "{\"total_input_tokens\":1234,\"total_output_tokens\":567,\"total_tokens\":1801}"
}

usage_warning

Quota approaching (typically over 80%).

In text mode, content_type is "text" and content is a human-readable string with no metadata:

{
  "type": "usage_warning",
  "content_type": "text",
  "content": "⚠️ Quota warning: 85% used (8500 / 10000 tokens)."
}

In voice mode, the structured form is used: content_type is "json", content carries the usage fields, and metadata.content carries the human string:

{
  "type": "usage_warning",
  "content_type": "json",
  "content": {
    "current_usage": 8500,
    "limit": 10000,
    "percentage": 85
  },
  "metadata": { "content": "Quota warning: 85% used." }
}

usage_error

Quota exceeded or limit hit. Same text-vs-voice split as usage_warning. The chat input should be disabled in your UI when this fires.

Error

error (server → client)

error envelopes come in two shapes — pick the one that matches content_type. See Errors & Lifecycle for the full split and a recommended client-side parser.

Structured (handshake validation, TIMED_OUT, handshake-time QUOTA_EXCEEDED):

{
  "type": "error",
  "content_type": "json",
  "content": {
    "code": "INVALID_INPUT_FORMAT",
    "message": "Invalid input format: 'mp3'"
  }
}

Plain text (mid-session errors, AUDIO_STREAM_NOT_STARTED, mid-session voice quota, tool execution errors):

{
  "type": "error",
  "content_type": "text",
  "content": "human-readable message",
  "metadata": { "code": "ERROR_CODE" }
}

metadata.code is set on some text-shaped errors (e.g. AUDIO_STREAM_NOT_STARTED) and absent on others (e.g. mid-session voice quota, tool execution errors).

Common metadata fields

The server attaches these to most outgoing messages. They're optional reading for clients but useful for debugging:

Field	Meaning
metadata.role	"user" / "assistant" for audio messages.
metadata.sequence_number	Monotonically increasing across messages within a session. Useful when you need ordering. Not attached to handshake_ack.
metadata.timestamp	Server time in milliseconds. Not attached to handshake_ack.
metadata.code	On text-shaped error messages — the machine-readable error code (when set). For json-shaped errors, the code lives in content.code.
metadata.transcript	true on voice-mode user envelopes that carry transcribed user speech.
metadata.generation_stage	"speculative" or "final" on voice-mode assistant envelopes.
metadata.interrupted_at_ms	On barge_in — server time of the detected interruption.
metadata.total_duration_ms	On text-mode transaction_end — turn duration in ms.

Quick lookup

Direction	Type	content_type	Notes
C→S	handshake	json	First message.
S→C	handshake_ack	json	Confirms params + conversationId.
S→C	local_tools_discover	json	Mid-handshake; only when capabilities includes local_tools_discover.
C→S	local_tools_supported	json	Reply to local_tools_discover.
C→S	user	text / multimodal	A user turn.
S→C	assistant	text	Final reply (or per-stage transcript in voice).
S→C	assistant_delta	text	Streaming chunk (text-mode streaming).
S→C	thinking / thinking_delta	text	Extended-thinking blocks.
S→C	tool_call_info / tool_result_info	text	Server-side tool info.
S→C	tool_request	json	Local tool to run.
C→S	tool_response	json	Local tool result.
C↔S	audio_stream_start / audio_stream_end	json	Audio framing.
S→C	barge_in	text	Server-detected interruption notice.
C→S	event	text	Voice-mode graceful end ("end"/"stop"/"close").
C↔S	(binary frame)	—	Raw audio bytes.
S→C	transaction_end	text	End of turn.
S→C	usage_info	text	Per-turn token usage (text mode only; JSON-encoded string in content).
S→C	usage_warning / usage_error	text (text mode) / json (voice mode)	Quota / billing.
S→C	error	json (handshake/lifecycle) / text (mid-session)	Recoverable or fatal.