Skip to main content

Tool Calls

When the agent decides to call a tool, you see those decisions over the WebSocket. Some tools run server-side; local tools run on your client and require you to respond.

Two kinds of tool flow​

Tool kindWhere it runsYour client's role
Server-side (Cloud, Remote/MCP)Platform / external serverRead-only — you see informational messages, the agent gets the result.
Local (device-side)Your clientYou receive tool_request, execute, send tool_response.

Server-side tools — informational only​

When a server-side tool runs, you receive these messages mid-turn:

S → C  { "type": "tool_call_info",   "content": "weather: {\"city\":\"Berlin\"}" }
S → C  { "type": "tool_result_info", "content": "weather: {\"temp_c\":14}" }
S → C  { "type": "assistant",        "content": "It's 14°C in Berlin." }
S → C  { "type": "transaction_end",  "content": "done" }

tool_call_info and tool_result_info are purely informational — they let your UI show that a tool was used. You don't need to reply.

The content is a single string formatted as <tool_name>: <JSON>. Parse it if you want to render a card.

Local tools — request / response​

Local tools (defined by the firmware on an ESP device, for example) can't be called by the platform. The agent asks you to run them and report back.

Step 1: receive tool_request​

{
  "type": "tool_request",
  "content_type": "json",
  "content": {
    "request_id": "req_abc123",
    "tool_name": "set_volume",
    "input": { "level": 30 },
    "server_name": "Living Room MCP"
  }
}

server_name is optional. It's set when the local tool was discovered via a specific local MCP server (see Handshake → Local tool discovery) — clients running multiple local MCP servers should use it to route the request to the right backend. For tools registered without an MCP server, server_name is omitted.

Step 2: execute on the client​

Look up the tool by name, run it with the provided input, capture the result.

Step 3: send tool_response​

{
  "type": "tool_response",
  "content_type": "json",
  "content": {
    "request_id": "req_abc123",
    "result": { "ok": true, "previous": 25 }
  }
}

The request_id must match the request. Mismatched or missing IDs are dropped silently and the agent waits forever — so be careful.

Step 4: agent continues​

The agent receives the result, may emit a tool_result_info for visibility, and continues generating its reply.

End-to-end example​

C → S  { "type": "user", "content": "Turn the volume down to 30" }

S → C  { "type": "tool_call_info",
         "content": "set_volume: {\"level\":30}" }

S → C  { "type": "tool_request",
         "content": { "request_id": "req_abc", "tool_name": "set_volume", "input": { "level": 30 } } }

C → S  { "type": "tool_response",
         "content": { "request_id": "req_abc", "result": { "ok": true } } }

S → C  { "type": "tool_result_info",
         "content": "set_volume: {\"ok\":true}" }

S → C  { "type": "assistant", "content": "Done — volume is now 30." }

S → C  { "type": "transaction_end", "content": "done" }

Errors during tool execution​

If your tool fails, return a structured error in result:

{
  "type": "tool_response",
  "content_type": "json",
  "content": {
    "request_id": "req_abc",
    "result": {
      "ok": false,
      "error": "device_unreachable",
      "message": "Speaker did not respond."
    }
  }
}

The agent treats this as input for its next turn — usually it'll explain the failure to the user and offer alternatives.

Don't throw away the request. Always send a tool_response, even on failure. Otherwise the agent stalls.

Tools and timeouts​

The server gives the client 30 seconds per local tool request. If you don't reply with a tool_response in time, the server feeds the agent a synthetic error like:

{ "error": "Local tool 'set_volume' execution timed out" }

…and the conversation continues — usually with the agent telling the user the tool was unresponsive. The WebSocket itself stays open. If your tool genuinely takes longer than 30 seconds, return a quick "in progress" tool_response and surface the result through a separate channel.

Tips​

  • Don't render local tool inputs/outputs by default. If the user wants to see them, surface a "show details" toggle.
  • Idempotent tools are easier. A reconnect mid-tool-call may cause the agent to re-issue the request.
  • Validate inputs before executing. The schema is enforced server-side, but defensive validation on the client catches edge cases the LLM dreams up.
  • Send the response promptly. A reply within a few hundred milliseconds keeps the conversation snappy.

Reference​

On this page