跳到主要内容

Local Tools

A local tool is a function that runs on the device. The agent calls it from the cloud — "set volume to 30", "what time is it", "turn off the bedroom light" — and the firmware executes it locally and reports back.

This is different from a cloud tool, which runs on the platform.

Tools that ship today​

Voice Chat (friend)​

ToolWhat it does
set_volumeSet speaker volume (0–100).
set_emotionSet the on-screen emoji to one of: Sad, Happy, Confused, Shocked, Angry, Cry, Wink, Neutral.
set_reminderSchedule a local reminder that fires later.
get_local_timeReturns current device time.

Matter Controller (matter_controller)​

All of the above, plus:

ToolWhat it does
get_device_listLists Matter nodes in the controller's fabric.
control_deviceSends a Matter command.

How they work​

When the agent decides to call a local tool, it sends a tool_request over the WebSocket. The firmware's tool handler runs, returns a result, and the firmware sends a tool_response back. The agent then continues its reply with the tool's output incorporated.

Cloud agent  ──── tool_request ────► Device (firmware)
                                          │
                                          ▼
                                   handler runs
                                          │
Cloud agent  ◄──── tool_response ────  result returned
        │
        ▼
Agent reply continues with tool output

Tool handlers are short, synchronous C functions. They don't block the audio path — they run on a separate task.

Adding a tool​

Steps to add say_hello (a trivial example):

1. Write the handler​

In examples/voice_chat/main/app_tools.c:

static esp_err_t say_hello_handler(
    esp_agent_handle_t handle,
    const char *tool_name,
    esp_agent_tool_param_t params[],
    size_t num_params,
    void *user_data,
    char **result)
{
    // Read the "name" parameter
    const char *name = NULL;
    for (size_t i = 0; i < num_params; i++) {
        if (strcmp(params[i].name, "name") == 0) {
            name = params[i].value.string_val;
        }
    }
    if (!name) name = "stranger";

    // Build the result
    asprintf(result, "{\"greeting\":\"Hello, %s!\"}", name);
    return ESP_OK;
}

2. Register the handler​

In app_tools_register():

app_agent_register_tool("say_hello", say_hello_handler, NULL);

3. Declare the tool to the agent​

The tool's JSON schema lives in the agent's configuration (in the Admin Dashboard, not the firmware). Add a local-type tool entry to your agent:

{
  "type": "local",
  "name": "say_hello",
  "description": "Greet the user by name.",
  "inputSchema": {
    "json": {
      "type": "object",
      "properties": {
        "name": { "type": "string", "description": "Name of the person to greet" }
      },
      "required": ["name"]
    }
  }
}

You declare this in Advanced mode of the agent editor (see Advanced Configuration). The name field must match the C string you registered.

4. Build and flash​

idf.py build flash monitor

5. Test​

In a chat with the agent: "Say hello to Alice." The agent calls say_hello with name="Alice"; the device returns {"greeting":"Hello, Alice!"}; the agent replies "Hello, Alice!".

Conventions​

  • Names are exact strings. A typo between the firmware and the agent config means the tool isn't found at runtime.
  • Return JSON strings. The agent treats the result as text, but most agents work better with structured JSON than free text.
  • Keep handlers fast. They block the agent's reply until they return. Anything > 100 ms degrades the conversation.
  • Don't access shared state without locks. Handlers run on a separate task from audio.
  • Validate input. The schema is checked server-side before the call, but defensive validation in the handler protects against unusual inputs.
  • Handle errors clearly. Return JSON with an error field — {"error": "device_unreachable", "message": "..."} — so the agent can explain the failure to the user.

Where the implementation lives​

PathTools
examples/common/app_common/src/app_common_tools.cset_volume, set_reminder, get_local_time (shared).
examples/voice_chat/main/app_tools.cVoice Chat–specific (set_emotion).
examples/matter_controller/main/app_tools.cMatter Controller–specific (get_device_list, control_device).
examples/matter_controller/main/controller_rest_apis/matter_device.cWhere control_device translates JSON → Matter commands.

Tips​

  • Look at existing tools first. Copying set_volume is faster than starting from scratch.
  • Iterate on the agent side first. A good tool description in the agent config matters as much as the firmware code.
  • Test in a fresh conversation after agent changes — the LLM caches tool descriptions per turn.
  • Watch the serial monitor during development. Tool calls and errors are logged.

On this page