How It Works
ESP Private Agents has four moving parts. This page shows how they fit together.
The big picture
Everything in the AWS Cloud box runs in your AWS account when you self-deploy. (If you use Espressif's hosted platform, it's in Espressif's account instead.)
The four pieces
1. ESP Device (firmware)
The hardware. Captures your voice with a microphone, plays the agent's reply through a speaker, and runs local tools like volume control, reminders, or Matter commands. Source: esp-agents-firmware.
→ See Firmware
2. Mobile App (ESP RainMaker Home)
How you set up devices. Pairs over Bluetooth, pushes Wi-Fi credentials, assigns agents via QR scan from the Admin Dashboard.
→ See Mobile App
3. Web Dashboards
Two browser-based UIs:
- Admin Dashboard (
/) — admins create and configure agents. - User Dashboard (
/try/) — end users chat with agents from the browser.
→ See Admin Guide and End User Guide
4. Backend (FastAPI on ECS Fargate)
The brain. Requests enter through an Application Load Balancer (ALB), which terminates TLS and spreads every REST and WebSocket request across one or more identical FastAPI tasks running on ECS Fargate. The task count auto-scales with load, and each task's container image is pulled from ECR.
Every task handles:
- Authenticating users (validating Cognito JWTs).
- Routing chats to the right LLM.
- Executing server-side tools (cloud + remote MCP).
- Pulling chunks from Knowledge Bases.
- Streaming audio to and from voice models.
- Tracking usage and quotas.
It talks to the device over a single WebSocket for both text and voice — the ALB keeps these long-lived connections open, and each connection is pinned to one backend task for its lifetime.
Elsewhere in these docs (and in the codebase) this FastAPI service is called "the gateway" — it's the one service every client talks to. Architecturally it is the backend server behind the ALB.
→ See WebSocket Protocol and API Reference
What flows where
A typical voice turn
Voice paths use Amazon Nova Sonic — a single bidirectional speech model that does ASR, reasoning, and TTS in one pass. Text agents use a regular text LLM (Claude, GPT, Gemini, etc.) — same flow, no audio streams.
Setting up a new device
Where data lives
| Data | Where it's stored | Encrypted with |
|---|---|---|
| Conversation messages | DynamoDB | AWS-managed |
| User profile (email, mobile) | DynamoDB | KMS (your key) |
| OAuth tokens for connectors | DynamoDB | KMS (your key) |
| File attachments (images, PDFs) | S3 | S3-managed |
| Knowledge Base source documents | S3 | S3-managed |
| Knowledge Base vectors | Bedrock-managed vector store | Bedrock-managed |
| Agent configurations | DynamoDB | AWS-managed |
| Usage counters | DynamoDB | AWS-managed |
Everything stays in one AWS account — yours, if you self-deploy.
Where to go next
Pick the role that fits you:
| You are… | Read |
|---|---|
| An end user with a paired device | Quick Start → User Guide |
| An admin building agents | Admin Guide |
| A maker flashing or modifying firmware | Firmware |
| A developer building a custom client | WebSocket Protocol and API Reference |