Audio & Wake Word

How the firmware captures, transmits, and plays audio — and how it knows when you're talking to it.

Audio rates

Direction	Sample rate	Frame duration	Format
Mic → cloud (upload)	8 kHz	20 ms	PCM 16-bit
Cloud → speaker (download)	16 kHz	60 ms	PCM 16-bit

These defaults are tuned for low bandwidth and natural-sounding speech. They live in idf.py menuconfig → App Common Config:

• Upload sample rate (default 8000)
• Upload frame duration (default 20 ms)
• Download sample rate (default 16000)
• Download frame duration (default 60 ms)

If you change the device's rates, the cloud's voice model has to support them — not all combinations work. The defaults are the safe pick.

Acoustic Echo Cancellation (AEC)

AEC lets the device hear you even while it's speaking — the basis for natural barge-in.

It's off by default (CONFIG_ENABLE_AEC=n). Turn it on in idf.py menuconfig → Audio Config → ENABLE_AEC. The board has to support hardware AEC for this to be worth it; without hardware support, software AEC eats CPU and the result is poor.

Wake word

The firmware uses WakeNet9 — Espressif's on-device wake-word engine. Two model variants are loaded by default:

• WN9_HILEXIN — "Hi Lexin"
• WN9_HIESP — "Hi ESP"

Either trigger works. Most users say "Hi, ESP".

When the wake word fires:

1. The device plays a chime.
2. The mic capture stream opens.
3. The cloud starts transcribing.
4. After you finish speaking, the agent replies.
5. After ~15 seconds of silence post-reply, the device sleeps. Wake it again.

Switching wake words

In idf.py menuconfig → WakeNet → choose a different model. Espressif ships several pre-trained models. Custom wake words require training a new model (out of scope for these docs).

Tap-to-wake

You can also wake the device by tapping the screen. This bypasses the wake word entirely — useful in noisy rooms.

Barge-in

Barge-in lets you interrupt the agent. Two ways:

• Tap the screen while the agent is speaking. The agent stops; mic re-opens.
• Speak into the mic while the agent is speaking. With AEC enabled and a board that supports it, the device hears you over its own speaker; the firmware sends a barge_in message to the cloud, which cancels the in-flight reply.

Without AEC, voice barge-in is unreliable on most boards — touch barge-in always works.

Sleep behavior

The device sleeps after 15 seconds of inactivity post-conversation. While sleeping:

• Wake-word detection stays on (low-power mode).
• Touch detection stays on.
• Display dims.

The 15-second timeout is fixed in the current firmware. Adjust it by patching app_common if you need different behavior.

Tuning audio quality

Things that affect how well the device hears you:

Knob	Effect
Move closer	The cheapest fix. Most boards are tuned for 0.5–2 m.
Reduce background noise	The wake word handles modest noise; ASR (speech-to-text) suffers above ~60 dB SNR.
AEC on	Lets you barge in while the agent talks. Worth it on supported boards.
Higher upload rate	16 kHz upload (vs default 8 kHz) improves ASR for whispered or accented speech, at ~2× bandwidth.

Tuning the agent's voice

The voice the agent uses is determined by the Speech Model configured on the agent (see LLM). Different speech models have different voices, accents, and pacing. Try a few in the Admin Dashboard.

Common audio issues

Symptom	Likely cause
Wake word doesn't trigger	Mic is muted, blocked, or the room is too noisy. Try tap-to-wake to confirm everything else works.
Wake word triggers from agent's reply	AEC isn't on. Enable it.
Speaker is too quiet / too loud	set_volume tool, or change Default Playback Volume in menuconfig.
Choppy audio	Wi-Fi is unstable. idf.py monitor shows packet drops. Move closer to the AP.
High latency	Network round-trip to the gateway is slow. Try a closer hosting region in your deployment.