A voice assistant with no cloud escape hatch
Check this out: an ESP32-S3 microcontroller, unplugged from the internet, responds to your voice. In a recent demo, a tiny $10 chip heard spoken English or French commands and changed its onboard LED color. You press a button, speak, and the board processes your request locally—no Wi-Fi, no Bluetooth, no cloud connection.
The hardware is remarkably simple: a roughly $10 ESP32-S3 microcontroller with 16 MB of flash storage and 8 MB of memory. Add a small digital microphone, a push button to activate recording, and an RGB LED. The laptop shown in the demo merely displays output; all the heavy lifting, from listening to execution, happens directly on the board itself.
This local speech recognition matters because it bypasses common issues with cloud-based voice assistants. It eliminates network latency, ensures privacy by not sending your voice recordings to remote servers, and functions entirely offline. While this demo showcases a constrained command interface—like “flash the light purple”—it’s a powerful step towards fully independent voice control on tiny, affordable hardware, not a general-purpose smart speaker.
The model was bigger than the chip
Whistle, the speech-to-text model from Cactus Compute, presented an immediate challenge: its single model file weighed in at 16.9 MB. That’s already larger than the ESP32-S3’s entire 16 MB of flash storage, even before accounting for the device firmware. For context, OpenAI’s Whisper base model is a hefty 145 MB, making Whistle nearly nine times smaller, yet still too big for our tiny chip.
Despite its compact size, Whistle is remarkably capable. It supports seven languages:
- English
- French
- German
- Spanish
- Italian
- Dutch
- Polish
It automatically detects the language spoken, provides precise word timestamps, and can even generate audio embeddings directly from speech, offering more than just transcription.
The key to fitting Whistle on the chip lies in its unique "intelligence ladder" architecture. While the encoder, which processes the raw audio, always runs with all eight of its layers, the decoder, responsible for transcribing the text, can be pruned. Cactus Compute designed the decoder with selectable depths, meaning it still functions correctly even with fewer layers.
The creator opted for a four-layer decoder, which, after carefully extending the model partition to the very end of the flash chip, left a mere 98 KB of flash spare. A two-layer option would have fit more easily but introduced noticeably more transcription errors, proving the value of those extra layers.
How a tiny board turns sound into action
How does a tiny chip actually "hear" you? The process starts when the microphone captures your voice at 16 kHz. This raw audio transforms into a Mel spectrogram, a visual "heat map" of your voice's pitch and energy over time, sliced into 10-millisecond segments.
Next, a series of convolutional layers compresses this heat map. Each slice now represents about 80 milliseconds of audio, turning a two-second command into just 25 frames. This efficient representation is crucial for the resource-constrained ESP32-S3.
Then, Whistle's encoder takes over. With eight layers, it allows each 80-millisecond frame to "see" every other frame in the clip simultaneously. This means the model understands words in the context of your entire sentence, not just in isolation.
Finally, the decoder generates the transcript, one piece at a time. It first identifies the language (English or French, for example) using a language token. As it produces each word, it uses "cross-attention" to check back with the encoder’s output, ensuring consistency with what it heard.
A small parser, not Needle, then interprets the transcribed text, mapping words like "red" or "five seconds" to specific LED colors and blink patterns. For more on Whistle's capabilities, check out Whistle: Speech to Text in 16.9 MB - Cactus Compute.
Initial attempts to process a two-second command took nearly 24 seconds—far too slow for a voice assistant. Significant optimizations were needed. Cactus Compute engineers sped things up by processing multiple frames per pass, splitting the workload across both of the ESP32-S3's cores, and using 8-bit integer operations on the chip’s vector unit. These changes brought typical response times down to a much more usable 10 seconds.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
A compelling demo, not a dictation replacement
Accuracy claims for Whistle are impressive, but context is key. Cactus Compute reported about 78% success on synthetic English color commands, like “flash the light blue.” Adding recorded microphone noise to these tests caused little change, showing robustness in limited, controlled environments.
Practical findings highlight the model’s strengths with specific phrasing. Short phrases such as “make it red” or “turn it green” worked 82–93% of the time in tests. A lone color word, however, performed at a lower 44% accuracy. Keyword biasing and retrying with another language like French improved results for fixed command sets.
Whistle’s Apache 2.0 release and shared runtime design with Needle point toward a future of private, voice-to-tool systems on edge devices. However, the current ESP32-S3 board cannot fit both models simultaneously, as they would require roughly 24 MB of flash. Its roughly 10-second latency and focus on command accuracy also limit broader applications.
This compelling demo showcases a significant step for local voice control, not a dictation replacement. You can explore the project further in the GitHub repository and review the model card.
Frequently Asked Questions
What is Whistle?
Whistle is Cactus Compute’s compact speech-to-text model, designed to transcribe audio locally across seven languages.
Can Whistle run on an ESP32-S3?
Yes, with a custom build and a pruned decoder. The demonstrated board runs a four-layer version within its 16 MB flash limit.
Does the ESP32-S3 voice project need Wi-Fi?
No. Speech capture, transcription, and the LED command logic run on the board without Wi-Fi or Bluetooth.
How accurate and fast is the demo?
The creator reports roughly 78% accuracy on synthetic English color commands and about 10 seconds of processing per command on average.
Can Whistle run alongside Needle on this board?
Not in the demonstrated setup: the combined models would need about 24 MB, more than the board’s 16 MB of flash.