This $8 Chip Runs a 29M-Param LLM

Ukrainian developer Slava S. has achieved an extraordinary feat, successfully running a 28.9 million parameter language model on an $8 ESP32-S3 microcontroller. This groundbreaking system operates entirely offline, generating text locally without any Wi-Fi or server connection. This demonstrates a remarkable leap in embedded AI capabilities, making powerful language processing accessible on minimal hardware, far from cloud infrastructure.

This accomplishment dramatically redefines what's possible for microcontrollers. Previously, the largest language models successfully deployed on such chips topped out at approximately 260,000 parameters. Slava S.'s innovation scales this by an astounding 110x increase, fundamentally reshaping the landscape for on-device AI and proving that severe size limitations are not always insurmountable.

The raw specifications of the ESP32-S3 make this achievement even more remarkable. The chip offers a mere 512 kilobytes of SRAM—a memory capacity less than a typical personal computer from the 1990s. Despite this profound constraint, the system generates text at a respectable nine tokens per second, showcasing how meticulous engineering can overcome severe hardware limitations to enable sophisticated AI inference in ultra-low-power, disconnected environments.

Ukrainian developer Slava S. leveraged a clever workaround, inspired by Google's Gemma models. This technique, dubbed per-layer embeddings, rethinks how large language model parameters function. The core insight recognizes that most of an LLM's parameters reside in a static lookup table—a vast dictionary of word representations—rather than actively participating in real-time computation.

This fundamental distinction enables a radical memory strategy. Instead of attempting to cram the entire model into the ESP32-S3's minuscule 512KB of fast SRAM, the massive 25-million-row embedding table, weighing in at approximately 15MB, is strategically relocated. It finds its home in the chip's significantly larger 16MB of slower, cheaper flash memory.

During inference, the system dynamically retrieves only the handful of embedding rows absolutely required for the current token. For instance, just six rows—totaling roughly 450 bytes—are pulled from flash memory into SRAM. This ingenious approach leaves the limited, high-speed SRAM predominantly free for the model's actual computational work: the attention heads and feed-forward layers that process and generate the next token.

Slava S.'s achievement, while groundbreaking, does not yield a general-purpose large language model. This 28.9 million parameter model is exclusively trained on Microsoft's TinyStories dataset. This curated collection features simple, coherent narratives, specifically designed for small models—even those with just a few million parameters—to learn coherent text generation without producing the typical gibberish one would expect from a model of this size on a broader dataset.

Significant limitations emerge during interaction. The model consistently defaults to a specific story about a little girl, irrespective of the user's initial prompt. For instance, even inputs like "a story about a robot" or a "Star Wars" themed opening phrase quickly redirect to this default narrative after a few sentences, demonstrating an extremely narrow understanding and an inability to sustain novel conversational threads. This behavior highlights the model's constrained learned patterns.

Moreover, dynamic interaction is not possible. Adjusting the prompt necessitates a complete reflash of the ESP32-S3 microcontroller, a process that overwrites the device's memory. The system can only execute one pre-baked prompt at a time, making it a compelling, albeit constrained, proof of concept rather than a versatile, dynamic tool for arbitrary text generation. For those interested in exploring the underlying code and methodology, consult the project's GitHub repository: Running a 28.9M parameter LLM on an $8 microcontroller - GitHub.

This impressive project builds directly upon Andrej Karpathy's seminal 'llama2.c' work. Karpathy demonstrated that efficient Large Language Model inference was achievable in pure, portable C code, completely eschewing Python or complex dependencies. His blueprint proved that LLMs could run on minimal systems, a foundational insight for this ESP32-S3 implementation, enabling the entire concept.

Replicating this low-cost LLM requires specific hardware, a non-negotiable component for success. An ESP32-S3 N16R8 variant is mandatory, selected for its generous memory configuration. This particular model features 16MB of flash and 8MB of PSRAM. The trained 15MB model file itself crucially demands this larger flash capacity; boards with less than 16MB flash, such as common 4MB or 8MB versions, simply cannot store the entire model.

Slava S.'s original GitHub repository hosts all the necessary code for those keen to explore the implementation details and the underlying mechanics. For a more direct path to deployment, the video creator provides a simplified one-shot script, significantly streamlining the setup. This comprehensive script automates the entire build process, handling toolchain installation, data preparation, model training on the TinyStories dataset, and finally flashing the ESP32-S3 with the compiled model.

How does a 28.9M parameter LLM fit on a microcontroller with only 512KB of SRAM?

The model uses a memory hack where the largest part—a 25M parameter embedding table—is stored in the chip's larger, slower 16MB flash memory. Only the small compute portion and the specific embedding rows needed for the current token are loaded into the fast 512KB SRAM.

What specific hardware is required to replicate this project?

You need an ESP32-S3 microcontroller variant with at least 16 megabytes of flash memory and 8 megabytes of PSRAM. This is commonly known as the N16R8 variant.

What are the limitations of the LLM running on the ESP32?

It's a very simple model trained on the TinyStories dataset, so it can only generate simple narratives and cannot perform complex tasks. It also tends to drift back to a default story and requires reflashing the entire chip to change the initial prompt.

What open-source projects made this possible?

This project, created by developer Slava S., builds upon Andrej Karpathy's llama2.c project, which demonstrated running LLM inference in plain C. The model itself was trained on the TinyStories dataset developed by Microsoft Research.