AI That Gives Orders, Not Answers
Cactus Compute has unveiled Needle 2, a groundbreaking 45 million parameter AI model. This compact 14 MB model operates entirely offline on a $10 ESP32-S3 microcontroller, pushing the boundaries of what is possible for on-device intelligence without cloud dependency. Open-sourced under the Apache 2.0 license, Needle 2 demonstrates a new paradigm for highly constrained edge AI.
Crucially, Needle 2 functions as a specialized tool call dispatcher. Unlike general-purpose chatbots, it interprets natural language commands to execute predefined device functions. For instance, you can tell a smart device to "flash a red light for three seconds," and Needle 2 translates that into the precise sequence of actions required to control an LED, rotate a servo, or manage complex mobile or smart home operations.
This remarkable efficiency stems from a deliberate, critical trade-off. Cactus Compute sacrificed broad general knowledge and conversational ability; Needle 2 cannot answer factual questions like "What is the capital of France?" Instead, its entire training data focuses exclusively on enabling device actions. This makes Needle 2 an expert in direct action and control, not information retrieval, enabling its tiny footprint and high efficiency for dedicated device intelligence.
How Cactus Hacked LLM Architecture
Cactus Compute engineered Needle 2 with novel architectural choices to achieve its compact size and efficiency. A primary innovation involves replacing traditional learnable weights, which normally demand extensive matrix multiplications for every token, with hashed n-gram lookup tables. This design allows the model to retrieve knowledge directly from memory, instead of executing computationally expensive calculations, drastically reducing Floating Point Operations (FLOPs) to just 70 MFLOPs/token compared to 460 MFLOPs for larger models.
Further optimizing the network, Cactus swapped standard MLP (Multi-Layer Perceptron) layers for a Hadamard transform. Traditional MLP layers perform network mixing using huge, learnable matrices, consuming a large portion of a model's parameter budget for learning these connections. The Hadamard transform, a fixed mathematical operation, provides this essential mixing "for free," requiring almost no learnable parameters and thus saving millions of parameters.
The final piece of Needle 2's efficiency puzzle is its training-native 2-bit quantization, leveraging proprietary Cactus Quants. Many small models suffer significant performance degradation when quantized post-training, as they weren't designed for such aggressive compression. Needle 2, however, was explicitly trained against its own 2-bit compression from inception, ensuring optimal performance and eliminating quality loss at this ultra-low precision. This means the deployed 2-bit model is identical to the one trained.
Beating Giants Five Times Its Size
Needle 2 challenges the notion that larger models inherently mean superior performance. Cactus Compute's 45-million-parameter model, running at 2-bit precision, goes head-to-head with models like LFM2.5 230M—five to seven times its size—even beating them in specific, harder tool-calling tests. This performance is particularly striking given LFM2.5 operates at 16-bit precision.
Consider its specialized domain: on the Mobile Actions benchmark, Needle 2 achieved a remarkable 98.3% accuracy in picking the correct function name. This figure surpasses every single competing model it was compared against in that particular task, demonstrating its exceptional aptitude for precise command interpretation. For those interested in its open-source implementation and further details, the project is available on GitHub - cactus-compute/needle: 14MB foundation model for tiny devices; phones, wearables, smart home, and robots..
The efficiency gains are equally impressive. Needle 2 uses drastically fewer Floating Point Operations (FLOPs) per token than its larger counterparts. While models like LFM2.5 230M require around 460 MFLOPs/token, Needle 2 operates with just 70 MFLOPs/token. This represents a 7x to 85x reduction in computational cost, delivering competitive or even superior results in its niche.
This efficiency allows sophisticated AI to run on a $10 chip, making advanced agentic capabilities accessible for power-constrained edge devices where larger, more resource-intensive models simply cannot operate.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
From Theory to a Flashing Light
Witnessing Needle 2 in action on a bare ESP32-S3 microcontroller reveals its practical prowess. Users issue commands like "shine a purple light for seven seconds," and the model translates these into precise tool calls, triggering physical actions with a built-in confidence score. While this reasoning and execution takes approximately 40 seconds on the tiny, GPU-less chip, the capability itself is transformative.
This performance scales dramatically on more capable hardware. Needle 2 achieves up to 500 tokens/second on a Raspberry Pi 5, making it highly viable for real-time applications such as robotics and interactive devices. On VR devices, it can reach 400-1,500 tokens/second, and on budget smartphones, 300-700 tokens/second, demonstrating its adaptability across various edge platforms.
Needle 2 now paves the way for a new class of intelligent, low-cost products. Its ability to operate completely offline as an agentic tool call dispatcher benefits:
- IoT devices
- Wearables, like the Pebble Index 01 ring
- Smart home systems
This enables sophisticated, privacy-preserving AI directly within the devices we use daily, without cloud dependency.
Frequently Asked Questions
What is Needle 2?
Needle 2 is a 45-million-parameter, 14MB agentic LLM created by Cactus Compute. It's specifically designed to run on low-power edge devices like microcontrollers for tool-calling and function-dispatching tasks.
Can Needle 2 work as a general-purpose chatbot?
No. Needle 2 was not trained on general knowledge or conversation. It is highly specialized to interpret natural language commands and execute predefined device functions, making it a tool call dispatcher, not a chatbot.
What makes Needle 2 so small and efficient?
Its efficiency comes from a novel architecture using hashed n-gram lookup tables instead of weights, a Hadamard transform MLP that requires almost no learnable parameters, and a specialized 2-bit quantization applied from the start of training.
What kind of hardware can run Needle 2?
It can run on highly constrained hardware without a GPU or NPU, including the ESP32-S3 microcontroller, Raspberry Pi 5, VR headsets like the Meta Quest, and budget smartphones.