Your Watch is Now an AI Brain

A 90 million parameter Falcon H1 model recently ran completely offline on a 2020 Apple Watch Series 6, fundamentally shifting our understanding of on-device AI. This demonstration, conducted on a wearable device released four years ago, proves that sophisticated artificial intelligence can operate independently of the cloud, directly from your wrist. It represents a real step towards ubiquitous, private AI.

The performance metrics achieved were genuinely startling, particularly for a device of this form factor. The Apple Watch Series 6 generated text at an average rate of 15 tokens per second, a remarkably fluid pace for local inference. This speed constitutes a staggering 50x performance advantage over a first-generation Raspberry Pi, which previously crawled at just 0.3 tokens per second when running a similar AI model.

This massive performance chasm directly reflects the Series 6's significantly more robust hardware specifications compared to its low-power counterpart. The Apple Watch Series 6 integrates a powerful dual-core 1.8 GHz CPU and a substantial 1GB of RAM. These specs easily overpower Raspberry Pi 1’s more modest single 700 MHz core and 512 MB of RAM, decisively explaining the dramatic processing power disparity and the "performance shocker" witnessed.

Falcon H1’s remarkable efficiency stems from its hybrid architecture. It uniquely combines standard attention layers—the workhorses of most large language models—with a novel state space model known as Mamba 2. This dual-component design is key to its compact performance on constrained hardware like an Apple Watch.

Mamba 2 excels in memory efficiency. Instead of accumulating a large, growing key-value cache like traditional attention, Mamba 2 operates with a fixed-size memory state. It updates this state token-by-token, dynamically deciding what information to keep or override. This "scan" process ensures that, whether processing one token or a thousand, the memory footprint remains constant, making it exceptionally suited for devices with limited RAM.

This architectural choice explains why Apple’s own Core ML framework fails to support Falcon H1. Core ML requires a fixed, pre-defined computation graph, optimizing for static operations. Mamba 2’s dynamic, sequential "scan" structure, where each step’s computation relies on the previous state, creates a looping dependency that Core ML cannot natively handle. The framework simply lacks a building block for this type of state space computation.

Bypassing Apple’s native frameworks, the project leveraged a custom-compiled version of Llama.cpp. This popular inference library typically supports iOS and macOS, but WatchOS required specific build flags and a one-line source guard to enable compilation. Core ML, Apple's on-device ML framework, proved unsuitable for Falcon H1 due to its hybrid Mamba 2 architecture, which lacks direct Core ML building blocks.

Crucial to this WatchOS deployment was targeting the ARM64_32 architecture. Despite its name, this isn't a slower, 32-bit instruction set. Instead, it utilizes the full 64-bit ARM instruction set, including the Neon vector unit for batching math operations, ensuring no performance degradation in computational speed.

WatchOS employs 32-bit memory pointers. On a standard 64-bit system, memory addresses are 64 bits long, but on the watch, they are half that size. This design conserves RAM across the system, a critical optimization for devices with limited memory.

However, this 32-bit addressing scheme imposes a hard 2GB RAM ceiling. This memory limitation prevents running multi-billion parameter models, as they simply cannot address enough memory. Yet, Falcon H1, at a mere 57MB, fits perfectly within this constraint, demonstrating the viability of compact models on resource-constrained wearables. For further details on the device's hardware, consult the Apple Watch Series 6 - Technical Specifications.

Proof-of-concept app on the Apple Watch Series 6 transcended a mere technical stunt, delivering a fully functional AI assistant. It robustly integrated native voice input and demonstrated practical tool use, allowing users to query Wikipedia for facts or check local weather. This proved the model's utility far beyond a simple benchmark, answering prompts like "What is the capital of France?" almost instantly.

While short, direct queries often yielded impressive speeds—up to 24 tokens per second with the Falcon H1 model—the device’s inherent limitations became evident with more complex requests. Generating longer, multi-paragraph responses noticeably degraded performance. This exposed the memory and processing constraints of the 2020 hardware, specifically the ARM64\_32 architecture's approximate 2 GB memory addressing ceiling.

This successful, albeit constrained, deployment on years-old consumer hardware poses a fundamental question for device manufacturers, particularly Apple. If a hobbyist project can unlock significant, real-time local LLM capabilities on an Apple Watch, why are general-purpose, on-device AI models not yet a standard, integrated feature across new devices? The technical feasibility now appears undeniable.

What LLM was used on the Apple Watch?

A 90 million parameter model called Falcon H1 was used. It's small and highly efficient due to its hybrid architecture combining traditional attention layers with a Mamba 2 state space model.

How fast did the LLM run on the Apple Watch?

The Falcon H1 model ran at an average of 15 tokens per second, with speeds reaching up to 24 tokens/sec for simple queries. This is about 50 times faster than running a similar model on a Raspberry Pi 1.

Why can't Apple's Core ML run the Falcon H1 model?

Apple's Core ML framework doesn't natively support the state space model (Mamba 2) component of Falcon H1's architecture. Core ML expects a fixed computation graph, which conflicts with Mamba's sequential, looping nature.

What is ARM64_32 on the Apple Watch?

It's a variant of the ARM64 architecture where memory addresses are 32-bit instead of 64-bit. This saves memory on a device with limited RAM but also creates a hard 2GB ceiling, making it unsuitable for very large models. The core 64-bit instruction set remains fully intact.