This Trick Runs a 35B LLM on Your iPhone

AI researchers recently achieved a remarkable feat: deploying the Qwen 3.5 large language model, a 35-billion parameter behemoth, directly onto a stock iPhone. This isn't merely a demonstration; the model delivers a usable 11 tokens per second, transforming a pocket-sized device into a powerful local AI engine.

Such a model typically demands immense memory. A 35B parameter model, even when quantized to 4-bit precision, requires approximately 20 gigabytes (GB) of RAM. This far exceeds the physical memory available on most mobile devices, which often cap at 8 GB, creating a fundamental barrier for on-device AI.

Overcoming this constraint hinges on the Mixture of Experts (MoE) architecture. Unlike traditional dense models where every parameter activates for every computation, MoE models are sparsely activated. For any given input, a 'router' component intelligently selects and activates only a small subset of specialized 'experts' within the model.

This sparse activation fundamentally alters memory management. Instead of loading the entire 20 GB model into RAM, only the currently active experts, alongside essential components like attention mechanisms, need to reside in active memory. This significantly shrinks the real-time memory footprint, allowing the bulk of the model to remain on slower, larger storage like the iPhone's SSD until needed.

Running a 35B parameter LLM on an iPhone's constrained memory requires a clever memory partitioning strategy. A compact 1.4GB core of essential model components remains resident in RAM, comprising:

• Embeddings
• Attention mechanisms
• Routers
• One shared expert

The remaining 12GB, consisting of the model's 40 expert files (each around 300MB), sits on the iPhone's high-speed SSD.

For every token generated, the system initiates a dynamic retrieval process. An internal router selects eight specific experts per layer from the model's 256 options, across all 40 layers. The engine then reads these chosen expert files directly from the SSD into GPU memory. This results in approximately 320 small reads per token, ensuring only the necessary parameters are loaded for immediate computation.

This on-demand expert streaming drastically reduces the active memory footprint. While the Qwen 3.5 model boasts 35 billion parameters, only about 3 billion are active for any single token. Such efficient resource allocation is key to operating a model of this magnitude on consumer hardware, leveraging the SSD as an extended, high-speed "virtual RAM" for the vast majority of inactive parameters.

Developers initially implemented a substantial 9.8GB custom cache within their application to manage expert data. Counter-intuitively, deleting this bespoke system and instead relying on the native iOS page cache delivered a remarkable 38% performance boost. This unexpected outcome highlights a fundamental principle of operating system design.

Allocating such a large, fixed block of RAM for the app’s exclusive cache proved detrimental. This strategy inadvertently starved two critical system resources: the GPU, which requires significant memory for computation, and the operating system's own sophisticated, dynamic caching mechanisms. The app's explicit memory reservation essentially competed with, rather than complemented, the OS.

The iOS page cache emerged as the unsung hero, dynamically managing spare RAM. It intelligently retains frequently accessed experts in memory, anticipating future needs. This system is far more efficient, adapting to overall system demands and ensuring optimal memory allocation across all processes without requiring the app to pre-allocate.

Meeting the demanding read-speed requirements of over 5 GB/s is critical for sustaining 11 tokens per second. The iPhone's physical flash storage, however, peaks at roughly 1.6 GB/s. Consequently, the operating system orchestrates the majority of expert reads directly from RAM via its page cache, a critical factor for models like the Qwen 3.5 A3B variant. Learn more about its architecture at Qwen3.5-35B-A3B - ModelScope.

Achieving this iPhone-native performance also required a nuanced approach to model compression: tiered quantization. Developers observed that roughly 25% of the Mixture of Experts (MoE) model’s 'experts' handled about 80% of the work. These frequently accessed "hot" experts maintain a higher-quality 4-bit precision, preserving critical information.

In contrast, the remaining, less-used "cold" experts undergo a deeper compression, reducing their size to just 2-bit. This intelligent differential treatment reduces the model's overall storage footprint by a substantial 34%, shrinking it from an initial 19GB down to 13GB.

This significant reduction in disk size directly impacts performance, allowing more of the Qwen 3.5 model's weights to reside within the efficient iOS page cache. Fitting more data into fast memory contributes substantially to the observed 11 tokens per second generation speed, minimizing reliance on slower SSD reads.

Such intense, sustained computation on a compact mobile device inevitably generates substantial heat; users report the iPhone becoming noticeably hot, even "melting through my hand" according to one developer. This highlights the practical thermal challenges of pushing AI inference to its limits on consumer hardware.

Furthermore, developers encountered a critical bug where the model would get stuck in an endless loop, repeating tokens after a few words. The async_pread_wait function, responsible for reading expert data from storage, required an urgent fix. It initially failed to correctly validate reads for the half-sized 2-bit "cold" experts, silently skipping them and effectively running the model with incomplete information. Correcting this oversight was crucial for stable, coherent output.

What is the core technique that allows a 35B model to run on an iPhone?

The technique leverages a Mixture of Experts (MoE) model architecture. Instead of loading the entire 20GB model into RAM, only a small 1.4GB core stays resident while the rest of the model's 'experts' are streamed from the phone's fast SSD storage on-demand as they are needed for each token generation.

What model was used in the demonstration?

The demonstration used Qwen 3.5, a 35 billion parameter Mixture of Experts model. Specifically, it's the A3B variant, which means only about 3 billion parameters are active for any single token.

What is a Mixture of Experts (MoE) model?

An MoE is a neural network architecture where the model is composed of many smaller 'expert' networks. For any given input, a routing mechanism selects a small subset of these experts to process it. This sparse activation makes them highly efficient for inference.

What kind of performance was achieved on the iPhone?

The setup achieved a generation speed of 11 tokens per second. However, this level of processing caused the iPhone to get noticeably hot very quickly.