The Trick to Squeezing a 35B LLM on an iPhone

Running a 35 billion parameter large language model directly on an iPhone initially seems like a computational impossibility. A model of This 35 billion parameter scale, when saved as a normal 4-bit file, occupies roughly 20GB. Usually, this entire 20GB dataset must reside in RAM for efficient processing, a capacity far beyond any mobile device.

Yet, a groundbreaking method has shattered this barrier. Researchers demonstrated an MoE model with s 35 billion parameters running on an iPhone, achieving an active memory footprint of only 1.4GB. This represents a staggering reduction, cutting the typical memory requirement by over 90%.

The trick lies in how the model manages its vast data. Instead of loading the entire 20GB into RAM, only essential components and currently active "experts" are streamed on demand from the device's SSD. This innovative approach exploits the sparse activation patterns inherent in Mixture-of-Experts architectures.

This breakthrough signals a profound shift for artificial intelligence. We are moving towards powerful, private on-device AI capabilities, unburdening complex tasks from constant reliance on cloud servers. And this opens the door to a new generation of intelligent, localized applications.

Solving the memory crunch for This 35 billion parameter model required a fundamental shift in architecture: enter Mixture-of-Experts (MoE). Instead of one massive, monolithic model where every part is active for every token, MoE employs a team of smaller, specialized 'experts'. A dedicated 'router' network dynamically selects only a handful of these experts relevant to a given task.

This architecture proves ideal for resource-constrained devices like an iPhone. Usually, a 35 billion parameter model, as a normal 4-bit file, demands roughly 20GB of RAM. But with MoE, only selected experts load into active memory, leaving the vast majority dormant on slower storage like an SSD. This drastically reduces the active RAM footprint to a mere 1.4GB.

MoE is not just for shrinking models; it is also the key to scaling them to trillions of parameters efficiently. Now, it enables sophisticated LLMs to run directly on edge devices.

For the iPhone, the model is cleverly partitioned. Crucial shared components—embeddings, attention mechanisms, routers, and a shared expert—comprise the 1.4GB that resides permanently in RAM. Meanwhile, 40 individual expert files, each around 300MB and totaling 12GB, await on the SSD. For each token, after attention runs on the GPU, the router picks eight specific experts, which the engine then streams from the SSD into GPU memory, running them alongside the shared expert. This involves 320 small SSD reads for every single token.

This 35 billion parameter model sidesteps the 20GB memory barrier by segmenting its components. A core 1.4GB of essential elements load once into RAM, forming the model's ever-present operational backbone and remaining resident throughout inference:

• embeddings (input token representations)
• attention mechanisms (contextual understanding)
• routers (expert selection)
• a shared expert (a general-purpose baseline)

Usually, entire models reside in RAM, but this 35 billion parameter architecture innovates. Instead, the remaining 12GB of specialized experts reside on the iPhone’s fast SSD. This architecture enables streaming experts: the system fetches these large, inactive portions only when the model's router specifically calls for them, shunting them into GPU memory on-demand.

For every single token the model generates, a rapid-fire sequence of data transfers ensues. The router dynamically selects 8 experts within each of the 40 layers, prompting 320 distinct, tiny reads directly from the SSD. This constant, high-speed data shuttle allows the massive s 35 model to operate within the iPhone's tight memory constraints.

This 35 billion parameter MoE model running directly on an iPhone truly opens a new frontier for edge AI. Imagine deeply private assistants, processing sensitive user data entirely on-device, free from cloud transfer, or zero-latency creative tools generating intricate designs or text responses instantly. These advancements enable fully offline AI applications, transforming capabilities previously confined to distant data centers into tangible, personal realities.

But, this breakthrough isn't without its immediate challenges. The on-demand streaming engine, while ingenious, generates a heavy I/O load, necessitating 320 small reads from the SSD for every single token. This intensive, constant data access naturally increases battery consumption and demands robust thermal management from the compact device hardware.

However, these hurdles are temporary. The future promises a powerful synergy between innovative software techniques, like the current expert streaming engine, and increasingly specialized hardware acceleration. Apple's Neural Engine, for instance, offers dedicated silicon precisely designed for AI workloads. Combined with continuously optimized frameworks and memory architectures, this integrated approach will accelerate the journey towards mainstream on-device superintelligence, making powerful AI ubiquitous and seamlessly integrated into our daily lives.

What is a Mixture-of-Experts (MoE) model?

An MoE model is a neural network architecture that uses multiple specialized 'expert' sub-networks. For any given input, a routing mechanism selects only a few experts to process it, making inference much more efficient than dense models where the entire network is used.

How was the 35B model's memory footprint reduced so dramatically?

The model's resident RAM was cut to just 1.4GB by keeping only shared components in memory. The bulk of the model, the 'experts,' reside on the phone's fast SSD and are streamed into memory on-demand during inference.

What is 'streaming experts'?

It's a technique where parts of a large AI model (the 'experts' in an MoE architecture) are stored on slower SSD storage and loaded dynamically into faster RAM or GPU memory only when needed for a specific calculation, then discarded.

What are the benefits of running large AI models on a phone?

The primary benefits are enhanced privacy since data never leaves the device, lower latency with no network round-trip, full offline functionality, and reduced reliance on costly cloud infrastructure for inference.