The Engine That Unlocks 744B Models

Running a frontier-class AI model like GLM-5.2 on your laptop sounds impossible, and for good reason. This 744-billion-parameter behemoth is designed for data centers, demanding hundreds of gigabytes of storage just for its weights. That’s far more RAM than any consumer machine, even high-end ones, can offer.

So, how do you fit something that simply doesn’t fit? The secret lies in GLM-5.2’s clever Mixture of Experts (MoE) architecture. Instead of one monolithic neural network, it’s structured as thousands of specialized, smaller ‘expert’ sub-networks, each tailored for different tasks or data types.

This design creates a massive optimization opportunity. For any specific task or input, the model doesn’t engage all 744 billion parameters. It smartly routes the request to only a tiny fraction of these experts—about 40 billion parameters, roughly 5% of the total. This sparsity means the active memory footprint at any given instant is dramatically smaller, even if the full model is enormous. This is the gap a tool like Colibrì exploits to bring such models to your desk.

Colibrì's real trick isn't just throwing more GPU at the problem. It cleverly treats your machine's hardware as a three-tier memory hierarchy: your GPU's VRAM, your computer's system RAM, and your speedy NVMe SSD. This layered approach is crucial for handling models like GLM-5.2 locally.

The core of the model, those small, dense components used for every token (around 17 billion parameters), stay permanently in your system's RAM. These are the always-on parts, consuming less than 10GB at int4 quantization, ensuring essential operations are fast and responsive.

The thousands of massive Mixture of Experts (MoE) networks, which make up the bulk of the 744-billion-parameter GLM-5.2, reside on your SSD. This is where the model's huge size is managed without overwhelming your machine's limited RAM.

When GLM-5.2 needs a specific expert for a particular token, Colibrì doesn't load the entire model. Instead, it streams just that necessary piece directly from the disk into memory, uses it, and then caches it. This on-demand loading and caching is a game-changer for efficiency.

And Colibrì achieves this with remarkable efficiency. Vincenzo Fornaro built the entire engine in pure C with zero external dependencies. This lean implementation means less overhead, faster execution, and a truly portable tool capable of running these massive models where they simply shouldn't fit.

On a MacBook M2 Max with 32GB RAM, running GLM-5.2 from a slow external SSD proved painfully slow. The model crawled along at roughly 0.1 tokens per second. Profiling revealed the stark truth: over 80% of the processing time was spent simply waiting for the disk to deliver necessary model components. The actual computation, even on the M2 Max, took a mere 7 seconds for a full turn, dwarfed by over two minutes of disk latency.

Switching to a beefier workstation with an RTX 5090 GPU and 64GB RAM, the difference was immediate. Storing the 357GB model on a fast internal NVMe SSD, performance jumped eightfold to 0.8 tokens per second. The first word appeared in just 17 seconds, a vast improvement over the MacBook's two-minute wait. Disk wait time significantly dropped to 48%, meaning the machine spent nearly half its time computing rather than waiting.

This stark contrast exposes the real bottleneck for running 744B Mixture of Experts models with Colibrì: not raw GPU power, but RAM capacity and disk speed. The workstation's powerful RTX 5090 GPU saw minimal utilization; the true limitation was the constant need to stream model experts from storage. For deeper technical insights, including Colibrì's lean C implementation, check its GitHub - JustVugg/Colibrì: Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. page. Optimizing these memory tiers is key to unlocking immense models locally.

Experiments confirm a surprising truth for massive local AI models: system RAM now matters more than your high-end GPU. Our tests with the 744-billion-parameter GLM-5.2 showed that a powerful M2 Max chip on a MacBook, even with 32GB RAM, spent over 80% of its time waiting for disk, crippled by insufficient memory.

Even with 32GB, the model couldn't fully reside in memory, forcing constant, slow fetches from the external SSD. In contrast, the workstation’s 64GB RAM and fast internal drive drastically cut that wait time to 48%, significantly boosting performance to 0.8 tokens/second.

This improvement hinges on Colibrì’s clever learning cache. With more RAM, this cache can keep a larger pool of frequently used Mixture of Experts components readily available. This turns agonizing disk-wait time into actual compute time, directly leveraging the model’s sparse architecture where only ~5% of parameters are active per token.

While the dream of running frontier models like GLM-5.2 at blazing speeds on every laptop isn't here just yet, Colibrì undeniably proves it's possible. The future of powerful local AI isn't solely about brute-force chip speed; it's about intelligent memory management and making every gigabyte of RAM count.

What is Colibri?

Colibri is a lean, open-source inference engine written in pure C that allows massive Mixture-of-Experts (MoE) language models, like the 744B parameter GLM-5.2, to run on consumer-grade hardware with limited RAM.

How does Colibri run such a large model on normal hardware?

It exploits the MoE architecture, where only a fraction of the model is active per token. Colibri keeps the small, dense parts of the model in RAM and streams the larger 'expert' networks from your NVMe SSD on demand, treating your storage as an extension of your memory.

What hardware do I need to use Colibri with GLM-5.2?

You need a fast NVMe SSD with at least 360 GB of free space for the model, plus a minimum of 25 GB of system RAM. Performance scales significantly with more RAM and faster disk I/O speeds.

Is Colibri as fast as using a cloud-based AI model?

No. While it makes local execution possible, it's not a direct replacement for the speed of cloud-based models. Performance can be slow, especially on initial runs, with speed heavily dependent on your RAM and SSD.

How is Colibri different from tools like Ollama or llama.cpp?

Colibri is specifically designed for huge MoE models and uses a sophisticated three-tier memory system (VRAM, RAM, SSD) to handle models an order of magnitude larger than what tools like Ollama typically support on consumer hardware.