This Tool Runs Datacenter AI on Your PC

Giant AI models, especially mixture of experts (MoE) models, present a unique challenge for consumer GPUs. Consider DeepSeek V4-Flash: it boasts a staggering 284 billion total parameters. Yet, for any single token, only a tiny fraction—around 13 billion—is actively used. While this active portion could fit on many GPUs, the entire 284 billion parameters must reside somewhere, typically system RAM, because you don't know which experts will be needed next.

Traditional inference engines like Ollama struggle with this dynamic reality. When an MoE model exceeds your GPU's VRAM—say, a 38GB Qwen 3.6 35B model on a 32GB card—Ollama resorts to a static solution. It permanently splits the model, offloading layers to the CPU (e.g., 30% to CPU, 70% to GPU). This means every single token, regardless of which experts it needs, must traverse every layer, forcing a slow detour through the CPU for a significant portion. In tests, this yielded a mere 58 tokens per second.

FreeToken offers a fundamentally different approach. It abandons static placement, re-framing the problem as dynamic scheduling. Your GPU's VRAM becomes an intelligent cache for frequently accessed experts, with the full model residing in system RAM; only actively needed experts stream to the GPU on demand. This dynamic, cache-based system is a much better fit for MoE models, which often reuse the same experts across tokens, keeping computation on the faster GPU.

FreeToken rethinks how your PC handles massive mixture of experts models. Instead of trying to cram the entire 284 billion parameters of DeepSeek V4-Flash onto your GPU, FreeToken treats your GPU VRAM as a high-speed cache. The full model lives in system RAM, with only the most recently used experts dynamically pulled into VRAM. Since models often reuse experts across tokens, this strategy dramatically reduces memory traffic.

When you first send a prompt (the "prefill" phase), models typically need to access many experts, which can slow things down. FreeToken employs double buffering to hide this latency. While your GPU processes the current layer, the tool simultaneously streams the next layer's experts from system RAM over PCIe, keeping the GPU busy and avoiding idle waits.

During actual generation, a cache miss happens when an expert isn't on the GPU. FreeToken's clever Q\* policy then kicks in. It automatically measures your PC's specific PCIe and RAM speeds, deciding on-the-fly whether it's faster to pull the missing expert to the GPU or compute it directly on the CPU. This dynamic split optimizes for your unique hardware, whether you have a narrow PCIe link or fast RAM.

So, did FreeToken actually deliver on its promises? The video put it to a proper real-world test, pitting it against Ollama with a challenging scenario: a 38GB mixture of experts model on a consumer GPU equipped with only 32GB of VRAM. This setup is specifically designed to break traditional inference engines, forcing them to offload a significant portion of the model to slower system RAM.

In this situation, Ollama splits the model by layer, pushing about 30% onto the CPU. This means every single token takes a detour through the slow CPU for certain layers, causing significant bottlenecks. As a result, Ollama crawled through the task, taking over 14 minutes and 20 seconds at a sluggish 58 tokens per second.

FreeToken, in contrast, completed the exact same task in a swift 4 minutes and 40 seconds, delivering an impressive 132 tokens per second. That's a nearly 3x speedup, making a significant difference for anyone running larger models locally where the model size exceeds GPU VRAM.

FreeToken's clever caching strategy also got a live demonstration. The tool allows you to resize its GPU expert cache on the fly, visually proving that a small slice of 'hot' experts does most of the heavy lifting. Researchers observed that dropping the cache from 58% down to 40% of the total model size only resulted in a 7% speed reduction. This validates the core idea: often just 20-40% of the model's experts are actively used, confirming the efficiency of FreeToken's dynamic caching method. For deeper dives into the technical implementation, you can explore the project's GitHub - FlashML-org/FreeToken: FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently..

FreeToken isn't a magic wand for every AI task. Its impressive speed gains appear only when your mixture of experts model exceeds your GPU's VRAM capacity. Remember that crucial video test: a 38GB model on a 32GB GPU, a scenario built to break traditional inference engines.

But for smaller models, the story changes. A 4-bit quantized model, for instance, that comfortably fits within your 32GB VRAM, actually runs faster on Ollama. The video demonstrated 240 tokens per second with Ollama versus FreeToken's 225 tokens per second in this case. FreeToken's sophisticated streaming architecture, designed for VRAM overloads, becomes pure overhead when there’s nothing left to stream from system RAM.

So, here’s the straightforward advice: if your model fits entirely on your GPU, stick with Ollama or Llama.cpp for optimal performance. Only when you want to run those massive, datacenter-scale mixture of experts models that otherwise wouldn't fit your consumer hardware, should you switch to FreeToken. It unlocks and dramatically accelerates those previously unreachable models.

What is FreeToken?

FreeToken is an open-source inference engine designed specifically to run large Mixture of Experts (MoE) AI models efficiently on consumer-grade hardware, serving as a high-performance alternative to tools like Ollama or Llama.cpp for this specific task.

How does FreeToken make AI models faster?

Instead of permanently splitting model layers between CPU and GPU, FreeToken treats GPU VRAM as a dynamic cache for the most-used 'experts'. It uses intelligent scheduling, double buffering, and adaptive CPU-GPU co-execution to minimize data transfer latency, especially when the model is larger than the available VRAM.

Is FreeToken always faster than Ollama?

No. FreeToken's primary advantage is when an MoE model is too large to fit in your GPU's VRAM. If a model fits comfortably in VRAM, Ollama or Llama.cpp will likely be faster because FreeToken's streaming architecture introduces unnecessary overhead.

What kinds of models does FreeToken specialize in?

FreeToken is built specifically for Mixture of Experts (MoE) models, such as DeepSeek-V4-Flash, Qwen3.6-35B, and GLM-5.2. Its architecture is optimized to handle the dynamic activation of 'experts' unique to these models.