Google’s 1,000 Token/Sec AI Is Here

Most large language models (LLMs) operate on an autoregressive principle, generating text one token at a time, left to right. This sequential process means the model writes a word, then evaluates everything written to predict the next. For commercial servers, this inefficiency is mitigated by batching hundreds of users, loading model weights once to serve 256 users simultaneously.

However, local LLM deployments face a significant bottleneck: they are memory-bound. A local GPU spends most of its operational time waiting for model weights to load from memory, not actively computing. It loads a massive portion of weights, performs a minute computation for one token, then idles before repeating the cycle for the next token, leaving expensive hardware largely underutilized.

Google DeepMind's DiffusionGemma introduces a radically different paradigm to overcome this. Instead of the traditional "one token for 256 users" approach, DiffusionGemma generates 256 tokens for a single user, all at once, by starting with a canvas of random placeholder tokens, or "noise." It then refines all positions simultaneously into coherent text, providing the GPU with a substantial computational load that transitions it from memory-bound to compute-bound, theoretically unlocking speeds beyond 1,000 tokens per second.

Instead of generating text sequentially, DiffusionGemma steals a trick from image AI: it begins with a canvas of random placeholder tokens, essentially "noise." Much like an image diffusion model refines noisy pixels into a coherent picture, DiffusionGemma iteratively transforms this textual noise into meaningful output over multiple bidirectional passes. This parallel processing allows the model to work on the entire output simultaneously, a radically different approach from the one-word-at-a-time generation.

Google DeepMind introduced Uniform State Diffusion to apply this concept to text. Here, randomly swapped-out words are considered "noise." During training, real words are replaced with random ones, and the model learns to identify and correct these corruptions. This method enables a crucial capability: the model can re-evaluate and modify any token on the canvas at any point in the generation process.

This contrasts sharply with simpler methods like Masked Diffusion, where tokens are merely blanked out. Masked Diffusion suffers from a significant limitation: once the model commits to a token, it becomes permanently locked in, similar to the rigid left-to-right generation of autoregressive models. Uniform State Diffusion overcomes this by always holding a token in every position, allowing the model to self-correct by swapping out even previously accepted words if they no longer fit the evolving context.

DiffusionGemma employs an innovative Encode-Denoise Patch architecture, built atop the existing 26 billion parameter Gemma 4 model. This design dynamically switches between two operational modes: an encoder mode to interpret the user's prompt, extracting context and guidance, and a denoiser mode to refine the text canvas. The encoder populates a KV-cache, passing crucial information directly to the denoiser.

During denoising, the model leverages bidirectional attention, allowing it to "see" and process all tokens on its "canvas" simultaneously, regardless of their position. Crucially, it retains all confidence scores (logits) for every token at each position throughout its multiple passes. This constant visibility and iterative refinement, where previous guesses inform subsequent corrections, are fundamental to its parallel processing capability. For a deeper dive into this architecture, see DiffusionGemma - Google DeepMind.

This architectural shift fundamentally reconfigures the computational bottleneck. Unlike autoregressive models, which are often memory-bound due to sequential token generation, DiffusionGemma keeps the GPU constantly active. By processing hundreds of tokens in parallel, the model flips from being memory-bound to compute-bound, unlocking the immense processing power of modern GPUs and achieving generation speeds exceeding 1,000 tokens per second.

Real-world deployment of DiffusionGemma reveals a compelling performance profile. Benchmarks conducted on an H100 GPU demonstrated impressive speeds, consistently achieving around 700 tokens per second. While this did not quite reach the theoretical 1,000+ tokens per second predicted for the architecture, it still represents a radical leap beyond the one-token-at-a-time pace of traditional autoregressive models.

This breakthrough in speed introduces a clear operational tradeoff. DiffusionGemma is engineered for scenarios demanding critical velocity, where rapid output outweighs the pursuit of absolute textual perfection. Conversely, standard autoregressive models, with their sequential generation and meticulous refinement, continue to serve as the preferred choice for tasks requiring maximum output quality and coherence.

Consequently, DiffusionGemma finds its ideal application in use cases where low latency is paramount. This includes tasks like intelligent code in-filling, where quick suggestions enhance developer workflow. It also excels in rapid creative iteration, allowing users to quickly explore numerous textual drafts. Furthermore, it revolutionizes non-linear generative tasks, enabling instant, multi-token responses that fundamentally change user interaction paradigms.

What is DiffusionGemma?

A new text generation model from Google DeepMind that uses diffusion techniques, similar to AI image generators, to produce text at very high speeds, potentially exceeding 1,000 tokens per second.

How is DiffusionGemma faster than traditional LLMs?

It generates hundreds of tokens at once in parallel "passes" rather than one-by-one (autoregressively). This flips the process from being memory-bound (waiting for data) to compute-bound (fully utilizing the GPU).

What is the main tradeoff with DiffusionGemma?

The primary tradeoff is speed for maximum quality. While incredibly fast, for tasks requiring the highest possible accuracy and coherence, standard autoregressive models are often still superior.

What is uniform state diffusion?

It's the core technique used to apply "noise" to text for training. Instead of just masking words, it replaces real words with random ones, allowing the model to learn to correct and even swap out its own previous guesses.