Attention’s quadratic bill—and a radical escape route
Today’s large language models, from GPT to Gemini, rely on the Transformer architecture’s attention mechanism. This system allows every token in a sequence to compare itself to every other token, creating an L-by-L matrix of interactions. While powerful, this global comparison incurs a computational and memory cost that grows quadratically with sequence length, posing a significant scaling challenge.
A recent paper from December 2025, “Attention Is Not What You Need” by Zhang Chong, proposes a radical departure. Instead of the costly L-by-L attention matrix, Chong’s "Grassmann Flows" architecture replaces it with a mechanism rooted in 19th-century differential geometry. This isn't an established alternative, but a novel experimental concept.
The core idea involves shrinking token representations from 256 to 32 dimensions and pairing them with a fixed set of neighbors at specific distances (1, 2, 4, 8, 12, and 16 positions). Each pair forms a plane within a 32-dimensional Grassmann manifold, encoded by 496 Plücker coordinates. A learned gate then mixes this geometric information into the token’s original representation.
Because each token interacts only with a fixed number of neighbors, this approach scales linearly with sequence length, dramatically reducing complexity. Early results on small models (13M to 18M parameters) show perplexity within 10-15% of size-matched Transformers, and even a slight edge on one classification task. However, lower computational complexity alone does not guarantee useful language model performance; the true test lies in scaling and independent reproducibility.
The 19th-century geometry hiding in each token pair
Instead of a full attention matrix, this new architecture compresses each token’s 256-dimensional state to just 32 dimensions. It then pairs each token with a select set of neighbors at fixed offsets: 1, 2, 4, 8, 12, and 16 positions. This local, multi-scale interaction avoids the quadratic cost of global attention.
Two 32-dimensional vectors, representing a token pair, define a plane within a 32-dimensional space. In mathematics, the collection of all such planes forms a Grassmann manifold. The model encodes each of these planes using Plücker coordinates, generating 496 distinct numerical features for every pair.
A learned gate then dictates how much of this rich 496-dimensional geometric information integrates into the token’s original state. This mechanism offers a linear scaling cost with sequence length. The paper claims this geometrical approach allows for mathematically inspectable representations, offering a potential avenue for interpretability—a compelling research angle, though not yet a proven benefit.
Promising numbers meet an ugly reproducibility test
Initial results from the paper, “Attention Is Not What You Need,” seemed promising. The author tested models with roughly 13–18 million parameters on Wikitext-2, claiming perplexity within 10–15% of comparable Transformers. For a classification task, a DistilBERT head using this geometric approach achieved 85.38% accuracy, slightly outperforming a Transformer baseline at 85.11%.
However, independent reproducibility efforts have raised questions. One reproduction by Elliot Arledge reported a 22.6% performance gap, significantly wider than the paper’s claims. While a second attempt yielded closer numbers, the discrepancy suggests that undocumented implementation choices may critically influence performance.
Two central objections persist. First, the author provided no code, making it impossible to verify the implementation details directly. Second, a key formula in the paper describes pairing each token with future tokens ($t + \Delta$), which could compromise causal evaluation by allowing the model to "peek" at upcoming information during training. If applied as described, this data leakage would invalidate the reported performance metrics for autoregressive tasks.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
A research lead, not an attention replacement
Evidence from models of 13–18 million parameters on Wikitext-2, even with reported perplexity within 10–15% of Transformers and a slight edge on DistilBERT classification, does not establish viability at scale. These are toy models by current standards, and their performance on small benchmarks cannot predict behavior in billion-parameter language models or long-context applications.
This geometric approach enters a crowded field of linear-time alternatives, including Mamba, RWKV, and RetNet. Grassmann flows must demonstrate superior quality and reproducibility, not merely linear complexity, to gain traction. The absence of public code, coupled with potential causal leakage in the paper's formulation and conflicting reproduction results—one reporting a 22.6% performance gap—raises significant concerns.
Until the research community has access to:
- Publicly available, reproducible code
- Rigorous causal tests to rule out data leakage
- Independent replications on larger models and datasets
The "Grassmann flow" is best viewed as a promising research lead for exploring structured representations, particularly for interpretability. Broader claims for replacing attention require far more robust validation.
Frequently Asked Questions
What are Grassmann flows?
They are a proposed sequence-modeling architecture that represents pairs of token vectors as geometric subspaces and uses those features instead of standard self-attention.
How could Grassmann flows reduce computation?
They pair each token with a fixed set of neighbors rather than comparing every token with every other token, aiming for linear rather than quadratic sequence-length scaling.
Has Grassmann flows been shown to outperform Transformers?
No. The paper reports small-model results near, but generally below, size-matched Transformers; independent reproductions have reported differing performance.
Why are researchers questioning the paper’s results?
The paper lacks reference code, implementation details are disputed, and commenters have raised concerns about future-token indexing in causal language modeling.