Introduction
Transformers have become indispensable in the field of artificial intelligence, serving as the backbone for applications ranging from computer vision to language modeling. At the heart of their success lies the QKV attention mechanism, based on three projections: query (Q), key (K), and value (V). But is this complexity truly necessary? This article delves into a recent study exploring the impact of reducing these projections to one or two.
Projection Sharing Variants
Q-K=V: Shared Keys and Values
This approach considers that keys and values can occupy similar representational spaces, allowing for a 50% reduction in KV cache with only a 3.1% increase in perplexity for language modeling. In an on-device inference context, this reduction is significant.
Q=K-V: Shared Query and Key
While this variant creates symmetric attention maps, it breaks the directionality of attention, which can harm result quality. However, it offers interesting efficiency gains in specific cases.
Q=K=V: Single Projection
Using a single projection greatly simplifies the model but may limit its ability to capture complex dynamics between queries, keys, and values. The study shows that in some cases, this simplification can still be sufficient for specific tasks.
Experiments and Results
The authors of the study tested these variants on synthetic tasks and datasets for vision (MNIST, CIFAR, TinyImageNet) and language modeling with models ranging from 300M to 1.2B parameters over 10 billion tokens. The results indicate that transformers with projection sharing can compete with, and sometimes outperform, traditional QKV transformers, particularly in contexts where on-device inference is crucial.
Practical Implications
Reducing the memory required for attention cache has direct implications for deployment on resource-constrained devices. Projection sharing, combined with techniques like head sharing (GQA/MQA), allows for a significant reduction in memory needs, thus facilitating inference on embedded devices.
Conclusion
The systematic study of QKV variants shows that transformers do not necessarily need three projections to be effective. By exploring less complex configurations, it is possible to reduce resource consumption while maintaining competitive performance. For developers and decision-makers, this paves the way for lighter and more adaptable models. Let's discuss your project in 15 minutes.
References
- Kayyam, A., Madan Gopal, A., Lewis, M. A. (2026). "Do Transformers Need Three Projections? Systematic Study of QKV Variants." ICML 2026.
Source Code
The study's code is publicly available at this [URL].