Introduction
Reinforcement learning (RL) has become central to enhancing large language models (LLMs). However, little research has focused on the individual contribution of Transformer layers in this process. Most current approaches uniformly update all model parameters, implicitly assuming each layer contributes equally to the achieved gains. This assumption is challenged by recent research conducted by Zijian Zhang and colleagues.
Key Findings
The study reveals that training a single Transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases, even surpass it. To quantify this phenomenon, the researchers introduced the concept of "layer contribution," which measures the fraction of total RL improvement recovered by training a layer in isolation.
Concentration of Gains
The study was conducted on seven models across two model families (Qwen3 and Qwen2.5), three different RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains, including mathematical reasoning, code generation, and agentic decision-making. The results show a remarkably stable pattern: RL gains are highly concentrated in a small subset, or even a single, Transformer layer, often situated in the middle of the architecture. Layers near the input and output ends contribute significantly less.
Practical Implications
This discovery has significant implications for the development of language models. By focusing training efforts on a limited number of layers, one can potentially save substantial computational resources and time while achieving comparable or even superior performance.
Concrete Examples
Consider a code generation application. Using the single-layer approach, a company could halve the training time while maintaining the model's accuracy and efficiency, representing a substantial cost-saving in infrastructure expenses.
Open Questions
While the results are promising, several questions remain. For instance, why are middle architecture layers more effective? How do these results translate into different model architectures or other types of tasks?
Conclusion
This study paves the way for a new approach to training language models via reinforcement learning. By optimizing the use of computational resources, companies can improve efficiency and reduce costs. Let's discuss your project in 15 minutes.