Language Models Learn to Improve Language Models

The paper proposes LM2LM, which trains an LLM proposer with reinforcement learning to generate language-model improvement proposals. A frozen research world model, grounded in real experimental records, predicts each proposal's gain and serves as the reward. To limit exploitation of overestimated predictions, the authors periodically execute top-rated proposals and feed the results back into the world model's context. Reported real-execution evaluations across eight development environments claim Qwen3.8-27B outperforms GPT-6-sol and requires 33% lower data-generation cost, with generalization to an unseen model family.

Key takeaways

• A frozen LLM grounded in real experimental records serves as a cheaper reward model than running a full training job per proposal.
• Periodically executing top-rated policy proposals and appending their outcomes calibrates the world model against overestimation.
• The reported 200 calibration runs cost $40.1k and 343 hours, versus $60.0k and 510 hours for the 300-record ICL baseline.
• Qwen3.8-27B trained with LM2LM is reported to outperform GPT-6-sol given 300 real experimental records in context.

Why it matters

Automating language-model improvement is a possible route toward recursive self-improvement. Cutting the cost of reward collection from hundreds of thousands of GPU-hours could make learned improvement proposers more practical to train.

Read with care

Source claims not independently checked; results are reported by the authors on their own held-out tasks and environments, and the cost comparisons depend on their GPU pricing assumptions.