Looped Uniform Diffusion Language Models

The paper studies looping early Transformer layers within each denoising step of uniform diffusion language models (UDLMs), keeping the training objective and sampler unchanged. The authors report that looped UDLMs reach the validation NLL of a non-looped UDLM with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. At a matched NFE budget, they report up to 9.8% lower generation perplexity, mostly at moderate loop counts. Sampling smaller loop counts more often reportedly used 12% fewer loops on downstream tasks without degrading accuracy.

Why it matters

If the gains hold, looping could make UDLMs, which have stronger reasoning performance but high training cost, cheaper to train and run.

Read with care

Source claims not independently checked; results come from the authors' reported experiments on specific datasets.