# Unified training runs pretraining, RL and fine-tuning under one likelihood-based loss

> Researchers from CMU, Johns Hopkins and Aurora Innovation show that all foundation-model training stages can run under one likelihood-based algorithm.

- Status: official (From the company, authors or authority themselves, with a link to their own page.)
- Topic: Research
- Published: 2026-11-01T17:34:56+00:00
- Updated: 2026-11-01T17:34:56+00:00
- Web page: https://ai-news.ikangai.com/story/unified-training-runs-pretraining-rl-and-fine-tuning-under-one-likelihood-based/
- Seen in: DAIR.AI (Elvis Saravia)
- Change: research (Research)

## What changed

Pretraining, midtraining, supervised fine-tuning and reinforcement learning can all run under one likelihood-based algorithm, according to the paper.

A group from Carnegie Mellon, Johns Hopkins and Aurora Innovation argues that the separate stages used to build foundation models are not fundamental. Pretraining, midtraining, supervised fine-tuning and reinforcement learning can all run under one likelihood-based algorithm, which they call unified training. The paper, submitted to arXiv on 29 October 2026, derives suitable RL objectives as maximum-likelihood fitting to model-generated target distributions. In practice, RL generations and static data are placed under one loss and trained by one trainer. The design question therefore shifts from how to switch between recipes to which data to train on, in what proportions, and when. Conventional staged training becomes one special case.

Under controlled conditions, a 124M model trained from scratch beats staged training on RL tasks and on larger reasoning puzzles it never saw during midtraining or RL. Under a fixed compute budget, unified training has the highest accuracy at the smallest and largest budgets, while staged training leads at intermediate budgets.

The midtraining results are more nuanced. On a 0.76B model, mixing in 0% to 5% midtraining data kept math accuracy similar while retention improved, but coding accuracy dropped as the mix grew. After math training, however, all unified models beat staged training on final coding accuracy despite starting lower, and the 5% mix did best on all four metrics. That pattern suggests the mixing ratio involves a tradeoff: retention improved while coding accuracy fell as the mix grew.

The authors are careful about how far this goes. The experiments are small-scale, and they have not found a mixture or schedule that works best across the board. The paper offers a framework for choosing data and timing rather than a recipe that replaces staged training. The abstract also claims gains in accuracy-compute tradeoff, task coverage and a stronger starting point for continual learning. The paper is 67 pages with 31 figures and 22 tables.

## Why it matters

If the approach holds at scale, one training pipeline could simplify how teams build models. The results so far come from small experiments, so the benefit is not yet established.

## Original source

- [(2610.40564) Training Foundation Models with a Single Algorithm](https://arxiv.org/abs/2610.40564)
- [Training Foundation Models with a Single Algorithm | DAIR.AI Academy | DAIR.AI Academy](https://academy.dair.ai/papers/training-foundation-models-with-a-single-algorithm-2610.40564)

---

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it.

Cite as: IKANGAI AI News, "Unified training runs pretraining, RL and fine-tuning under one likelihood-based loss", 2026-11-01, https://ai-news.ikangai.com/story/unified-training-runs-pretraining-rl-and-fine-tuning-under-one-likelihood-based/
