AI News
All stories
Official

SII and GAIR apply agent-driven self-improvement to a full pre-training run

Researchers from SII and GAIR apply recursive self-improvement to an entire frontier pre-training run from scratch and report the details.

Research · · 2 min read

Watch the short

Under a minute, read by an AI anchor. Made .

YouTube loads only when you press play.

Researchers from SII and GAIR report applying recursive self-improvement to an entire frontier pre-training run from scratch. The run produced 801-30B-A3B-Base, a 30B mixture-of-experts model trained on 11.8T tokens in three months by a newly formed team. The authors say the base model beats Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B and trails only the larger Qwen3.5-35B-A3B. They also say it reaches the highest average on four agentic benchmarks against fully post-trained models. These comparisons are the authors' own, and the agentic result rests on a lightweight SFT stage of 6B tokens with no RL.

The authors start from a problem with the usual recipe. Public recursive self-improvement work has targeted post-training, where a run starts from an existing base model, each trial costs GPU-hours, and the result comes back as a verifiable score. Pre-training breaks all three conditions. A single run consumes hundreds of thousands of GPU-hours over months and cannot be repeated. Its goal, post-training headroom, cannot be read off any one score. What has to improve is a whole production system of data pipelines, training infrastructure and the teams that run them. The paper's answer is to split the loop into smaller cycles that can each close, with each one repairing a step the full run breaks.

According to the paper's key points, those cycles fall into three groups. Agents run debugging loops over infrastructure and data. A headroom loop uses tiny SFT runs and learning-aptitude probes to check whether the base model is improving in the ways post-training needs. Experience-to-metric loops turn expert judgment into metrics for data classifiers and crawl coverage. All of them share a context infrastructure that keeps negative results, so a later proposal does not retry something an earlier one already ruled out. The system spans six data pipelines and more than 130 infrastructure versions.

The model was evaluated as a base model. The authors also report that it keeps its position over Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B under identical SFT. The agentic claim is different. It uses 6B tokens of SFT and no reinforcement learning, and it is compared against models that received full post-training, including Qwen3.6-35B-A3B. The paper presents the lightweight SFT as a probe of post-training headroom rather than as a finished post-training recipe. The abstract says the lightweight SFT "only probes post-training headroom," which is the authors' own framing.

The authors release both the base model and the lightweight SFT data. The abstract points to a GitHub repository, GAIR-NLP/801-30B, for them. Their argument is that post-training RSI assumes a strong base model already exists, so applying self-improvement to pre-training changes the foundation every later stage builds on.

What changed

Recursive self-improvement is applied to an entire frontier pre-training run from scratch, producing the 801-30B-A3B-Base model trained on 11.8T tokens.

Type: Research · Applies to: 801-30B-A3B-Base, Qwen3-30B-A3B, Nemotron-3.5-Lightning-30B-A3B, Qwen3.5-35B-A3B, Qwen3.6-35B-A3B

Why it matters

Improving pre-training with agents could lift the base model that every later stage builds on, and the open release lets others test the approach.

Original source

Where we saw it

1 source

DAIR.AI (Elvis Saravia)

GAIRSII

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works