MLE-World predicts ML training-run outputs before code runs, matching 77.9% of logs

MLE-World, a fine-tuned Qwen3.5-4B model, predicts what a machine learning program would print when run. That includes training logs, errors and validation scores. An agent can use these predictions to compare candidate code before paying for a real training run. The system is trained on records of successful and failed program executions. Each prediction takes four inputs: the candidate program, a description of the task's data, the software packages available, and the execution time limit. The output is a predicted execution report.

The practical use is in the search loop of an agent that writes several versions of a training script. Finding out which version works can take much longer than writing it, since each attempt has to train a model and return a validation score. A predicted error trace lets the agent fix broken code without running it first. A predicted validation score lets it rank working candidates. The researchers build three search strategies on these reports. One picks the candidate with the highest predicted score. Another uses predicted feedback to revise a program before execution. A third combines ideas from several candidates. A combined program is new code, so it gets a fresh prediction rather than inheriting its parents' reports.

A convincing predicted log can still be wrong. An agent that repeatedly trusts an inaccurate score could spend an entire search improving a program that never performs well. To guard against this, each prediction carries a confidence score, and low confidence triggers a real execution. The system also verifies new predicted best results, and final submissions are selected using real validation scores. The predictor narrows the set of runs that need to happen. It does not replace measurement.

The headline figure is 77.9%. Across 1,700 held-out programs from 17 MLE-bench Lite tasks, the predicted terminal output matched the observed output on that share of programs under the researchers' judging procedure. The strongest prompted baseline reached 41.1%. The match criterion matters.

The main test uses new programs from tasks that appeared during training. The result supports prediction within those task environments. It does not establish the same accuracy for an unfamiliar dataset or training setup. The predicted reports are also a cheaper signal, not a measured one. According to the source, they help the agent decide which experiments deserve real measurement, and the final validation score still comes from an actual run.