AI News
All stories
Official

Meta's AutoRefine recalibrates LLM code judges as coding agents change

Meta researchers present AutoRefine, a method for keeping LLM code judges calibrated as the coding agent they evaluate evolves.

Research · · 2 min read

Meta researchers have proposed AutoRefine, a method that recalibrates the prompt of an LLM code judge against outcomes a development workflow already records. That keeps the judge tracking a coding agent as the agent changes. The paper is an arXiv preprint submitted on 6 November 2026 and prepared for the CAIN 2027 industrial track.

The problem is that LLM judges increasingly act as quality gates for code changes that have no test oracle. A judge prompt has to be calibrated for its task, and it has to be recalibrated when the system it judges changes. Hand labels are the usual calibration data, but the paper says they are resource intensive and go stale. Development workflows emit outcome labels at no cost: tests pass or fail, and changes land or are abandoned. The catch is that these labels are noisy.

AutoRefine handles that noise with a teacher model. The teacher sees both the judge's input and the outcome, explains each label it decides to keep, and abstains when the outcome cannot be explained from what the judge sees. The kept labels then rewrite the deployable judge's prompt so that it reproduces them. Abstention does two jobs. It removes labels the teacher cannot justify, and it flags changes whose outcome the judge's input does not contain enough information to explain. The authors read that second signal as a sign that a judge needs better inputs rather than a better prompt.

The paper reports results on two datasets. On a held-out set with an execution oracle, calibration raised Scott's pi from 0.36 to 0.60. For the noise test, the authors randomly flipped 25% of a deployed agent's outcome labels. The teacher set aside flipped fail labels at more than twice the rate of genuine ones, which cut overall label noise from 25% to 20%. The paper gives that 20% figure for the whole set, not for any single outcome type.

The gain from 0.16 to 0.51 was measured on 17 independently held-out diffs from the agent's current workflow. On 16 diffs from older workflows, the previous judge stayed ahead. The authors present this as evidence that a judge must be recalibrated as the agent evolves.

The DAIR.AI write-up summarizes the paper among its weekly picks and attributes the work to Meta. The arXiv record lists eight authors, including Nachiappan Nagappan. The arXiv abstract and the DAIR.AI summary give the same figures, but neither describes how the 17 and 16 diffs were selected. That limits how far the 0.16 to 0.51 figure can be read beyond Meta's own agent.

Why it matters

Teams that use LLM judges to grade agent output may find them drifting as agents change, and workflow outcomes could serve as the calibration signal.

Original source

Where we saw it

1 source

DAIR.AI (Elvis Saravia)

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works