Physicist Suvrat Raju says GPT-6 Astra solved nearly 90% of his physics benchmark
Raju says GPT-6 Astra answered nearly 90% of the unpublished theoretical physics problems in TheoPhysEval, which he now considers saturated.
Suvrat Raju, a theoretical physicist at the International Centre for Theoretical Sciences in Bengaluru, says GPT-6 Astra answered nearly 90% of the unpublished problems in TheoPhysEval, a benchmark he helped build. He now considers the benchmark saturated. Kagan-Kans reported that figure in Understanding AI. The arXiv preprint puts the result higher. Its abstract says the best-performing agent, GPT-6-Astra harnessed in Codex, reached near-perfect accuracy of 98% on the 50 problems, at an average cost under USD 3 per problem.
The benchmark was designed to avoid a known problem with testing language models on physics. Models are trained on large swaths of the internet, so a problem that has been discussed online may already be memorized. Raju's group therefore collected problems that had never been published. Other physicists supplied minor but real results they had never written up. The authors describe these as problems that take top humans in the field about a day to solve. Raju wrote that contributors either gave up a possible publication elsewhere or did a small research project for the benchmark. At a September colloquium, Raju shared preliminary results in which leading models solved about 60% of the problems. By the time the full set was finished, Astra was solving nearly 90%.
When the team gave Astra the human-written solutions and asked where it had gone wrong, Astra argued that the humans had made the errors. Kagan-Kans reported that it identified subtle mistakes in those answers. One of the subtly wrong human answers had been written by Andreas Karch, a physicist at the University of Texas at Austin. Kagan-Kans reported that Astra had in fact answered 49 of 50 problems correctly, and that the one it missed was something of a trick question.
Raju put the cost in terms of coffee. Kagan-Kans reported that each correct answer cost less than $3 of compute on average. Raju described this in a talk as less than the price of coffee in Paris. Each answer came after a single human prompt. Kagan-Kans also reported that Raju does not plan a second version of the benchmark. Raju told Kagan-Kans, "We've kind of hit the ceiling of well-defined calculations that people can do." He added that open research could still be explored, but that it is not an evaluation.
The abstract also reports a finding that bears on what the scores mean. When agents were denied web search and access to background research papers, their accuracy changed only slightly. The authors read that as suggesting significant memorization of the literature. That is the authors' interpretation of their own experiment. The post does not describe the details of the ablation.
The distinction Raju draws matters for how the result should be understood. TheoPhysEval tests whether agents can carry out the concrete calculations required across theoretical physics research. The preprint's abstract says its results suggest state-of-the-art agents can perform them accurately. The abstract also discusses possible explanations for the current lack of agent-led breakthroughs in theoretical physics. Kagan-Kans reported that, as far as anyone can tell, the flood of physics papers has not yet produced any major AI-aided discovery.
What changed
TheoPhysEval, a benchmark of unpublished theoretical physics problems, is now considered saturated after GPT-6 Astra solved nearly 90% of its problems, and the arXiv preprint reports 98% accuracy on the 50 problems.
Type: Research · Applies to: GPT-6 Astra, TheoPhysEval
Why it matters
Expert-written research benchmarks can saturate within months of release. Buyers who rely on public tests to judge research-grade AI may be measuring less than they assume.
What you can do
Keep private, frequently refreshed evaluation sets for specialist technical tasks rather than relying on public benchmarks.
Original source
Where we saw it
Understanding AI
AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works