Realtime-Bench finds frontier AI models struggle with time-sensitive tasks
A 50-task benchmark of games and web tasks shows top models pass only 36% of tasks under real-time latency, versus up to about 80% with a paused clock.
Researchers from the University of Texas at Austin, Ohio State, the University of Illinois at Urbana-Champaign, NVIDIA, Northwestern and Steadyworks built Realtime-Bench, a suite of 50 interactive game and web tasks. In these tasks, a late action can change whether the task succeeds. The paper's abstract puts the best success rate under wall clock timing at 44%, and the best Pass^3 score at 36%. GPT-6 Astra (high) was reported at 36% on that wall clock setting.
The 50 tasks split into two groups. The 30 web tasks cover travel, e-commerce, service and security. The 20 game tasks cover arcade, puzzle, strategy and simulation. The authors designed the suite to probe three abilities. The first, RT-Perceive, is temporal state awareness: a model must use what it observes to track changes in the world. The second, RT-Deliberate, is adaptive deliberation, in which the model alters its reasoning length and depth according to how much time is available. The third, RT-Act, is time-aware action selection, meaning choosing when to act. The paper's example is an agent buying a ticket. The agent must notice whether the sale status it saw is still current, limit further deliberation when the purchase window is closing, and decide whether to wait for sales to open or buy while tickets remain available.
The benchmark runs each task under three clock settings. Wall clock advances the environment according to real inference latency, so a slow model can miss its window. Paused clock removes that delay entirely and serves as an idealized reference for what a model can do if time is not a factor. The third setting, Token clock, ties simulated time to output. In the paper's words, it "advances the environment at a fixed rate per generation token". So an action takes effect at the simulated time when it is committed, and a model that writes more tokens acts later even if its hardware is fast. The authors introduced this setting to separate a model's real-time capability from the speed of the serving infrastructure. The wall clock numbers alone cannot make that distinction.
The models tested came from OpenAI (GPT-6-Astra, GPT-5.6 Sol, GPT-Realtime-2 and OpenAI Decisions), Anthropic (Claude Fable 5.1 and Claude Opus 5), Google (Gemini 3.8 Flash, Gemini 3.1 Flash Live and Gemini 3.8 Live), Jev 1.13, DeepSeek (V4.1) and Moonshot AI (Kimi-K3). The paper finds that temporal costs limit success in evolving environments. Higher reasoning effort has mixed effects under time pressure. The authors read this as a trade-off between decision quality and timely execution. They suggest adaptive reasoning with temporal awareness, and continuous perception and action, as directions for future work.
Why it matters
The gap between paused and real-time results suggests current agents are unreliable wherever response speed matters, such as buying tickets or bidding in auctions. Teams should not assume that strong offline benchmark scores carry over to live, latency-sensitive workflows.
What you can do
Test any agent that must act on live information under realistic latency before deploying it in time-critical workflows.
Original source
Further reporting
Where we saw it
Import AI (Jack Clark)
AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works