AI News
All stories
Official

IBM's JevAct adds a runtime gate that checks agent tool calls without retraining

IBM researchers introduce JevAct, a runtime gate between an agent and its tools that checks write calls and hand-offs while leaving the model and tools unchanged.

Research · · 2 min read

IBM researchers have proposed JevAct, a runtime gate that sits between a tool-using agent and its tools. It checks selected actions before they run, without changing the agent's model or the tools themselves. The gate targets two failure modes the authors describe. One is consequential write calls that run without any independent check on policy compliance. The other is agents that end a task before it is actually finished. On τ²-bench Telecom, the paper reports that Opus 4.6's success across all three runs rises from 39.5% to 80.7%. All eight agents tested there improve by 16.9 points on average in three-run success.

The design splits the work into an offline step and a runtime step. Offline, an LLM reads the domain policy, the tool schemas and unlabeled logs. It turns them into code checks, yes/no questions, rules, and a checklist for hand-offs. At runtime, the gate decides whether a selected write call or hand-off should proceed. Code verifies facts against the tool records, which is deterministic. A separate decision model answers the conversational questions. That model is prefill-only, meaning it returns probabilities rather than generated text. When a call fails the check, the feedback goes back to the agent, which can revise it.

The evaluation covers eleven LLM agents across τ²-bench Airline and Telecom and BFCL. The paper's abstract describes gains as particularly consistent on Telecom and BFCL, with Airline more mixed. The fuller picture is more uneven. All nine agents on BFCL improve in three-run success. On the at-least-once measure, however, the BFCL results are mixed. Airline changes are smaller and vary by agent. The authors also list limits of their own. The Telecom gain does not appear when a gpt-oss-120b model serves as the user simulator. On 260 labelled development write calls, 42 of the 101 incorrect calls break no compiled question. In other words, about two in five incorrect calls on that development set pass without tripping any compiled question.

The claim that the gains come from the harness alone is measured on benchmark tasks with a simulated user. The authors' own note that the Telecom gain does not appear with a gpt-oss-120b user simulator shows that the result depends on that setup.

What changed

IBM researchers propose JevAct, a runtime gate that checks selected write calls and hand-offs before they run, without changing the agent's model or tools.

Type: Research · Applies to: Opus 4.6

Why it matters

Teams deploying tool-using agents can catch invalid actions and premature task endings through a harness layer, without retraining the model.

What you can do

Add deterministic checks on consequential tool calls and hand-offs before widening agent autonomy.

Original source

Where we saw it

1 source

DAIR.AI (Elvis Saravia)

IBM

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works