AI News
All stories
Reported

Microsoft paper shows history-aware routing helps stateful agents choose cheaper models

A Microsoft paper finds that letting a router read session history improves agent task success while deciding which turns go to a cheaper model.

Research · · 2 min read

Microsoft researchers report that a router reading session history picks models better for stateful agents than one seeing only the current request. The paper, Context-Guided Model Routing for Stateful Agents, has the router choose one model for an entire user turn before the agent's tool-calling loop starts. It does not swap models between individual tool calls. That placement matters because a single user request can trigger several model calls and tool executions before the agent responds. The runtime keeps the conversation and application state, so the selected model can pick up from saved records rather than hidden state passed between providers.

The case for history rests on a simple problem. A follow-up request, such as applying a change to another sheet, reads the same whether the earlier attempt succeeded or left an error behind. If the previous turn established a correct formula, applying it elsewhere is routine. If the previous turn produced inconsistent totals, the same follow-up could spread the mistake to other cells. A router that sees only the new request cannot tell continuation from recovery, so the paper tests whether giving it the session history restores that information.

On the multi-turn SheetCopilot evaluation, history-aware routing passed 28.3% of tasks, compared with 25.0% without history. Both approaches sent roughly 23% of turns to the cheaper model. That detail shows the gain came from choosing which turns went to the cheap model, not from spending more on the expensive one. The two routers used the cheap model at about the same rate, so the difference lies in the turns each one assigned.

Switching models has a cost that a token price does not show. Keeping a conversation on one model lets later requests reuse a cached prompt prefix. A switch can force the accumulated conversation to be processed again. In the paper's evaluated conditions, moving from the expensive model to the cheap one recovered that cache cost within two model calls. Moving from cheap to expensive did not break even. Escalation can still be necessary when a task needs a more capable model to finish correctly, but in these conditions it carries a cost.

Extra calls also resend more history. The cheaper model's per-task cost advantage narrowed from 6.8 times on shorter sessions to 3.2 times on longer ones. The benchmarks also differed in task content, so the drop cannot be attributed to session length alone. The two figures show the direction of the trend, but they do not isolate length as the only variable.

Why it matters

Companies routing agent traffic to cheaper models should judge them on completed sessions, since per-token prices omit retries, extra calls and resent history. Switching models mid-session can also add cache costs.

What you can do

Compare routing setups on completed tasks by measuring success rate, total spend including routing calls and cached versus uncached input, and execution length including retries.

Original source

We have no link to the original source for this story yet.

Where we saw it

1 source

AI Engineering

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works