AI News
All stories
Reported

Microsoft's ThinkingBox uses database-state rewards to lift Qwen3.8-27B task passes

Microsoft's ThinkingBox team used executable checks on final database state as RL rewards, raising Qwen3.8-27B from 38 to 107 tasks passed across 20 attempts.

Research · · 4 min read

Microsoft's ThinkingBox team trained Qwen3.8-27B with reinforcement learning, using the same executable checks it uses for grading as the reward signal. The number of tasks the model completed correctly on all 20 attempts rose from 38 to 107 on its 507-task benchmark, ThinkingBox-Bench. That moves the share of tasks from 7.50% to 21.10%. Microsoft and Hugging Face said the training code is public on GitHub.

The grader checks the database, not the agent's words. Each rollout gets a fresh backend, a set of MCP tools and a simulated user, which the team ran with GPT-5 Chat. When the episode ends, the sandbox extracts what actually changed in the backend and runs deterministic checks against the final state. A trajectory passes only if it produces the correct end state, and wrong, missing or extra side effects fail. The reward is 1 or 0. A minority of narrow requirements that can't be read cleanly from the database go to an LLM judge, which answers a binary rubric question. The simulated user plays a role in the environment and does not grade the agent.

Task selection shaped the work, and the team's reasoning rests on how GRPO compares rollouts within a group. If all eight rollouts fail, every reward is 0. If all succeed, every reward is 1. In both cases no trajectory does better than another, so the update carries no task-level signal. A mixed group is what teaches the model something. Because each rollout is expensive, with its own backend, tool session and user model, the team wanted to avoid wasting runs on tasks that never or always pass.

To find those tasks, the team ran each candidate repeatedly against the base model and estimated its success probability with a Jeffreys posterior, Beta(k + ½, n − k + ½). It then sorted tasks into five buckets with boundaries at 0.08, 0.40, 0.60 and 0.90, and kept the three middle buckets. The team calls this offline policy-relative difficulty. The pool was fixed before training, so difficulty was not re-estimated as the model changed. The final pool held 187 tasks. The repository ships a 39-task public development set, which is not the private 187-task pool used for the reported run.

The reference run was full-parameter, used token-mean GRPO and ran for 50 steps. Each update generated 18 groups of 8 rollouts, or 144 episodes, across 24 H100s, with an AdamW learning rate of 1e-6 and zero weight decay. Groups with zero variance were replaced DAPO-style. Backend timeouts were treated as system errors. The affected groups were refilled rather than scored as failures, since a timeout is not an incorrect agent outcome.

On the benchmark, average success (pass@1) rose from 51.70% to 60.78%. Observed 20/20 completions rose from 38 to 107, a gain of 69 tasks. Coverage moved the other way. Tasks solved at least once in 20 attempts fell from 453 to 446, or 89.35% to 87.97%. The team said the pattern is consistent with sharpening. One explanation it offers is that probability shifts toward trajectories that work repeatedly while some low-probability paths disappear. In the team's words, the model did not become broader; it became more repeatable.

The benchmark is out of distribution for training, since its domains, databases, policies and most tools were absent from the RL pool. The team also acknowledged that the two sets share the ThinkingBox framework, the simulated-user machinery, the grading method, a few common tools and a similar task style. Those shared elements limit how far the transfer claim extends. Claude Opus 5.5 completed 241 tasks on all 20 attempts and GPT-6 Astra completed 231. GPT-5.6 Sol had higher pass@1 at 61.91% against 60.78%, though it completed only 82 tasks on every attempt.

Breakdown by domain shows that pass@1 and observed 20/20 rose in all five domains. Pass@20 rose in neobank and consulting but fell in retail, travel and auto insurance. Auto insurance lost the most breadth, at 7.0 points, while retail gained the most repeatable completions, at 24.5 points. One hotel-booking example illustrates the change. Before training, the base model changed a booking and issued refunds without the required manager approval in 15 of 20 attempts. After training, it preserved booking and payment state in all 20. The figure caption, however, says all expected records matched in 19 of those attempts.

The cost figures are estimates. Using the same method as the team's earlier analysis, the 507 by 20 campaign fell from $925.80 to $855.58. Cost per successful attempt fell from $0.177 to $0.139. Cost per dependable task, meaning campaign cost divided by tasks completed on all 20 attempts, fell from $24.36 to $8.00. The team compared this with GPT-5.4 at $6.80, GPT-6 Astra at $7.45 and Claude Opus 5.5 at $7.80. The team said the dollar values are a comparative serving index rather than invoices. They exclude the cost of the training run and the policy probes used to build the pool.

What changed

Qwen3.8-27B trained with reinforcement learning on executable database-state rewards raised observed 20/20 completions on ThinkingBox-Bench from 38 to 107 tasks.

Type: Research · Applies to: Qwen3.8-27B, ThinkingBox-Bench

Why it matters

Verifiable outcome rewards tied to database state offer a way to train business agents on whether tasks actually completed, not how they sounded.

What you can do

Teams building database-facing agents can review the released training code to test whether state-based rewards fit their workflows.

Original source

Where we saw it

1 source

Demetrios from the AAIF

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works