# Study benchmarks decision models against LLMs and introduces open-weight SII-Decision

> Researchers including Fudan University's Zhongyu Wei benchmark decision models and find they trail LLMs in medicine, finance and law.

- Status: reported (One outlet, not independently verified yet.)
- Topic: Research
- Published: 2026-11-16T17:00:54+00:00
- Updated: 2026-11-16T17:11:06.195062+00:00
- Web page: https://ai-news.ikangai.com/story/study-benchmarks-decision-models-against-llms-and-introduces-open-weight-sii-dec/
- Seen in: IEEE Spectrum
- Change: research (Research)
- Applies to: SII-Decision

## What changed

A study benchmarked decision models against generative LLMs and introduced SII-Decision, a family of open-weight decision models in 4B and 27B sizes.

A study co-authored by Fudan University professor Zhongyu Wei benchmarks decision models against generative LLMs. It finds that the fast, single-pass models trail LLMs on questions that require specialist knowledge, including medicine, finance and law. The preprint, posted to arXiv in early October 2026, also introduces SII-Decision, a family of open-weight decision models in 4B and 27B sizes. The work examines the wave of decision models that followed TypeSafe AI's Jev, and its results show where that category is strong and where it still falls short.

A decision model is an LLM whose output has been constrained so that it must pick from a predefined set of answers. Wei describes the setup as giving the model a question and options such as A, B and C. The model chooses one and returns probabilities alongside its choice. The SII-Decision project page says its First-Token Decision Readout restricts an open-weight LLM's next-token probabilities to valid option labels. A structured decision therefore comes out of one forward pass rather than a long generated answer. That is why the models are fast, and it is also why they cannot produce free-form text.

To test this, the authors built JEVal, a bilingual benchmark of 11,257 instances drawn from 36 datasets across 10 application domains. They evaluated 25 model configurations spanning decision models and generative LLMs. The overall numbers do not favor the decision models. SII-Decision-27B reaches 79.24% accuracy, close to Jev at 78.38%. GPT-6-Luna in non-thinking mode scores 80.93%, and GLM-5.3-Flash in thinking mode scores 80.37%. SII-Decision-27B does have the lowest expected calibration error of the 25 configurations, at 0.0149. The authors' summary is that decision models are strongest when the evidence needed is in the input, while the best generative LLMs keep an advantage in medicine, finance and law.

Calibration is one of the weaknesses the authors flag. The paper's abstract says these models can often identify the most likely outcome while substantially overstating its probability. The project page gives a specific case. On prior items, Jev assigns the true mode 89.35% probability on average, compared with 35.60% in the known distribution. The project page's own conclusion is that ranking an option correctly does not ensure the probabilities are faithful.

Multi-step work exposes a different cost. In tau-bench, Jev chooses the next tool or response action, while the same base LLM writes arguments and user-facing answers. The project page reports that the base LLM reached 77.6% task success with a median episode time of 267.1 seconds. Jev plus the LLM reached 66.1% with a median of 160.6 seconds. P95 time rose from 695.5 seconds to 816.0 seconds. Faster local decisions shorten episodes, but as the abstract puts it, decision errors accumulate over long trajectories. A separate pre-print co-authored by Carnegie Mellon's Ramayya Krishnan reported a similar pattern. Jev did well when the answer could be read from the prompt, but its performance dropped on math and logic puzzles. Krishnan cautions that eliminating hallucinations in the output does not eliminate errors in decision-making.

The social simulation results have both strengths and costs. On 277,500 SocioBench respondent-question pairs, SII-Decision-27B reached 39.87% accuracy at 8.36 seconds per item. That is close to DeepSeek-V4-Flash-0731 in thinking mode, which scored 40.41% at 211.65 seconds. In ElectionSim, about 330,000 simulated voters produced correct calls for 47 of 51 states and 12 of 15 battleground states. The project page says vote-share errors were about twice those of the original generative pipeline, and that the simulation overestimates the Democratic share. Speed is clearly an advantage here, but the accuracy and aggregate errors still matter.

Wei points to reasoning as a key factor in the domain gap, but not the only one. Reasoning lets an LLM work through its response before giving a final answer, and the paper's SII-Decision method tries to carry that benefit into a single pass. Its Reasoning-to-Readout Self-Distillation trains the readout to match the distribution the same model reaches after reasoning, without gold labels or a stronger teacher. Wei also notes that some LLMs performed well without reasoning, so reasoning alone does not explain every domain gap. He says the training method for decision models matters too.

Krishnan and colleagues also tested passing Jev's low-confidence answers to GPT-6 for further consideration, as the IEEE Spectrum report describes. That combination slightly beat GPT-6 alone in accuracy and did so at 41 percent of the cost.

## Why it matters

Decision models look most suitable for document and tool-routing tasks, and less suitable for domain-expert work in regulated fields. Buyers should match the model to the task type.

## What you can do

Keep decision models out of expert-domain workflows such as medicine, finance or law until tested on your own data.

## Original source

- [(2610.03935) General Decision Models: Benchmarking and Insights Beyond Jev](https://arxiv.org/abs/2610.03935)
- [GitHub - sii-research/sii-decision · GitHub](https://github.com/sii-research/sii-decision)
- [JEVal & SII-Decision | General Decision Intelligence](https://sii-research.github.io/sii-decision/)

---

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it.

Cite as: IKANGAI AI News, "Study benchmarks decision models against LLMs and introduces open-weight SII-Decision", 2026-11-16, https://ai-news.ikangai.com/story/study-benchmarks-decision-models-against-llms-and-introduces-open-weight-sii-dec/
