AI News
All stories
Reported

Carnegie Mellon pre-print finds Jev fast but weak on math and logic judgments

A pre-print co-authored by Carnegie Mellon's Ramayya Krishnan reports Jev has a 150 ms median latency but accuracy drops on math and logic judgments.

Research · · 4 min read

A pre-print co-authored by Ramayya Krishnan, a Carnegie Mellon professor of information systems, reports that TypeSafe AI's Jev returned answers with a median latency of 150 milliseconds across benchmarks including RewardBench and JudgeBench. OpenAI's most capable model at the time, GPT-6 Astra, had a median of 1.89 seconds. The same testing found that Jev's accuracy falls off when a judgment has to be worked out rather than read from the prompt, a limit the speed figures alone do not reveal.

TypeSafe AI released Jev on 15 September and calls it a "System One model," borrowing the fast, instinctive thinking that Daniel Kahneman described in Thinking, Fast and Slow. The company argues that large language models can supply slow, deliberate reasoning but are too slow for quick decisions. It claims Jev responds end to end in 70 to 500 milliseconds. Pricing follows from the compute involved. TypeSafe charges US $0.042 per million input tokens and nothing for output, while IEEE Spectrum listed GPT-6 Astra at $10 per million input tokens and $50 per million output tokens.

The speed comes from how the model produces its answer. Michelle Chen, a group product manager at Cloudflare, which has released its own decision models, explained that an LLM generates output one token at a time, with each token conditioned on the ones before it. Decision models instead run a single forward pass that reads all the inputs and then emits one output. Jev's own output is constrained further. Zhongyu Wei, a Fudan University professor who co-authored a benchmark study and an open-weight alternative called SII-Decision, described the format. The model is given a question and options such as A, B and C, and it must choose among them, returning probabilities with its choice.

Decision models are not entirely new. Chen said classifiers, which sort data into classes and power tools such as spam filters, "are not new," and called decision models "almost like a rebranding." IEEE Spectrum identified the difference as generalization. Classifiers were traditionally built through domain-specific training, while decision models keep the general knowledge of an LLM, so one model can be pointed at many problems. Chen said a team once needed "a lot of data, stand up a machine learning team, and fine-tune" a classifier. Cloudflare now uses one decision model for tasks ranging from flagging spam pull requests to judging whether outage emails will affect its services.

TypeSafe's most striking claim is that Jev can't hallucinate, and IEEE Spectrum found that the narrow version of the claim holds up. LLMs choose among huge vocabularies one token at a time, so they can drift into wrong text. A decision model only ranks the options it was given. Krishnan, however, cautioned that "eliminating hallucinations in the output does not eliminate errors in decision-making." TypeSafe's blog makes a related argument, saying its type-safe outputs mean the model "never makes type errors," which it calls mathematically impossible. Removing malformed output does not make the chosen option correct, and the blog itself concedes that its speed and cost numbers are open to scrutiny.

The Carnegie Mellon work, posted to arXiv as "JEV-as-a-Judge: Accept When Confident, Escalate When Unsure" by Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman on 22 September, puts numbers on that gap. Jev comes within three points of GPT-6 wherever a verdict can be read off the text. It does so at 0.36% of GPT-6's fee and a 0.15-second median latency. It falls behind where the verdict must be derived, as in math, code and logic. The paper's central idea is to use Jev's confidence as a router. Confident verdicts are accepted, and the rest are escalated to a reasoning judge.

That cascade is where the paper finds Jev most useful. With a threshold set in advance, the combined system was 0.9 points more accurate than GPT-6 on 1,610 held-out pairs while costing 41% of GPT-6's fee. In a live test on two new workloads, it matched GPT-6's accuracy exactly. The abstract also says confidence routing weakens on style-adversarial pairs and reference-free prose, and it closes with a recipe for validating thresholds locally. IEEE Spectrum described the same result as the combination slightly beating GPT-6 alone at 41 percent of the cost.

Wei's study reaches a related picture across domains. He said the best decision models slightly beat LLMs on document understanding and come close on tool use, such as choosing the right software tool for a task. They fall behind in fields that demand topic expertise, including medicine, finance and law. Wei said that reasoning is "a key factor," though not the whole story, since some LLMs did well without it. He pointed to training methods as well, noting that SII-Decision used decision-specific training.

TypeSafe's own workflow evaluations are the source of its headline multiples. The company's homepage figures of 193.6x faster and 444.6x cheaper come from those tests, which use the average of GPT-6 Astra and Fable 5.1 as the reference answer. TypeSafe acknowledges that this choice biases results toward OpenAI and Anthropic models. It also says the LLM numbers come from OpenRouter, where more complex queries might be routed to stronger models. Its published evals were generally run from laptops on the West Coast.

Wei expects the two kinds of models to be combined. "The decision model can call an LLM to give a more detailed explanation. And an LLM can call a decision model for a quick judgment." He called that arrangement a "hybrid architecture" and said he thinks it will become common.

Why it matters

The results suggest decision models can cut latency and cost for well-defined judgments, but they need checks before handling math or logic-heavy decisions. A hybrid design that escalates uncertain cases to an LLM is one tested option.

What you can do

Route only low-confidence or reasoning-heavy cases to a full LLM, and measure error rates on math and logic cases before relying on a decision model.

Original source

Further reporting

Where we saw it

1 source

IEEE Spectrum

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works