AI News
All stories
Official

Artificial Analysis launches Decision Index v2 with calibration-weighted scoring

Artificial Analysis has released version 2 of its Decision Index, with 45% of the score now built on private datasets and a headline metric that rewards calibrated confidence.

Research · · 3 min read

Private datasets now make up 45% of the Index, up from 25% in version 1. Artificial Analysis says the aim is to better reflect real-world utility and to guard against benchmark gaming.

The new headline metric, the Decision Score, rests on a simple setup. A decision model's answer is usually acted on only when its confidence clears some threshold. Otherwise the question goes to a person or a more capable model. Artificial Analysis says the score measures average payoff across all such thresholds, so it does not depend on one cutoff. For each question the score is 50 × (1 + c − (p − c)²), where p is the probability the model gives its chosen answer and c is 1 if that answer is right and 0 if not. A correct answer scores between 50 and 100, with confidence pushing it higher. A wrong answer scores between 0 and 50, with confidence pushing it lower. Across an evaluation this works out to 50 × (1 + Accuracy − Brier score), so the metric is in effect accuracy minus a calibration penalty.

The calibration term changes the rankings. On AA-Paperclip, Clef passes 36% of documents on raw accuracy and Celeris-1.1 Decision passes 24%. Yet Celeris-1.1 Decision ties Clef on the Decision Score, because its Brier score is 0.15 against 0.27 for Clef. Across the whole Index, GLiDE v1.2 leads Drex Max v1.1 on accuracy metrics, 60 to 58, while Drex Max v1.1 edges ahead on the Decision Index.

The largest new component is AA-Paperclip, weighted at 30%. It is a private set of 1,500 documents, including resumes, receipts, invoices and AI and machine learning research papers. Each document has five decisions and ten facts to extract. The whole document counts as one decision and is correct only if all 15 answers are right. That rule explains why headline accuracy and document pass rates diverge. Drex Max v1.1 gets 97% of individual answers right, close to Claude Haiku 5.5 (Max) at 99%. But it passes 63% of documents against Haiku's 92%. Artificial Analysis says decision models lose most documents on the decisions rather than the extraction. Drex Max v1.1 makes all five decisions correctly on 71% of documents and all ten extractions on 88%.

AA-Symmetry, weighted at 15%, tests whether a model's answer survives rewording. Artificial Analysis takes 120 base questions and produces 440 variations by reordering options, swapping labels and negating yes/no questions. The Symmetry Score is 0 if the most probable option changes between forms. Otherwise it is reduced by how much that option's probability moves across forms. Models from Baseten lead the test, which the company attributes to a high abstention rate. Mammoth 27B scores 96 and Elephant 9B scores 93. Mammoth 27B picks the same answer across variations on 99% of questions, against 91% for Drex Max v1.1, 88% for GLiDE v1.2 and 67% for Clef.

The weights of the older benchmarks fell. AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1 each drop from 25% to 15%. AA-Omniscience-Decision now varies the wording of its "I don't know" option across ten alternatives. HLE-MultiChoice adds an option to decline, using the same ten phrasings. When2Call from NVIDIA falls from 25% to 10%. WildGuardTest is removed because it is highly saturated. EvalSafe-O*NET-AA is added at 15%, described as Artificial Analysis's implementation of TypeSafe AI's O*NET benchmark.

At the top of the Index, Drex Max v1.1 from Nace AI and GLiDE v1.2 from Fastino Labs both score 73. Mammoth 27B from Baseten scores 72. Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) scores 76, about three points ahead, but they cost far more to run. GPT-6 Luna (Max) costs $1,530 per 1M Tasks against $102 for Drex Max v1.1. It takes 11.34 seconds per task against 275 milliseconds. Cost per 1M Tasks runs from $14 for Celeris-1.1 Decision to $859 for Clef. Artificial Analysis says Celeris-1.1 Decision, Decider v1.2 and Drex Max v1.1 sit on the cost-versus-Index Pareto frontier.

Cost depends on more than token price. Artificial Analysis says most decision models bill a document's input once for all its questions. GLiDE v1.2 and Decider Flash v1.2 instead evaluate each question in a separate pass and bill the input every time. An AA-Paperclip task uses about 63k input tokens on those models, against about 9k for Drex Max v1.1. As a result, GLiDE v1.2 costs $780 per 1M Tasks despite only 33% higher list prices, nearly 8x Drex Max v1.1. The same mechanism makes Decider Flash v1.2 cost $196, nearly 3x Decider v1.2, even though its token price is half that of Decider v1.2.

What changed

Artificial Analysis has released version 2 of its Decision Index, where private datasets make up 45% of the score (up from 25% in version 1) and a new headline Decision Score rewards calibrated confidence.

Type: Research · Applies to: Decision Index v2, AA-Omniscience-Decision v1.1, HLE-MultiChoice v1.1, When2Call, WildGuardTest, EvalSafe-O*NET-AA

Why it matters

Teams choosing decision models can see that cost per task and calibration can change rankings, and that per-question billing can make one model several times more expensive than another with similar list prices.

What you can do

Before selecting a decision model, compare its Decision Score, cost per 1M tasks and time per task on the new leaderboard rather than relying on accuracy alone.

Original source

Where we saw it

1 source

Artificial Analysis

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works