AI News
All stories
Reported

Google DeepMind's MKA benchmark puts Claude Fable 5.2 first at 57% Climb WSR

Artificial Analysis and Google DeepMind launched MKA, a benchmark of 200 visual knowledge-work tasks, where Claude Fable 5.2 (max) leads with a 57% Climb WSR.

Research · · Updated · 2 min read

Artificial Analysis and Google DeepMind have launched the Multimodal Knowledge Agent (MKA) evaluation, a benchmark of 200 private real-world tasks across 10 industries. Claude Fable 5.2 (max) comes out on top with a Climb WSR of 57%, ahead of Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%. The benchmark targets agents that must read dense visual project files, such as architectural drawings, engineering schematics and clinical scans, and then produce a deliverable: a slide deck, a spreadsheet, a PDF, a Word document or a marked-up image. Artificial Analysis describes these inputs as required to finish the task, not as supplementary context.

Each task is scored against expert rubrics averaging about 70 weighted items, split into two groups. Regression criteria are baseline checks that a competent model should pass. Climb criteria are harder checks meant to separate strong models from weaker ones. The headline number, Climb WSR (Weighted Satisfaction Rate), is the weighted share of Climb criteria a run completes, so the regression checks do not raise the score. Runs take place in Stirrup, Artificial Analysis's open-source agent harness, where code execution, web search, web fetch and image viewing are available for up to 250 turns. GPT-6 Luna (medium) then grades the submitted files as an agentic judge, meaning a language model reads the deliverable and scores it against the rubric.

Cost is where the ranking gets interesting. Claude Fable 5.2 averages $31.97 per task, more than five times the $6.29 of Gemini 4 Carbon, for about two extra points of Climb WSR. Against GPT-6.1 Sol at $1.11, the gap is roughly 29 times. Artificial Analysis headlines this as a spread of nearly 30 times among models within 10 points of the top score. The four OpenAI models it tested, GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra and GPT-6 Luna, all sit on the cost-performance Pareto frontier below Carbon, meaning no other tested model is both cheaper and better at that price. Claude Haiku 5.5 and MiMo-V2.6-Pro make up the rest of that frontier.

The leaderboard has no single winner once the results are broken down by category. Claude Fable 5.2 leads five of the 10 verticals: Software & Data, Engineering, Logistics & Mapping, Business, and Finance, the last one alongside Claude Sonnet 5.5. Claude Opus 5.5 (max) takes Education and Construction. Gemini 4 Argon (high) takes Science, Gemini 4 Carbon (high) takes Healthcare, and Claude Sonnet 5.5 (max) narrowly takes Law & Government. According to the post, output format matters less than industry, though it still shifts the results. Fable 5.2 leads four file types: PDF, Word Document, Spreadsheet (narrowly), and Presentation, where it shares the lead with Claude Opus 5.5 (max). Gemini 4 Argon and Carbon lead Image outputs, and DeepSeek V4.1 Flash 1017 leads Text Files. Word Document and Text File tasks are the hardest, and no model passes 52% on either.

What changed

Artificial Analysis and Google DeepMind launched the MKA benchmark, a set of 200 private visual knowledge-work tasks across 10 industries, on which Claude Fable 5.2 (max) scores highest with a 57% Climb WSR.

Type: Research

Why it matters

Cost per task varies nearly 30 times among models within about 10 points of the top score, so accuracy alone will not settle model choice for document-heavy knowledge work. Teams building visual document agents should weigh cost per task against performance.

What you can do

Run candidate models on your own visual project files and compare cost per task alongside accuracy before choosing a model for document work.

How the story developed

  1. Adds per-task costs of $31.97 for Fable, $6.29 for Carbon and $1.11 for GPT-6.1 Sol, plus Artificial Analysis life-science visual reasoning results.

Original source

Further reporting

Where we saw it

2 sources

Artificial Analysis, Latent Space

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works