Artificial Analysis launches VIALS-AA; Gemini 4 Carbon (high) leads at 64%
Artificial Analysis launched VIALS-AA, a benchmark for scientific image reasoning, where Gemini 4 Carbon (high) leads with 64% of tasks answered correctly in all three repeats.
Artificial Analysis has launched VIALS-AA, its implementation of Handshake AI's VIALS benchmark. The benchmark asks language models to interpret scientific images, such as gel blots, plasmid maps and flow cytometry plots, and to answer questions about the evidence in them. Gemini 4 Carbon (high) leads with 64%, answering all three repeats correctly on 103 of the 161 tasks, two more than GPT-6 Astra (max). The headline figure is pass^3, the share of tasks a model answers correctly in all three of its repeats. Artificial Analysis uses it because it says a result that changes between runs cannot be relied on in scientific work.
Each task runs three times with Stirrup, Artificial Analysis's open-source harness, which gives models code execution and image inspection tools. The benchmark also reports pass@3, the share of tasks answered correctly at least once. The gap between the two measures matters most here. Kimi K3.1 Pro (max) and GPT-6 Astra (max) each solve around three-quarters of tasks at least once, but Kimi reaches only 45% on pass^3, against 63% for Astra. According to the launch article, leading open-weights models solve about as many tasks at least once as closed frontier models, but they are much less consistent. Kimi K3.1 Pro (max), MiMo-V2.6-Pro and MiniMax-M3.1-Flash (max) each solve 114 to 119 tasks at least once, yet pass all three repeats on only 48 to 73 tasks. The seven top-scoring closed models pass all three repeats on 95 to 103 tasks.
Similar scores can come from very different working styles. Claude Sonnet 5.5 (max, with fallback) and GPT-6.1 Sol (max) score 60% and 59%, but Sonnet makes 3.6 times as many tool calls per task. The launch article classifies calls across 19 models and 9,177 rollouts. Of those calls, 38.3% view images and 47.3% prepare or measure them. Image measurement accounts for 5 to 6% of calls for GPT-6 Astra (max) and GPT-6.1 Sol (max), against 25 to 27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max). More calls do not reliably help. In the worked gel example, Grok 4.7 (xhigh) used 27 turns and 20 image views and still answered 50-70 kDa, against a reference of 40-50 kDa. The article suggests it anchored the range on the printed label and leader rather than the faint spot. Claude Sonnet 5.5's interpolation put the label near 53 kDa and the spot near 48.7 kDa, which is consistent with the reference range.
Cost varies widely too. Per-task cost runs from $0.02 for GPT-6 Luna (max) to $1.77 for Grok 4.7 (xhigh), a 99 times difference. According to the article, Sonnet costs about 18 times as much per task as GPT-6.1 Sol (max) for a similar score. The article points to output tokens: Sonnet uses 63.5k per task, including 54.6k reasoning tokens, about 25 times the output of Sol for a similar score. Reasoning makes up most of each model's output. Artificial Analysis lists Gemini 4 Carbon (high), GPT-6 Luna (max), GPT-6.1 Sol (max), GPT-6.1 Terra (max) and GPT-6 Astra (max) as the score-cost Pareto frontier among the models it tested.
By domain, Gemini 4 Carbon (high) leads or shares the lead in three of eight domain groups: plasmid maps, phylogenetics and flow cytometry. It shares the flow cytometry lead with Gemini 4 Argon (high). Claude Opus 5.5 (max) leads on blotting, Claude Sonnet 5.5 (max) on cell counting, and GPT-6.1 Terra (max) on protein structure and binding. The launch article also tested GLM-5.3 (max), a model without native vision, by giving it code execution on the same image files. It loaded images into arrays, ran OCR on thresholded regions and printed pixel coordinates. It got the spot-22 question wrong after 54 turns and 57 code calls with no image views. GLM-5.3-Flash (max) combined code with ten image views and answered correctly in 30 turns. The article also notes that some code calls do no analysis. A commentary or no-op call simply prints text that returns to the model in its next request.
Why it matters
A single-pass score can hide large differences in reliability and cost on image-based scientific work, where a result that changes between runs can send teams toward wrong conclusions.
What you can do
Ask for repeated-run results and cost per task, not single-pass scores, before choosing a model for image-based scientific analysis.
Original source
Where we saw it
Artificial Analysis
AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works