# Artificial Analysis Endpoint Accuracy Index v1.1: 24 of 103 endpoints trail reference

> Artificial Analysis says 24 of 103 serverless endpoints score statistically below its self-hosted reference in the updated Endpoint Accuracy Index v1.1.

- Status: official (From the company, authors or authority themselves, with a link to their own page.)
- Topic: Research
- Published: 2026-11-09T00:00:00+00:00
- Updated: 2026-11-11T02:10:04.055925+00:00
- Web page: https://ai-news.ikangai.com/story/artificial-analysis-endpoint-accuracy-index-v1-1-24-of-103-endpoints-trail-refer/
- Seen in: Artificial Analysis
- Change: research (Research)
- Applies to: Endpoint Accuracy Index v1.1, AA-LCR-25, BFCL-500, HLE-250

## What changed

Artificial Analysis's Endpoint Accuracy Index moves from v1.0 to v1.1, where AA-LCR-25 uses dataset version 1.1 with corrected answer keys and an updated equality-checker LLM, and repeats per question rise from 10 to 25, so raw AA-LCR-25 scores are not directly comparable with v1.0.

Artificial Analysis reported on 9 November 2026 that 24 of the 103 serverless endpoints it benchmarked score statistically below its self-hosted reference under version 1.1 of its Endpoint Accuracy Index. The other 79 are in line with it. The index is relative. Each endpoint runs three evaluations, and each score is expressed as a percentage of the reference deployment's result. The three scores are averaged with equal weight. An endpoint counts as below the reference only when the whole of its 95% confidence interval sits under 100. The 24 is therefore a count of gaps the company considers distinguishable from noise, not a count of every endpoint that scored lower.

The three evaluations are BFCL-500 for tool calling (500 tasks), HLE-250 for scientific reasoning (250 tasks), and AA-LCR-25 for long context recall, with about 100k input tokens per question. The company runs each reference on the lab's official weights at the recommended precision, using vLLM or SGLang on NVIDIA B200, B300 or GB200 systems. It runs each model's highest supported reasoning mode and each endpoint's highest supported output length and context window. The v1.1 results were measured between 8 October and 4 November 2026.

The main change in v1.1 is to AA-LCR-25. That evaluation moves to dataset version 1.1, which corrects the answer keys for two questions and uses an updated equality-checker LLM. As a result, raw AA-LCR-25 scores are not directly comparable with v1.0. The number of repeats per question also rises from 10 to 25. Artificial Analysis says this matters because AA-LCR-25 had the widest intervals in v1.0, at ±4.6 points on average, against ±2.9 for BFCL-500 and ±1.6 for HLE-250. Noise on long context recall could hide real differences in the other two. Across 48 endpoints and references scored under both versions, the average AA-LCR-25 interval narrows from ±4.7 to ±2.6 points. On the composite index, the interval narrows from ±5.5 to ±4.3 points across 45 endpoints, and the composite moves by 1.6 points on average.

Coverage now spans six open weights models. GLM-5.3 Flash and Muse Spark Open are added, DeepSeek V4.1 Flash 1017 replaces DeepSeek V4 Pro, and GLM-5.2 leaves. GLM-5.3, Kimi K3 and gpt-oss-120b carry over. The results table shows 22 of 22 DeepSeek V4.1 Flash 1017 endpoints in line with the reference. It shows 16 of 26 GLM-5.3 endpoints in line, with 10 below. For GLM-5.3 Flash, 17 of 22 endpoints are in line and 5 are below. For gpt-oss-120b, 11 of 16 are in line and 5 are below. For Kimi K3, 10 of 12 are in line and 2 are below. For Muse Spark Open, 3 of 5 are in line and 2 are below.

The company groups the causes of below-reference results into four kinds. Endpoints that cap output at 16K to 32K tokens cut off long HLE-250 reasoning before an answer appears. Some endpoints produce markedly fewer reasoning tokens than the reference at the same settings, with no truncation. Endpoints that parse or format tool calls differently fail BFCL-500 tasks that the reference passes. Fixed request timeouts and streams that end early remove the longest reasoning answers. One endpoint scores below the reference on HLE-250 and AA-LCR-25 without any truncation or reduced reasoning that the company could observe. Endpoints with a known cause carry a note on their provider page describing what was observed.

The GLM-5.3 Flash results show how these causes play out. Two endpoints cap output at about 32K tokens, and about half of their HLE-250 responses hit the cap before an answer. They score 22.3% and 28.5% on HLE-250, against 55.4% for the reference. Two other endpoints produce about 30% fewer reasoning tokens per response without truncation, scoring 48.4% and 48.6%. Two endpoints return object-typed tool arguments as JSON-encoded text, which fails those calls. All three endpoints serving FP4 weights are in line with the FP8 reference, scoring 97.2 to 104.2.

Muse Spark Open is measured against a BF16 reference. Three endpoints are in line, scoring 99.5 to 99.9, including one serving FP4 weights. The other two fail only on tool calling. They return arguments that the tool defines as text, such as booking IDs, as numbers. They score 49.9% and 50.7% on BFCL-500, against 59.1% for the reference. Their HLE-250 and AA-LCR-25 scores are within 2 points of the reference.

For GLM-5.3, the three scores below 80 come from output caps and reasoning loops. A 16,384-token cap brings HLE-250 to 7.6%, and a 32,000-token cap brings it to 24.8%, against 63.6% for the reference. On a third endpoint, most HLE-250 responses fall into repetitive reasoning and reach the output limit, scoring 19.6%. Three other endpoints score 50.7% to 51.9% on BFCL-500, against 57.3%. The causes are differences in how they order arguments, corrupt parameter names, or return object arguments as text. Five of the six endpoints serving FP4 weights are in line with the FP8 reference, scoring 97.7 to 99.3. The sixth is the endpoint with reasoning loops.

Kimi K3 shows two other failure modes. Where requests are cut off after about 10 minutes, about 9% of HLE-250 requests end before the model answers. One Kimi K3 endpoint falls to 65.3% against 72.1%. Another Kimi K3 endpoint reuses tool call IDs across conversation turns, which conflicts with the OpenAI tool calling format, and scores 42.1% on BFCL-500 against 55.2%. All 22 DeepSeek V4.1 Flash 1017 endpoints are in line, scoring 97.5 to 101.4.

The gpt-oss-120b results carry a caveat that the company raises itself. Its confidence intervals average about ±7.5 points, which is wider than those of the other models. The index expresses each score relative to the reference, and that reference scores lower, at 17.0% on HLE-250 and 37.1% on BFCL-500. The same run-to-run variation therefore makes up a larger share of each score. Two endpoints score 21.3% and 22.3% on BFCL-500. One returns function calls as plain text, and the other fills optional arguments with placeholder values such as 0 or -1. With output capped at 16,384 tokens, HLE-250 drops to 11.0%, against 17.0%.

Artificial Analysis also compared v1.0 and v1.1 on the same endpoints. Of the 45 endpoints scored under both versions, 10 changed classification. Six moved from in line to below the reference, and four moved from below to in line. The company says results from different index versions are not directly comparable. It will move to a new version when the suite changes.

## Why it matters

Two endpoints serving the same open-weights model can differ noticeably in accuracy, so price and speed alone are not a sound basis for choosing a provider. According to the company, tool-calling and output-length settings are the most common sources of loss.

## What you can do

Benchmark the specific hosted endpoints you rely on for tool calling and long outputs before committing to them. Don't assume they match the vendor's model results.

## Original source

- [Endpoint Accuracy Index v1.1: Tighter long context intervals and expanded model coverage | Artificial Analysis](https://artificialanalysis.ai/articles/endpoint-accuracy-index-v1-1)

---

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it.

Cite as: IKANGAI AI News, "Artificial Analysis Endpoint Accuracy Index v1.1: 24 of 103 endpoints trail reference", 2026-11-09, https://ai-news.ikangai.com/story/artificial-analysis-endpoint-accuracy-index-v1-1-24-of-103-endpoints-trail-refer/
