# Test finds AI coding agents needed human cleanup on nearly three in four tasks

> In a test of real work, AI coding agents needed human cleanup on nearly three out of four tasks, which shows that producing code is not the same as delivering finished work.

- Status: reported (One outlet, not independently verified yet.)
- Topic: AI in Practice
- Published: 2026-11-15T20:29:01+00:00
- Updated: 2026-11-15T21:10:00.773694+00:00
- Web page: https://ai-news.ikangai.com/story/test-finds-ai-coding-agents-needed-human-cleanup-on-nearly-three-in-four-tasks/
- Seen in: THE DECODER
- Change: research (Research)

According to the paper, the best of ten AI coding agents fully completed only 26.4 percent of tasks in Co-Design Bench, a new benchmark from researchers at Alibaba and several Asian universities. That leaves nearly three in four tasks with at least one requirement unmet. The top model, GPT-6 Astra, met about 70 percent of task requirements on average, with a mean strict completion of 69.9 percent. The gap between those two figures matters most. An agent can do most of a job and still hand back something a person has to fix.

The benchmark's authors built it from real user behavior rather than synthetic prompts. It contains 273 tasks drawn from 37,463 deduplicated queries across 26,314 real user coding sessions. In each task, an agent receives an existing application and a user's request, sometimes with an image, audio, video or document attached. Expert annotators wrote 2,331 hidden acceptance criteria that the agent never sees. The benchmark has two tracks: Multimodal Editing, with 100 localized edits, and Repository Editing, with 173 tasks across multi-file frontend projects. According to The Decoder, the researchers favored requests where users had to rephrase or roll back changes, so the test is weighted toward tasks that already caused trouble in practice.

Scoring is strict by design. Each hidden criterion is graded PASS, PARTIAL or FAIL, and a PARTIAL earns nothing under strict completion. A task's strict score is the share of its criteria that pass, while the full-pass rate counts only tasks where every criterion passes. An AI judge tests the finished application in a browser and gathers evidence for each requirement. A separate verifier then challenges every positive verdict, looking for stubs, hardcoded results and unreachable features. The 69.9 percent and 26.4 percent figures differ for this reason: the average credits partial work, while the full-pass rate counts only finished tasks.

Scores fall as a request combines more kinds of change. The paper reports that all ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. The Decoder reports that GPT-6 Astra fully solves only 41 percent of multimodal tasks and just under 18 percent of repository tasks. The authors argue these intertwined requirements are normal in real projects.

Extra effort does not reliably buy correctness. The paper's page gives one comparison on the Multimodal track. GLM-5.2 used a median of 69 turns and 7.29 million tokens per task, while GPT-6 Astra used 32 turns and 1.24 million tokens. The first scored 57.2 percent strict completion, against 75.4 percent for the second. The Decoder notes that GLM-5.2 used nearly six times as many tokens on smaller changes and still scored significantly worse. Because the page describes these profiles as effort rather than causal evidence, the token counts show where compute went, not why it failed to help.

The testing environment also changes the results. The Decoder reports that in a separate test, Qwen3.8-27B scored either 75 or 56 percent on larger overhauls, depending on the harness. The harness is the program through which an agent reads, writes and executes code. Which of two models came out ahead also depended on the environment. The authors recommend reporting the model and the working environment together. The paper's page also notes that the smaller Qwen3.8-27B climbs from fifth to second place on the Repository track while another endpoint drops, so a single combined ranking can hide setting-specific strengths.

The benchmark also has limits, as The Decoder notes. It tests one fixed request per task, so follow-up questions and mid-task corrections from users are not part of the evaluation, and the AI judges can make mistakes. The Decoder cites a separate METR study finding that project leads' real acceptance rate of AI-generated code was roughly 24 percentage points below scores on SWE-bench Verified. Co-Design Bench's authors justify the new test by pointing to older benchmarks, where models hit scores above 99 percent. The paper's page, however, also shows that on WebCompass-edit, switching one model from single-shot to agentic execution raised its graded completion from 73.6 percent to 95.1 percent. The execution setup, in other words, carries real weight in the results.

## Why it matters

Teams measuring coding agents by code output alone may overstate productivity gains, because review and cleanup still consume human time.

## What you can do

Track how much human cleanup agent-written work needs before counting it as delivered.

## Original source

- [(2610.35138) Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models](https://arxiv.org/abs/2610.35138)
- [Co-Design Bench · Multimodal collaborative coding evaluation](https://codesignbench.github.io/)

## Further reporting

- [AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test](https://the-decoder.com/ai-coding-agents-still-need-human-cleanup-on-nearly-three-out-of-four-tasks-in-a-real-world-test/)

---

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it.

Cite as: IKANGAI AI News, "Test finds AI coding agents needed human cleanup on nearly three in four tasks", 2026-11-15, https://ai-news.ikangai.com/story/test-finds-ai-coding-agents-needed-human-cleanup-on-nearly-three-in-four-tasks/
