AI News
All stories
Reported

Anthropic's Fable 5.2 completes 41.3% of real freelance projects in Remote Labor Index

Fable 5.2 completed 41.3% of real freelance projects to a professional standard in the Remote Labor Index, up from a top score of 2.5% a year earlier.

Models 路 路 3 min read

Watch the short

Under a minute, read by an AI anchor. Made .

YouTube loads only when you press play.

Anthropic's Fable 5.2 now leads the Remote Labor Index, a benchmark built by Scale Labs and the Center for AI Safety (CAIS), with an automation rate of 41.3%. Scale Labs announced the result on X on November 9, calling it 'a major leap in AI's ability to handle real-world work.' The company said it nearly doubles the next-best model, Opus 5.5, at 21.25%. The benchmark launched with a top score of 2.5%, set by Manus, so Fable 5.2 completes more than 16 times as many projects to a professional standard as the best agent did at the start.

The benchmark is built from 240 self-contained projects sourced from 358 verified freelancers on Upwork. Each project includes a brief, the input files, a human deliverable that a client accepted, and the reported completion time and cost. The projects total $143,991 in value. The average project is worth $632.60, but the median is $200. The average human completion time is 28.9 hours, against a median of 11.5 hours. Agents work in a virtual workspace with software such as Blender, GIMP and Audacity, plus tools for generating images, speech and video, and each task has a generation budget of $30. A project counts as automated when a reviewer rates the output at least 2 on a three-point scale, meaning it would be accepted by a reasonable client. The headline figure is simply the share of projects that clear that bar.

According to The Decoder, Fable 5 was at about 16% in July, and its direct predecessor, Fable 5.1, sits just under 18%. Fable 5.2 is therefore roughly two and a half times Fable 5's July score, four months later. The leaderboard lists GPT 6 Astra at 20.83% and Gemini Argon 4 at 17.08%. However, the leaderboard notes that there is no confidence interval for Automation Rate, so the difference between two close scores is not established by the published figures.

The gains are uneven across job types. Scale Labs said the biggest improvements came in audio and music, architecture and interior design, and 3D modeling. Industrial product design and animation remain far from solved. Nearly 50 of Fable 5.2's successful projects were ones that none of the earlier agents had completed. The leaderboard also lists the main reasons agents fail: poor quality, with agents producing amateur-looking work, at 45.6%; incomplete deliverables, such as truncated videos or missing files, at 35.7%; technical and file problems at 17.6%; and inconsistencies across files at 14.8%.

Every judgment is made by trained human experts who open the submitted files in the relevant professional software and compare them with the human deliverable. CAIS tried to automate that review with another AI, and according to The Decoder, it failed. The AI reviewers could barely operate the software, were fooled by submissions that looked polished but were technically flawed, and rated the models far too generously. Scale reports a 94.4% inter-annotator agreement for the Automation Rate, produced by a majority vote of three independent evaluators.

The benchmark also leaves out parts of freelance work. The leaderboard says it excludes projects that require direct client interaction, such as customer service and tutoring, as well as physical labor and long-term evaluation such as SEO. The Decoder described the test as measuring individual assignments rather than whole jobs. The leaderboard scores against a private set of 230 projects. The 10 public projects are released for qualitative analysis, and the leaderboard notes a risk that future models may train on them.

What changed

Fable 5.2 now leads the Remote Labor Index with a 41.3% automation rate, nearly doubling the next-best model, Opus 5.5, at 21.25%.

Type: Research 路 Applies to: Fable 5.2, Opus 5.5

Why it matters

Clearly defined digital services that are easy to compare and swap out may face pricing pressure first, since agents can now complete a sizable share of such assignments.

What you can do

Review which remote or freelance deliverables are easy to specify and verify, and where human responsibility for results still adds value.

Original source

Further reporting

Where we saw it

1 source

THE DECODER

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works