AI News
All stories
Official

Anthropic describes a "chaos loop" used to harden its Claude Docs service

Anthropic says it tested the service behind Claude Docs with a chaos loop and reports 60 server errors across billions of requests.

Products · · 4 min read

Anthropic says it hardened the service behind Claude Docs by letting Claude run a stress-testing loop against a staging copy for about two weeks. The launch, it reports, produced 60 server errors across 2 billion requests and no rollback. Vivek Nair, who wrote the engineering post, says Claude Docs was released on September 16, 2026, and that the whole Docs service ran on about 50 memory-optimized pods. Nair calls the method a "chaos loop" and presents it as one of two decisions that got the launch through.

Claude Docs is a document editor where people and Claude work in the same doc. A Claude session from chat or Claude Code can open it and read, write and comment live. Before launch, the service ran as a proof of concept on a single node used by a few thousand Anthropic staff. Nair says that told the team nothing about whether the service would survive launch. So on September 4 he had Claude drive a staging cluster that mirrored production at over a million synthetic docs a day, and break core components on purpose.

The loop was a Claude Code session that picked a scenario, ran the harness, read telemetry, proposed a fix, shipped it to staging and repeated. That coordinator handed most of the work to sub-agents, about 11 at a time in the busiest week and 30 at peak. Telemetry came through the Datadog MCP server. Scenarios included 10M new docs created in a single day, 22,000 live connections reconnecting at once, and a hot doc with hundreds of viewers while a server was killed mid-load and the database stalled. Nair set two invariants Claude had to respect throughout. Docs had to open almost instantly even with long histories, and once the server acknowledged an edit, tests had to show it was still there. Changes that were hard to undo, such as schema or key layout changes or how a doc is stored, required his approval.

The design choices follow from one idea: Claude can only fix what it can measure. Every database query reported its name and how many rows it scanned versus returned, so a bad query could be spotted and rewritten. Nair's team wrote the service in Rust and avoided runtimes with garbage collection, because a collector would add memory and latency noise on its own schedule. They also kept dependencies to a minimum and vendored the few they needed, so they could measure that code too. Nair says the team did not benchmark Rust against Go, so the choice reflects their experience rather than a comparison.

The memory work shows how this played out. Early on, one large doc used about 3 GB on its pod. Claude's first fix was to shrink a cache so docs would be evicted sooner, but the next run showed the memory sat in a cache inside a dependency, so that change was dropped. Finer-grained profiling then found that each keystroke copied the doc into several data structures, adding a few MB every time. Claude switched them to copy-on-write so they shared one reference. After about three days, that doc peaked around 180 MB, and the estimate of what launch would need fell from more than 500 memory-optimized pods to about 50.

At launch, the first million docs appeared in about 30 hours, and the count passed 4.6 million by September 22. Nair says the team rolled production 629 times in September and never rolled back. Seven of the ten scenarios they had rehearsed on staging happened in production. Fixes had been shipped behind runtime flags, so when part of the infrastructure came under pressure, the team usually flipped a flag.

Nair argues the approach only became workable recently. He says older models could read telemetry but began forgetting the optimization goal ten or twenty iterations into a long loop. Other Anthropic employees call this goal drift, and he had to restate the goal. Newer frontier models, he says, held all the invariants across many context compactions without being reminded. He also used an internal research model he describes as roughly comparable to Claude Fable 5.2.

Nair puts the inference cost at roughly $125,000 at Claude Fable 5.2's API list price as of November 2026, spent from the end of August to launch day. He guesses a team of engineers would have needed about six months for work on this scale a couple of years ago. He says a good chunk of the spend went to infrastructure that only makes sense at Docs' scale, and to tokens burned while he was still working out the process. He thinks about half could have been saved. For a small service, he estimates a two-week push at $3,000 to $10,000 at the same list price, after stripping out work a smaller stack would not need. He says he has not actually run that estimate.

For teams that want to try it, Nair's steps are these. Give Claude access to telemetry through an observability provider's MCP server. Have it write a load script that sends ten times real traffic at a staging copy while it kills instances, stalls the database and deploys mid-spike. Choose a stack Claude can measure easily. Before starting, write down the key technical invariants and hand them over. For any change that is hard to undo, have Claude stop and offer a few options with a recommendation.

What changed

Claude Docs, a document editor where people and Claude work in the same doc, was released on September 16, 2026, after its service was hardened with a Claude-run stress-testing loop.

Type: New capability · Takes effect: Wednesday, September 16, 2026 · Applies to: Claude Docs

Original source

Where we saw it

1 source

claude.dev

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works