# Core Automation trains one MoE model across two clusters 9,000 miles apart

> Core Automation reports training a roughly 32B-total mixture-of-experts model across two clusters about 9,000 miles apart, using different accelerators.

- Status: official (From the company, authors or authority themselves, with a link to their own page.)
- Topic: Research
- Published: 2026-11-09T13:31:34+00:00
- Updated: 2026-11-09T14:12:32.767821+00:00
- Web page: https://ai-news.ikangai.com/story/core-automation-trains-one-moe-model-across-two-clusters-9-000-miles-apart/
- Seen in: Import AI (Jack Clark)
- Change: research (Research)
- Applies to: B300 GPUs, GB300 GPUs

## What changed

Core Automation trained one roughly 32B-total mixture-of-experts model across two clusters about 9,000 miles apart, instead of a single site.

Core Automation, a startup founded by senior people from OpenAI and Google, says it trained a sparse mixture-of-experts model with roughly 32B total parameters and 32x sparsity across two clusters about 9,000 miles apart. One cluster has 160 B300 GPUs in the US and the other has 160 GB300 GPUs in Asia. The details appear in a post by Rohan Anil dated November 2, 2026, titled The Hitchhiker's Guide to Training on Compute Islands. The company's central claim is that the hard part of frontier training is no longer raw compute. It is turning scattered clusters into one usable training run.

The post's motivation is supply. It argues that power, grid equipment and land constrain deployment more than chips do. Much of the available compute comes as clusters of a few hundred to a few thousand GPUs spread across many providers. Its proposed answer is to let each cluster run on its own schedule. Within a site, training uses the fast interconnect heavily, while updates are exchanged across sites on a slower schedule. Each site trains on its own data shard with local expert parallelism, and the microbatch and gradient-accumulation settings were tuned separately so both sites take similar time per step. The two sites differ in accelerator, parallelism scheme and total batch size. The post presents this as a feature, saying it lets a team mix hardware generations to reach a given compute budget.

The sites exchange updates every 50 steps, a scheme designed around the link between them, which the post describes as a bidirectional 25 Gbps connection. Rather than sending the whole model at once, the team sent it in pieces: 'we synced it fragment by fragment, staggered across each 50-step window.' A late-arriving update can still be committed. Each site runs a variant of Distributed Shampoo as its local optimizer. The values exchanged between sites are compressed with stochastic rounding plus error feedback. The post credits the fragment-wise and overlapped update idea to earlier work it names, Streaming DiLoCo and Decoupled DiLoCo. It also describes a separate networking effort aimed at tail latency: staggering transfers, reusing warm connections, splitting payloads into chunks that any healthy connection can carry, re-requesting only missing pieces, and launching redundant transfers when a deadline passes. Each of these is simple, the post says, and the gains come from combining them.

The two-site run was compared with standard training at the same compute. The post identifies that standard run as the largest the team did.

The post also tests whether pairwise exchanges can replace the all-reduce, a technique it calls butterfly mixing. In that experiment, 64 GB300 GPUs at one site were split into eight independent cells of eight GPUs. Each cell trained its own copy of the 32B-parameter model. Cells exchanged fragments in pairs, with each fragment exchanged six times per 50-step window, about once every eight steps. After three rounds, every cell's update had reached all eight cells. Each exchange produces a candidate update that is blended into the training weights two steps later at equal weight. The post says training quality matched the baseline. It also notes that all eight cells ran at one site, so the experiment does not show how pairwise exchange behaves over long links.

The post frames all of this as groundwork for training in orbit. It cites Google's Project Suncatcher, a research program on constellations of solar-powered TPU satellites. A first step would be pretraining a small model across a few satellites and comparing it with a ground baseline. The post adds that a trillion-parameter orbital run is much further off. It reports no orbital training of its own.

## Why it matters

If distributed training across mixed hardware matures, smaller groups may build large models without a single hyperscale data center. The company argues that power, grid equipment and land, rather than chips, are the main bottlenecks.

## Original source

- [The Hitchhiker’s Guide to Training on Compute Islands | Core Automation](https://www.coreauto.com/hitchhikers-guide-to-training-on-compute-islands)

---

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it.

Cite as: IKANGAI AI News, "Core Automation trains one MoE model across two clusters 9,000 miles apart", 2026-11-09, https://ai-news.ikangai.com/story/core-automation-trains-one-moe-model-across-two-clusters-9-000-miles-apart/
