# Hugging Face publishes free ~70k-word handbook on Mixture-of-Experts training

> Hugging Face released a free handbook on training mixture-of-experts models, citing about 800k GPU-hours of experiments behind the guidance.

- Status: reported (One outlet, not independently verified yet.)
- Topic: Open Source
- Published: 2026-11-06T10:36:15+00:00
- Updated: 2026-11-06T10:36:15+00:00
- Web page: https://ai-news.ikangai.com/story/hugging-face-publishes-free-70k-word-handbook-on-mixture-of-experts-training/
- Seen in: Latent Space
- Change: research (Research)

Hugging Face has published a free handbook of roughly 70,000 words on training mixture-of-experts models. It says the guidance rests on about 800k GPU-hours of experiments, which it puts at around $2M.

The handbook's topics include expert count and sharing, hiding all-to-all communication, and router choice among softmax, sigmoid and sqrt-softplus, with fp32 routing, meaning the routing math is kept in 32-bit precision.

Other topics are auxiliary loss versus loss-free bias, upcycling, and serving with fewer active experts.

Thom Wolf notes that 18 of the top 20 open models are MoE.

## Why it matters

Teams training or self-hosting large open models get a concrete reference on MoE design and serving tradeoffs, which affects compute cost and latency.

## What you can do

Engineering leads planning MoE training or hosting could use the handbook as a checklist when comparing routing and serving choices.

---

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it.

Cite as: IKANGAI AI News, "Hugging Face publishes free ~70k-word handbook on Mixture-of-Experts training", 2026-11-06, https://ai-news.ikangai.com/story/hugging-face-publishes-free-70k-word-handbook-on-mixture-of-experts-training/
