Hugging Face publishes free ~70k-word handbook on Mixture-of-Experts training
Hugging Face released a free handbook on training mixture-of-experts models, citing about 800k GPU-hours of experiments behind the guidance.

Watch the short
Under a minute, read by an AI anchor. Made .
YouTube loads only when you press play.
Hugging Face has published a free handbook of roughly 70,000 words on training mixture-of-experts models. It says the guidance rests on about 800k GPU-hours of experiments, which it puts at around $2M.
The handbook's topics include expert count and sharing, hiding all-to-all communication, and router choice among softmax, sigmoid and sqrt-softplus, with fp32 routing, meaning the routing math is kept in 32-bit precision.
Other topics are auxiliary loss versus loss-free bias, upcycling, and serving with fewer active experts.
Thom Wolf notes that 18 of the top 20 open models are MoE.
Why it matters
Teams training or self-hosting large open models get a concrete reference on MoE design and serving tradeoffs, which affects compute cost and latency.
What you can do
Engineering leads planning MoE training or hosting could use the handbook as a checklist when comparing routing and serving choices.
Original source
We have no link to the original source for this story yet.
Where we saw it
Latent Space
AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works