llama.cpp reaches 1.0 with expert caching for MoE models larger than GPU memory
llama.cpp and ggml have reached 1.0, and the llama.cpp release adds a cache for frequently used experts in mixture-of-experts models.
llama.cpp and ggml have both reached version 1.0. The llama.cpp release adds a way for users of mixture-of-experts models larger than their GPU memory to keep a cache of frequently used experts. The 1.0 label comes with a stability declaration and the removal of 35 deprecated functions from llama.cpp's public header. Hugging Face announced the v1.0.0 release for the ggml-org projects and said the goal is to keep broad platform and model-type coverage while keeping the runtime "robust and lean" for downstream integrations. SGLang also announced its own 1.0 at the SGLang Summit.
The caching feature targets a specific bottleneck. A mixture-of-experts model routes each token to a small subset of its expert networks, so only some of its weights are used at a time, but the full set still has to be stored and moved. When the model does not fit in GPU memory, experts that are not resident must be loaded before they can run. A cache that holds the experts a workload selects most often avoids repeated loading and dispatch overhead, provided the routing actually repeats. A commenter in a LocalLLaMA discussion of the release argued that this advantage may shrink as models become sparser, because expert selections become less concentrated. The same thread included a question about whether MoE streaming methods used in other projects could be added to upstream llama.cpp, on the argument that many performance-focused forks make deployment choices harder. The announcement coverage does not include speed or memory figures for the cache, so how much it helps on a given model and hardware is not established.
Maintenance was the other concern raised in the discussion. A commenter pointed out that llama.cpp has "a ton of open issues and PRs" and asked for more maintainers. Hugging Face also announced Llama desktop apps built on llama.cpp, which it said are meant to cut the setup of running local models down to a few clicks.
What changed
llama.cpp and ggml reached version 1.0, with llama.cpp adding a cache for frequently used experts in mixture-of-experts models and removing 35 deprecated functions from its public header.
Type: New capability · Applies to: llama.cpp, ggml
What you can do
Teams that run local MoE models could test the 1.0 expert cache on their own hardware.
Original source
We have no link to the original source for this story yet.
Where we saw it
Latent Space
AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works