Mixture of experts

In 1991 Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton described a supervised learning system made of many separate networks. Each network learns to handle a subset of the training cases. On a vowel discrimination task, the procedure split the work into simpler subtasks, each small enough for a very simple expert network.¹

In 2017 Noam Shazeer and colleagues at Google put up to thousands of feed-forward sub-networks in one layer. A trainable gating network picks a sparse combination of those experts for each example. One of their models had up to 137 billion parameters and beat the published results they compared it with on language modelling and machine translation, at lower computational cost.²

Later work separated size from work per token more sharply. William Fedus, Barret Zoph and Shazeer routed each token to a single expert and trained sparse models with as many as 1.6 trillion parameters.³ Mixtral 8x7B, an open-weight model, puts eight experts in each layer and picks two for every token. Its authors say it has access to 47 billion parameters but uses 13 billion active parameters.⁴

The pattern is in models you can still call. DeepSeek-V3 is a mixture-of-experts language model with 671 billion parameters in total and 37 billion activated for each token.⁵ Mistral's March 2026 announcement of Mistral Small 4 describes 128 experts, with 4 active per token, 119 billion parameters in total, and 6 billion active per token, or 8 billion once the embedding and output layers are included.⁶

This is not a panel of separate chatbots voting on an answer. In these systems the experts are blocks inside one network, and for any one token most of them do nothing.²⁴ It is also not a dense model, which runs every parameter on every input.³