Mixture of experts

A mixture-of-experts layer replaces one big feed-forward network with many smaller ones, the experts, and a router that sends each token to only a few of them. The model stores hundreds of billions of parameters but uses only a few percent per token. The hard parts are keeping the experts evenly used and moving tokens between GPUs.

Key idea: Router scores → top-k experts per token → weighted sum of their outputs; balance the load (auxiliary loss or per-expert bias); shared experts; total vs active parameters
Real world: A hospital with a receptionist and many specialists. Each patient is sent to the two specialists who fit their symptoms best, not to every doctor in the building. The hospital employs many people, but each visit uses only a few.
In an LLM: DeepSeek-V3 has 671 billion parameters, but for each token its router picks 8 of 256 experts per layer, plus one shared expert. Only 37 billion parameters do any work for that token: a cost per token closer to a 37B model than a 671B one.

Where the parameters are

In a dense transformer, most parameters live in the feed-forward network of each block, the MLP. Every token goes through all of it. A bigger MLP stores more knowledge but costs more for every single token.

A layer breaks that link. It replaces the one MLP with $E$ smaller MLPs, the experts, and adds a router: a small linear layer that scores every expert for the current token. Only the top $k$ experts run. Their outputs are added up, weighted by the router’s scores:

y = \sum_{i \in \text{top-}k} g_i \, \operatorname{Expert}_i(x), \qquad g = \text{normalised router scores of the chosen experts}

The attention layers are unchanged. Only the feed-forward part becomes sparse.

ready

experts per token (k)2add a shared expert

router probabilities6 × 8

E1E2E3E4E5E6E7E8The0.070.030.060.420.080.040.060.23cat0.070.450.050.030.080.220.040.05sat0.090.060.500.060.150.050.070.03on0.060.040.080.300.070.050.130.27the0.070.040.060.390.080.050.060.26mat0.050.300.060.040.070.330.090.05

experts run per token2 of 8

expert parameters used25%

Each row of the matrix is one token’s router output: a softmax over the 8 experts. The outlined cells are each token’s top-k. Press play.

A mixture-of-experts layer with 8 experts. The router scores every expert for every token (the matrix); each token goes only to its top k. Press play to send the tokens one by one.

Each expert is an ordinary MLP. The router is just a matrix of shape (model width × number of experts), learned with everything else. Nobody assigns topics to experts; whatever specialisation appears is learned, and it is usually much messier than “the grammar expert” or “the maths expert”. The tidy pattern in the widget is hand-made so you can follow it.

example · the hospital

A receptionist (the router) listens to each patient for a moment and sends them to the two most suitable specialists (k = 2). The hospital can hire more specialists to cover more conditions without making any single visit longer: each patient still sees two. But the building has to be big enough for all the staff, and if every patient is sent to the same cardiologist, a queue forms while others sit idle.

example · one token in Mixtral

Mixtral 8x7B (Mistral AI, 2023) has 8 experts per layer and picks 2 for every token, separately in each of its 32 layers. The token “ cat” might go to experts 3 and 7 in layer 1, experts 1 and 4 in layer 2, and so on. Its name suggests 8 × 7 = 56 billion parameters, but the attention layers and embeddings are shared, so the real total is 46.7 billion, and each token uses 12.9 billion.

Keeping the experts busy

Training an MoE has a built-in trap. An expert that happens to receive a few more tokens early gets more training, becomes a bit better, and so the router sends it even more. The others fall behind. Left alone, the router collapses onto a few experts and the rest are dead weight.

Uneven load also hurts on hardware. Experts sit on different GPUs; if one gets twice its share, its GPU becomes the bottleneck while the others wait. Systems give each expert a capacity (say 1.25 × its fair share) and drop or delay tokens beyond it.

step 1

19%

16%

14%

no balancingauxiliary lossbias (DeepSeek-V3)

busiest expert19%

experts getting nothing0

tokens over capacity3%

At the start the router is random and every expert gets roughly its fair share. Press play and watch what training does.

A router with 8 experts and k = 2, trained for 120 steps on batches of 64 tokens. Bars: share of the batch each expert received at this step (dashed line = fair share, 12.5%). Chart: the busiest expert’s share over training.

Two fixes:

Auxiliary loss (Switch Transformer, Mixtral, Qwen3): add a small term to the training loss that is smallest when every expert gets the same share. It works, but it pulls on the router’s scores in a different direction from the main loss, so a strong balancing term costs quality and a weak one allows imbalance.
Auxiliary-loss-free balancing (Wang et al., 2024, used in DeepSeek-V3): give each expert a bias that is added to its score only for choosing the top k. After each batch, lower the bias of overloaded experts by a fixed step and raise it for idle ones. The weights $g_i$ that mix the outputs still come from the unbiased scores, so the main objective is left alone. DeepSeek-V3 keeps only a very small extra loss to stop extreme imbalance inside a single sequence.

Kimi K3 (2026) uses a variant of the bias method, “quantile balancing”, that computes each bias directly from one batch’s router scores so that, on that batch, every expert would get exactly its quota; the new biases are used from the next step.

Shared experts and fine-grained experts

Two refinements from DeepSeekMoE are now common:

Fine-grained experts: many small experts instead of a few big ones. Picking 8 of 256 offers far more combinations than 2 of 8, so experts can specialise more narrowly.
Shared experts: one or two experts that every token visits, outside the routing. They learn what every token needs (common grammar, frequent patterns), so the routed experts don’t all have to learn it again. DeepSeek-V3, Kimi K2 and Qwen3.5 have one; Kimi K3 has two. Qwen3 has none.

Total versus active

An MoE model has two sizes. Total parameters decide memory: every expert must be stored, because any token might need it. decide compute per token. Model names now often state both: Qwen3-235B-A22B has 235B total, 22B active.

experts per layer256active (k)8

one layer’s experts, with one token’s choice lit (green: shared, always on)

total

671B

active

37.6B

expertsattentionembeddings + dense layers

active share5.6%

memory at 8 bits671 GB

published671B total, 37B active

DeepSeek-V3: 256 experts per layer, 8 active plus 1 shared. Our count from the config gives 671B total and 37.6B active (published: 671B total, 37B active). Each token pays for 5.6% of the weights in compute, but all of them have to sit in memory.

Total parameters (what you must store) against active parameters (what each token uses), computed from each model’s published configuration. Change the number of experts or k to see which number moves.

Model	Released	Total	Active	Routed experts per layer, active	Shared
Mixtral 8x7B	Dec 2023	46.7B	12.9B	8, top-2	0
DeepSeek-V3	Dec 2024	671B	37B	256, top-8	1
Qwen3-235B-A22B	Apr 2025	235B	22B	128, top-8	0
Kimi K2	Jul 2025	1.04T	32.6B	384, top-8	1
gpt-oss-120b	Aug 2025	117B	5.1B	128, top-4	0
GLM-5	Feb 2026	744B	40B	256, top-8	1
Qwen3.5-397B-A17B	Feb 2026	397B	17B	512, top-10	1
DeepSeek-V4-Pro	Apr 2026	1.6T	49B	384, top-6	1
Kimi K3	Jul 2026	2.8T	104B	896, top-16	2

The trend is clear: more experts, smaller experts, and a smaller active fraction. Kimi K3 uses under 4% of its weights per token. Memory, not compute, is now what limits these models, which is why they are often shipped in 8-bit or 4-bit (see model memory).

Expert parallelism

No single GPU holds a trillion-parameter model, so the experts are spread out: each GPU holds a few experts of each layer. In every MoE layer, each token’s hidden state is sent to the GPUs holding its chosen experts (dispatch), processed, and sent back (combine). These are “all-to-all” exchanges, where every GPU talks to every other. The communication can cost as much as the computation, so models are designed around it. DeepSeek-V3, for example, sends each token to experts on at most 4 machines.

going deeper

The router in equations

For a token with hidden state $x$ and router matrix $W_r$ (one row $e_i$ per expert):

s_i = \operatorname{softmax}(W_r x)_i \;\; \text{(Mixtral, Qwen3)} \qquad\text{or}\qquad s_i = \sigma(e_i^\top x) \;\; \text{(DeepSeek-V3)}

DeepSeek-V3 picks experts by $s_i + b_i$ and mixes them with $g_i = s_i / \sum_{j \in \text{top-}k} s_j$ . After each step, $b_i \leftarrow b_i - \gamma$ if expert $i$ was overloaded and $b_i + \gamma$ if underloaded.

The Switch Transformer auxiliary loss, with $f_i$ the fraction of tokens routed to expert $i$ and $P_i$ the mean router probability for expert $i$ :

\mathcal{L}_\text{aux} = \alpha \, E \sum_{i=1}^{E} f_i \, P_i

which is smallest when both are uniform.

Counting parameters

Per MoE layer: attention $+$ router ( $d \times E$ ) $+$ $(E + E_\text{shared})$ experts, each $3 \times d \times d_\text{ff}$ for a SwiGLU expert. Active: attention $+$ router $+$ $(k + E_\text{shared})$ experts. Add the input embeddings and output head ( $2 \times V \times d$ when untied) and any dense layers. For DeepSeek-V3 this gives 671.0B and 37.6B; for Mixtral 46.7B and 12.9B, matching the published figures.

8 experts, k = 2, 64 random tokens per step. Each expert has a popularity term in its score that rises with its load (more tokens, more practice). The auxiliary loss pushes that term back down; the bias method adds a separate bias, moved ±0.08 per step, used only for selection.

In papers

Sparsely-gated MoE: Shazeer et al., 2017. Switch Transformer: Fedus et al., 2021.
Mixtral: Jiang et al., 2024. DeepSeekMoE: Dai et al., 2024.
Auxiliary-loss-free balancing: Wang et al., 2024; in DeepSeek-V3, section 2.1.2.
Qwen3 MoE (no shared experts, global-batch balance loss): Qwen Team, 2025.
Kimi K3 (896 experts, quantile balancing): technical report, 2026. Other figures from each model’s card on Hugging Face.