Making language models bigger tends to make them better, but running a model with hundreds of billions of parameters for every single word is extremely expensive. Mixture of Experts (MoE) is an architecture idea that lets a model have a huge number of parameters while using only a small fraction of them for each token. It is one of the main reasons some very large models are affordable to run.
The idea in plain words
Imagine a hospital with many specialists. When a patient arrives, a receptionist decides which two specialists should see them rather than sending the patient to every doctor. The hospital has a lot of expertise in total, but each patient uses only a little of it.
In an MoE model:
- The experts are separate small neural networks (usually feed-forward layers).
- A router (gating network) looks at each token and chooses which experts will process it.
- Only the selected experts run; the others stay idle for that token.
Dense versus sparse models
A dense transformer uses all of its parameters for every token. An MoE model is sparse: only a subset of parameters is active per token. The distinction matters:
- Total parameters: all the weights stored, which determine memory needs.
- Active parameters: the weights actually used per token, which determine computation per token.
For example, a model might hold eight experts per layer and route each token to two. Total parameters are large, but compute per token resembles a model with the size of two experts plus the shared parts.
Where experts sit in a transformer
A standard transformer layer has an attention block and a feed-forward network (FFN). MoE models typically replace the single FFN in some or all layers with a set of expert FFNs and a router, while keeping attention shared. Each layer makes its own routing decision, so different tokens can use different expert combinations at each layer.
How the router works
For each token vector x, the router computes a score for each expert, normally with a small linear layer followed by softmax. It selects the top-k experts (often k = 1 or 2), and the final output is the weighted sum of those experts' outputs, with weights from the router scores.
import numpy as np
def softmax(z):
e = np.exp(z - z.max())
return e / e.sum()
rng = np.random.default_rng(0)
d_model, n_experts, top_k = 8, 4, 2
W_router = rng.normal(size=(d_model, n_experts))
experts = [rng.normal(size=(d_model, d_model)) for _ in range(n_experts)] # toy linear experts
def moe_layer(x):
scores = softmax(x @ W_router)
chosen = np.argsort(scores)[::-1][:top_k] # top-k experts
weights = scores[chosen] / scores[chosen].sum() # renormalise
output = sum(w * (x @ experts[i]) for w, i in zip(weights, chosen))
return output, chosen
token = rng.normal(size=d_model)
out, used = moe_layer(token)
print("experts used:", used.tolist())In real models the experts are full feed-forward networks, and the computation is batched efficiently across thousands of tokens.
Training challenges
Load balancing
If the router sends most tokens to a few popular experts, those become overloaded while others are rarely trained, wasting capacity. Training adds an auxiliary load-balancing loss that encourages tokens to be spread evenly, and some designs set a capacity limit per expert, dropping or rerouting overflow tokens.
Training stability
Routing decisions are discrete, which can make optimisation less stable. Techniques include adding noise to router scores, regularising router outputs and careful initialisation.
Specialisation
Experts do not necessarily specialise in human-meaningful topics like "chemistry" or "poetry". Studies suggest they often specialise in lower-level patterns such as syntax, token types or positions. Interpreting what experts learn is an active research area.
Benefits
- More capacity per unit of compute: a model can store more knowledge and skills without a proportional rise in per-token cost.
- Faster training for a given quality compared with dense models of the same total size, in many reported cases.
- Faster inference per token relative to a dense model of the same total parameter count.
- Scalability: experts can be distributed across devices.
Drawbacks and trade-offs
- Memory: all experts must be stored and available, so you need memory for the total parameter count even though only some are active.
- Communication overhead: in distributed training and serving, tokens must be sent to the devices holding their chosen experts, which can bottleneck.
- Complex serving: batching and latency can be harder because different requests activate different experts.
- Fine-tuning difficulty: MoE models can overfit on small datasets and be trickier to fine-tune, although methods have improved.
- Engineering complexity: more moving parts than dense models.
- Uneven utilisation if load balancing is imperfect.
Why users should care
You may see model descriptions like "a 46B-parameter model with 12B active parameters". The active count suggests the speed and cost per token, while the total count suggests memory needs. When choosing models to run on your own hardware, check both numbers. A sparse model can be fast but still demand a lot of GPU memory.
MoE and quantisation
Because memory is the main constraint, quantisation (storing weights with fewer bits) is especially valuable for MoE models. Some systems also offload rarely used experts to slower memory, trading speed for footprint.
Related ideas
- Conditional computation: the broader idea of activating only parts of a network depending on the input.
- Switch Transformers: a design that routes each token to a single expert to simplify and speed up routing.
- Expert choice routing: experts select tokens instead of tokens selecting experts, improving balance.
- Adaptive depth and early exit: other ways of saving compute by skipping layers.
Frequently Asked Questions
Are MoE models always better than dense models?
No. They offer an efficiency advantage at large scale, but add complexity and memory requirements. For small models or small deployments, dense models are often simpler and just as good.
Do experts specialise in subjects like law or medicine?
Not usually in a clean, human-readable way. Specialisation tends to be more abstract.
Does a bigger MoE model need more memory?
Yes. Memory scales with total parameters, not active parameters.
Can I run an MoE model on a laptop?
Small MoE models can run with quantisation, but large ones need significant memory.
Conclusion
Mixture of experts lets models grow their total knowledge capacity while keeping per-token compute modest, by routing each token to a few specialised sub-networks. The price is higher memory use and more complex training and serving.