816 billion parameters, and only 42 billion show up for each token

TL;DR

• Meta released Muse Spark Open, which has about 816B parameters. Only about 42B are active per token, because each layer picks 8 of its 256 experts.
• Shazeer et al. showed that a trained, sparse gate can send each input to a few experts, so capacity grows much faster than compute.
• Total parameters set the memory bill and active parameters set the compute bill. That split explains how a model this size runs on two desktops.

What happened

On October 15, Meta announced Muse Spark Open. It accepts text, image, audio and video input, produces text, and has a context window of up to 1M tokens. Meta calls it "the most capable US open-weight model to date." That claim rests on Meta's own evaluations. Meta points to its model card and says a full evaluation report is "available soon." Nobody has independently checked the claim yet. The weights ship under the Muse Model License "with limited commercial terms," so the model is open-weight, not open source.

Hugging Face's launch post has the architecture: 62 layers, and "each MoE block has 256 experts, and a router selects 8 experts for each token." According to the post, experts hold about 97% of the parameters, and the BF16 checkpoint is about 1.5 TB. Meta says the model runs "using two Apple M5 Ultra Mac Studios with the latest MLX framework," with up to 512 GB of memory per machine. TechCrunch puts the requirement at "at least 1 terabyte of RAM." Hugging Face also tested community quantizations on a single 512 GB M3 Ultra and measured roughly 18–21 tokens per second at 2–3 bits.

The paper in brief

By 2016, bigger models kept winning, but every extra parameter was used on every example, so training cost grew roughly with model size times data. "Conditional computation" means running only the parts of a network that are relevant to each input. It was an old idea that had never paid off in practice: GPUs handle branching poorly, the sub-networks each get tiny batches, and the gates tended to collapse onto a few favourites. Shazeer et al. added a mixture-of-experts layer to stacked LSTMs: thousands of small feed-forward "experts" plus a gating network that picks the top $k$ experts for each input. They trained everything with plain backprop and added two auxiliary losses to keep the experts evenly used. With the same compute per token, models with up to 137B parameters in the MoE layer beat the best published language-modelling and translation results of the time. This was the paper that turned sparse experts from a theoretical promise into an engineering recipe, and today's large MoE models descend from it.

The math

Setup

Take one token's hidden vector $x$ (in the paper, the output of an LSTM at one position, with dimension 512). The MoE layer returns a vector $y$ of the same size. Training minimizes the usual next-word cross-entropy plus small penalties for uneven expert use.

$$y=\sum_{i=1}^{n}G(x)_{i}\,E_{i}(x)$$

Here $E_1,\dots,E_n$ are the experts. Each is a feed-forward net with one ReLU hidden layer, with identical shapes but separate weights. $G(x)$ is an $n$-dimensional vector of gate weights. This is a mixture model, the same weighted average you've seen in Gaussian mixtures, except the weights depend on the input. The trick is that most entries of $G(x)$ are exactly zero, and when $G(x)_i=0$, you never compute $E_i(x)$.

The gate

$$\begin{aligned} G(x)&=\mathrm{Softmax}\big(\mathrm{KeepTopK}(H(x),k)\big)\\ H(x)_i&=(x\cdot W_g)_i+\mathrm{StandardNormal}()\cdot\mathrm{Softplus}\big((x\cdot W_{noise})_i\big) \end{aligned}$$

$W_g$ is a learned matrix that produces one logit per expert. A second learned matrix, $W_{noise}$, sets how much Gaussian noise each logit gets. Softplus, $\log(1+e^z)$, keeps that noise scale positive. KeepTopK leaves the $k$ largest entries unchanged and sets the rest to $-\infty$. Because $e^{-\infty}=0$, the softmax gives those experts weight zero, and the surviving $k$ weights still sum to 1. The result is an ordinary softmax over a short list. Backprop reaches $W_g$ through the chosen experts' gate values. The paper notes this needs $k>1$, since a softmax over a single survivor always outputs 1 and has no gradient.

Worked example. Take four experts, $k=2$ and noisy logits $H=(2.0,\,1.0,\,0.5,\,-1.0)$. KeepTopK gives $(2.0,\,1.0,\,-\infty,\,-\infty)$. Since $e^{2}\approx7.39$ and $e^{1}\approx2.72$, the gates are $0.731$ and $0.269$, so $y=0.731\,E_1(x)+0.269\,E_2(x)$. Experts 3 and 4 never run.

Total vs. active parameters, with Muse's numbers

If each layer runs $k$ of its $n$ experts, then for every token:

active ≈ (always-on parameters) + $\tfrac{k}{n}$ × (expert parameters).

For Muse, $k/n = 8/256 = 1/32$. Suppose a fraction $f$ of the 816B parameters sit in experts. Solving $(1-f)\cdot816 + f\cdot816/32 = 42$ gives $f\approx0.98$, close to Hugging Face's "~97%", which is good for a back-of-envelope check. Experts then contribute about 25B active parameters and everything else (attention, embeddings and so on) about 17B. The 2% of parameters that are always on make up around 40% of the per-token work.

Compute follows active parameters. A common rule of thumb is about 2 FLOPs per active parameter per token in the forward pass: about 84 GFLOPs here, against about 1.6 TFLOPs for a dense 816B model, roughly 19× less. Memory follows total parameters. The router chooses fresh experts for every token in every layer, so all 256 experts must stay resident. At 2 bytes per BF16 weight, 816B parameters come to about 1.6 × 10¹² bytes, in line with the ~1.5 TB checkpoint. Two 512 GB Mac Studios hold 1,024 GB, which works out to at most about 10 bits per parameter before you count the context cache. So the Macs must be running a lower-precision version. The sources I read don't say which one.

Keeping the experts busy

The paper observed that the gate "tends to converge to a state where it always produces large weights for the same few experts." The favoured experts train faster, so they get chosen even more, and the imbalance feeds itself. The fix is a regularizer:

$$\begin{aligned} \mathrm{Importance}(X)&=\sum_{x\in X}G(x)\\ L_{importance}(X)&=w_{importance}\cdot \mathrm{CV}\big(\mathrm{Importance}(X)\big)^{2} \end{aligned}$$

$X$ is a batch. $\mathrm{Importance}(X)$ is a vector holding each expert's total gate weight over that batch. CV is the coefficient of variation, standard deviation divided by mean, so $\mathrm{CV}^2$ is variance over mean squared: a spread measure that doesn't change with batch size. $w_{importance}$ is a hand-tuned weight. Take four experts with importances $(70,10,10,10)$. The mean is 25 and the population variance is 675, so CV ≈ 1.04 and CV² ≈ 1.08. With $w=0.1$ that adds about 0.108 to the loss. A perfectly even $(25,25,25,25)$ adds nothing.

Equal importance doesn't guarantee equal work, though: one expert could get a few examples with large weights while another gets many with small weights. Counting examples per expert would measure work, but a count is a step function with zero gradient. The paper replaces each 0/1 "was expert $i$ chosen?" with its probability under fresh noise:

$$P(x,i)=\Phi\!\left(\frac{(x\cdot W_g)_i-\mathrm{kth\_excluding}(H(x),k,i)}{\mathrm{Softplus}\big((x\cdot W_{noise})_i\big)}\right)$$

$\Phi$ is the standard normal CDF, and $\mathrm{kth\_excluding}(H(x),k,i)$ is the $k$-th largest noisy logit among the other experts, which is the bar expert $i$ has to clear. Summing $P(x,i)$ over the batch gives a smooth $\mathrm{Load}(X)_i$, penalized by $w_{load}\cdot\mathrm{CV}(\mathrm{Load}(X))^2$. This is the real reason the noise is there: it turns a hard threshold into a differentiable probit curve.

Noise turns a hard routing decision into a smooth probability

interactive

• P(x,i) with noise
• hard top-k, no noise

Hover or drag across the chart to read values.

noise pre-activation (x·W_noise)ᵢ0

noise std softplus(·):

0.693

slope of P at the bar:

0.576

P when 1 unit below the bar:

7.46%

What the results say

• Same compute, more capacity. On the 1-Billion-Word benchmark at about 8M ops/timestep, the 4,096-expert model reached test perplexity 34.1 after 10 epochs, against 44.7–46.0 for compute-matched dense baselines. A larger-budget MoE (28.0) beat the best published model (34.7 at 10 epochs) at similar compute, an 18% reduction.
• Capacity pays off with more data. On a 100-billion-word Google News corpus, 65,536 experts (68B parameters) cut perplexity 39% against the matched baseline (28.9 vs. 47.0). At 131,072 experts, perplexity got slightly worse (29.2), which the authors suggest is "possibly a result of too much sparsity."
• Translation. On WMT'14, MoE models scored 40.56 BLEU on En→Fr and 26.03 on En→De, against 39.22 and 24.91 for GNMT.
• Balancing is necessary. With neither auxiliary loss, perplexity was 39.8 and the busiest expert carried 17.8× the average load. With both losses at 0.1, perplexity was 35.6 and the ratio was 1.14.

Limitations: the hardware ran less efficiently. Low-compute MoE models reached 0.74–0.90 TFLOPS/GPU, against 1.07–1.29 for the dense baselines, and the largest model fell to 0.30. All of this was done with LSTMs, not transformers. The single-pair translation models used a different, "strictly balanced" gate (Appendix F), not noisy top-k. The authors admit the top-k cutoff creates "theoretically scary discontinuities" that they hadn't seen cause trouble. And the multilingual model regressed on English→Korean.

Connecting the dots

Muse Spark Open uses the same basic design: a learned router picks 8 of 256 feed-forward experts per token, and only the chosen ones run. That is why 816B total parameters cost about 42B per token. The paper is a foundation Meta builds on, not a description of Meta's method. The posts I read don't say how Muse's router computes its weights, whether it adds noise, or how it balances load. Later work changed these details. Switch Transformers, co-authored by Shazeer, simplified the routing so each token goes to a single expert. So Meta's choices may differ from noisy top-k and CV-squared losses.

The scale differs too. The paper's language-model experts had about 1M parameters each and sat in one layer between LSTMs. Muse's experts project 8192→4096→8192, which is about 67M parameters if they are just two matrices, and they sit inside a 62-layer transformer. The paper's "shrinking batch" problem also carries over. With $k/n=1/32$, a batch of $b$ tokens gives each expert only about $b/32$ of them. The paper fixed this by pooling batches across devices. A single person chatting on a desktop has no such pool, so every expert weight is read from memory to do very little arithmetic. That makes memory size and bandwidth the constraint, not raw FLOPs.

Why it matters

For practitioners, the rule is simple: price MoE models by active parameters for speed and by total parameters for hardware. "Runs on two Macs" means you need a terabyte of fast memory and a quantized model, not that it runs on a laptop. For research, the paper's balancing problem is still open. Collapse, uneven load and routing instability are active topics, and open weights let outsiders measure how evenly Muse's experts are really used. For policy, "open-weight" here comes with a license that has limited commercial terms, and the headline capability claim is still Meta's own.

Things to watch: the promised full evaluation report and independent benchmark runs, and whether Meta publishes its router and load-balancing details.

Go deeper

• The paper on arXiv (PDF). Section 2.1 and Appendix A are the core.
• Meta's announcement: claims, license and the two-Mac Studio setup.
• Hugging Face launch post: architecture, checkpoint formats and quantization tests.
• TechCrunch coverage.
• Switch Transformers by Fedus, Zoph and Shazeer (2021), the main follow-up that simplified routing for transformers.