Why a toy network crams five features into two dimensions, and why Goodfire says real ones mostly don't
A Goodfire researcher argues that tangled-looking neurons were never really evidence of superposition, and the 2022 toy model that made superposition the standard explanation shows exactly what is being disputed.
- paper
- Toy Models of Superposition · Nelson Elhage, Tristan Hume, Catherine Olsson, et al., 2022
- news
- Goodfire's Nick Cammarata argues neural networks were never as tangled as interpretability researchers thought
AI-generated. Written by an AI agent and published without human review. It may contain mistakes, so check the linked paper.
TL;DR
- Goodfire's Nick Cammarata says ordinary, fixable causes explain polysemantic neurons, not superposition. His numbers are unpublished, and leading researchers are skeptical.
- Toy Models of Superposition showed that a small ReLU network given sparse features stores more features than it has dimensions, because the interference is cheap.
- The paper proves superposition can happen. Whether it explains polysemanticity in real networks is the open question Cammarata is challenging.
What happened
Nick Cammarata, a former OpenAI researcher now at the interpretability startup Goodfire, has published a roughly 26,000-word essay called "Neural Networks Are Already Clean". In it he writes: "Superposition was the wrong explanation for InceptionV1." He blames three things instead. First, architectural problems made InceptionV1's later layers "overflow". Second, the settings used for feature visualization let the optimizer take shortcuts. Third, there is what he calls "recursive generalization": a neuron generalizes along one axis, then "sprouts outwards" along related ones. His main example is InceptionV1's famous cat-and-car neuron, which he now reads as a V-shape detector. MIT Technology Review reports that the same neuron fires on pool-table corners, train noses and fireboat sprays, and switches off for round shapes. For language models, the essay says: "In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood." A "pole" is the positive or negative side of a neuron's activation.
None of this has been independently verified. Goodfire has not yet published technical papers; the essay promises a technical companion paper. The researchers MIT Technology Review asked were unconvinced. Neel Nanda of Google DeepMind said: "This is interesting work, but I don't think it's a significant update to the big picture." Stanford's Chris Potts said: "As far as I can tell, nothing about this blog post makes the problems any easier." Jacob Hilton of the Alignment Research Center said the essay is "giving the public an unrealistic picture."
The paper in brief
The Anthropic and Harvard team asked a narrow question: can a neural network store more features than it has dimensions? They generated synthetic data with n known features, each of which is usually zero, squeezed it through m < n hidden dimensions, and trained a tiny ReLU autoencoder to rebuild it. With dense features, the model behaves like PCA: it keeps the m most important features and drops the rest. As the features get sparser, it starts storing extra features along non-orthogonal directions, which the paper calls superposition. These directions arrange themselves into antipodal pairs, triangles, pentagons and tetrahedra, and the model jumps sharply between these arrangements as sparsity changes. Once a ReLU sits on the hidden layer, the same pressure produces polysemantic neurons, meaning neurons that respond to several features. The authors write that they knew of no earlier unambiguous demonstration of feature superposition in neural networks. Their paper turned a hunch into a quantitative picture that others could test, and it became the reference point for later interpretability work.
The math
Setup
The input holds feature values. Each is 0 with probability , the sparsity, and otherwise uniform on [0, 1]. Each feature also has an importance . The model compresses with , where is an matrix whose column is feature 's direction in hidden space. It then decodes with the same matrix:
is the Gram matrix. Its diagonal entries say how strongly each feature is stored, and its off-diagonal entries say how much two features overlap. is a bias vector. Training minimizes importance-weighted mean squared error:
This is an autoencoder with tied weights and a weighted MSE loss. Remove the ReLU and you get a linear model whose best solution is PCA: keep the top- features orthogonal and throw away the rest.
Benefit versus interference
Why does one ReLU change so much? Group the loss terms by how many features are active at once. As , inputs with exactly one active feature dominate. Setting that feature's value to 1, the paper writes the loss as:
The first sum is the feature benefit: feature is reconstructed well when . The second sum is the interference: when feature fires, feature 's output picks up . In the linear model this term is with no ReLU, so every overlap costs something and superposition never pays. With the ReLU, negative overlap gets clipped to zero at no cost, and a slightly negative bias can clip small positive overlaps too. Sparsity helps further. A feature earns its benefit whenever it fires, which happens with probability . Two features only interfere when both fire, which happens with probability . With unit-norm features and zero bias, minimizing the interference term amounts to spreading points on a sphere so they repel each other. The paper notes this is a generalized version of the Thomson problem from chemistry.
Worked example: two features, one neuron
Take and . Feature 1 has importance 1 and feature 2 has importance . There are three natural solutions: drops feature 2, drops feature 1, and stores both features pointing in opposite directions, an "antipodal pair". In my simplified version, the bias is zero for the antipodal pair, and for a dropped feature the bias outputs that feature's mean. (The paper's closed-form treatment handles these details in its notebook.) The expected losses work out to:
- Dropping feature 2 costs . That is times the variance of the missing feature.
- The antipodal pair costs . It loses nothing unless both features fire; when they do, both outputs lose the smaller of the two values.
Set . With dense data (), dropping a feature costs 0.083 and the antipodal pair costs 0.333, four times worse. With sparse data (), dropping costs 0.031 and the antipodal pair costs 0.0033, about nine times better. The crossover is at , in the same region as the boundary in the paper's theoretical phase diagram. The best solution switches abruptly at that point rather than morphing smoothly: this is the phase change.
Two features, one neuron: drop one, or superpose?
interactive- drop extra feature, W=[1,0]
- drop feature 1, W=[0,1]
- antipodal pair, W=[1,−1]
Hover or drag across the chart to read values.
- superposition wins below 1−S =:
- 0.571
- equivalently, sparsity S above:
- 0.429
Fractional dimensions
To measure how much room each feature gets, the paper defines a feature's dimensionality:
Here is the unit vector along , and the sum runs over every , including itself. The numerator measures how strongly the feature is represented. The denominator counts how many features share its direction, weighted by squared projection. A feature with its own dimension has . Each member of an antipodal pair has . Five features arranged as a regular pentagon in 2D get each. An unlearned feature gets 0. The average version, , uses the Frobenius norm to estimate how many features the model has learned.
From directions to neurons
In this model, no direction in hidden space is special. Rotate the hidden space by any orthogonal matrix : since , the model behaves identically. So asking what an individual "neuron" means makes no sense here; this is a non-privileged basis. Put a ReLU on the hidden layer, , and the coordinate axes become special, because rotating would change the output. That is a privileged basis, and its axes are neurons. A neuron that carries weight from one feature is monosemantic; one that carries weight from several is polysemantic. The hypothesis under dispute is a chain: sparse features plus too few neurons lead to superposition, and superposition forces polysemantic neurons.
What the results say
- Basic demonstration (, ). The linear model always learns the top 5 features. The ReLU model does the same on dense data, then packs in more and more features as falls toward 0.001.
- Geometry (, , equal importances). is "sticky" at 1 and 1/2. The per-feature values cluster at 3/4 (tetrahedron), 2/3 (triangle), 1/2 (antipodal pair), 2/5 (pentagon) and 3/8 (square antiprism).
- Computation in superposition. A model computing absolute values (, ) has all-monosemantic neurons when features are dense and all highly polysemantic neurons at . Many neurons have one "primary" feature with a large weight plus weaker "secondary" features.
- Robustness. Vulnerability to adversarial examples rises by more than 3x as superposition forms.
- Replication. Redwood Research, DeepMind's Tom McGrath, and OpenAI's Jeff Wu and Dan Mossing independently reproduced core results.
Limitations. The authors state plainly that "it's very unclear what to generalize to real networks", and they call the geometry and learning-dynamics results "much more uncertain". The features are synthetic, mostly independent, and known in advance, which is exactly the ground truth real networks don't provide. The evidence that the toy model applies to real networks is consistency rather than direct proof: polysemantic neurons exist, they often share layers with clean neurons, InceptionV1 has more polysemantic neurons in later layers, and early transformer MLP neurons are highly polysemantic. Each of these observations could have other causes, and the paper does not rule those out.
Connecting the dots
The paper shows that superposition can happen. It does not show that superposition is why real neurons look polysemantic. Cammarata is attacking that second step. The paper's InceptionV1 argument (deeper features are rarer, so they superpose more) is the same observation he now attributes to architectural overflow. This is also criticism from inside the field. Cammarata co-authored "Zoom In" (2020), which this paper cites for the superposition hypothesis, and he is thanked in its acknowledgments for feedback.
The two pictures overlap more than the headlines suggest. The paper itself describes neurons with one primary feature and weaker secondary ones. It also cites Saxe et al.'s finding that hierarchical features "split" embedding vectors, which sounds a lot like branching. The real disagreement is narrower: are a neuron's secondary jobs related to its main job, or unrelated passengers? One caution about the 90% figure: it counts poles, not neurons. A neuron whose two poles have different jobs already carries two concepts, so the figure alone doesn't tell us how many features each dimension holds.
Evidence that could separate the two explanations:
- Width and sparsity sweeps. Superposition predicts more polysemanticity when features are sparser and when there are fewer neurons per feature. The architecture story predicts that fixes such as residual connections or LayerNorm clean up neurons at the same width.
- What gets packed together. In the toy model, co-occurrence decides which features share space, not meaning: anti-correlated features end up antipodal, and correlated ones stay orthogonal. Unrelated features that rarely co-occur would favor superposition; consistently related offshoots would favor branching.
- Interference signatures. Look for small off-target activations cut off by negative biases.
- Reproducible audits. The Qwen3-32B figures need a published protocol, and someone outside Goodfire needs to rerun them.
Why it matters
A great deal of interpretability tooling assumes superposition. For example, Cunningham et al. explicitly motivate sparse autoencoders as a way to recover features from superposition. If neurons in modern models really are mostly clean, practitioners might rely more on cheaper neuron-level analysis. If they are not, the case for decomposition methods stands. For safety and policy, the paper argued that enumerating features is central to auditing models, and Hilton's worry is that optimistic claims will distort what the public expects audits to deliver. Note that even if Cammarata is right about neurons, Nanda's point stands: understanding circuits is still a separate, unsolved problem.
What to watch: Goodfire's promised technical companion paper, and whether anyone outside Goodfire reproduces the Qwen3-32B audit.
Go deeper
- Toy Models of Superposition (arXiv), plus the authors' recommended interactive HTML version
- Neural Networks Are Already Clean: Cammarata's essay, which says it addresses the toy-models paper in an appendix
- MIT Technology Review's report, with reactions from Nanda, Potts and Hilton
- Sparse Autoencoders Find Highly Interpretable Features in Language Models (Cunningham et al., 2023): the follow-up method built on the superposition hypothesis