More Features Than Neurons

0.One neuron, many meanings

If you open up a trained language model and pick a single neuron, what you have is one number in one layer, which takes some value on every input you feed the model and sits at zero for most of them. If networks were built the way people would build them, you could describe what each of those numbers is for in a short phrase, so one neuron would track quotation marks, another wetness, another the Golden Gate Bridge, and you could work your way down the list until the whole model was catalogued.

Trained networks mostly don't work that way, and you can see it in the four GPT-2 XL neurons below, each shown with the text that activates it most strongly, copied exactly from OpenAI's public dataset of all 307,200 of them.^[9] The brighter a token is, the more strongly the neuron fired on it. Try finishing the sentence “this neuron detects ___” for any of the four.

Neuron 3013 in layer 12 is another one. It responds to Kalamazoo, a sumo wrestler's name, and the second half of the word “warhead”, which you could count as three things or one depending on how you feel about tokenization. Neurons like these turn up all over language models. When OpenAI had GPT-4 write an explanation for every neuron in GPT-2 XL and then scored each explanation by how well it predicted the neuron's activity, only a bit over a thousand of the 307,200 scored 0.8 or higher.^[9] A low score doesn't prove that a neuron has no single meaning, because the explanations can miss things, but it does suggest that neurons with one clean meaning are uncommon. There is more than one proposed reason for this.^[13] The best-studied one, and the one this page is about, is that the network has more things to represent than it has neurons.

GPT-2 small, for example, has 768 numbers per layer and a vocabulary of 50,257 tokens, so even if all it had to record was which token it just read, with no syntax, topic, or speaker, it would need about 65 tokens per dimension. The vocabulary is the easy part, since the model also has to track who is talking, what language the text is in, whether the tone is sarcastic, and whether the current token is inside a quotation, a URL, or a legal citation, along with many things nobody has named. When Anthropic trained a sparse dictionary on one of its production models, they gave it room for 34 million features^[3], and while not every entry ended up being used, that is still far more than the number of dimensions in any layer of the model.

So models store more concepts than they have dimensions, layered on top of one another, and accept the interference that comes with it. This is called superposition, and there is a theory of it detailed enough to make testable predictions.^[1] The model at the center of that theory has only 18 parameters and trains in about a second, which is small enough that its main experiments can run right here on the page instead of being shown as static figures.

You get to try the first experiment yourself before any network does. Knowing what a vector is will be enough to follow along, and the expandable notes cover the rest as it comes up.

1.The packing problem

If you take away everything except the storage problem, what is left is a puzzle you can solve with a mouse. You have six features to store, meaning six concepts the model cares about, and two dimensions to store them in. Each feature is stored as a direction, which is an arrow in the flat space below, and an arrow needs a length of 1 to count as fully stored. The difficulty is that any two arrows that aren't perpendicular interfere, which means reading one of them also picks up part of the other, and the score adds a penalty for all of that overlap.

Two dimensions only have room for two perpendicular arrows, so with six features, at least some of them will have to overlap.

There is one more rule, which is that the six features aren't equally valuable. F1 matters most and F6 least, and the thicker an arrow is drawn, the more it is worth. Drag the arrow tips to get the score as low as you can, and then hand your layout to gradient descent. It starts from your arrangement rather than from scratch, so whatever it ends up with is an improvement on your layout.

After a minute of dragging you have probably found both of the strategies this problem allows. One is to give the two most valuable features perpendicular arrows and drop the other four, so that nothing interferes but four concepts aren't stored at all. The other is to fit the extra arrows into the gaps, so that every feature overlaps a little with its neighbors, each one gets a small amount of interference, and none are dropped.

In this game the overlapping layout scores better, but which strategy works better in general depends on how often the features actually show up, and that is what the next two sections are about.

2.The toy model

Next we give the same problem to an actual network. This is the toy model from Elhage et al., set up the way they published it^[1]. It's an autoencoder with two hidden neurons between a six-number input and a six-number output, and it has 18 parameters in total.

Each of the six features in the input x is off most of the time, and when one does turn on (5% of the time here) it takes a random strength. Importance falls from F1 to F6, so an error on F1 counts about six times as much in the loss as the same error on F6. The model has to pass all six features through the two hidden neurons and then reconstruct them at the output.

Since so much depends on the nonlinearity, the figure below lets you run the model both with it and without it. The model is training in your browser while you read this, and the arrows on the left are the columns of W moving under gradient descent.

The linear model can't use overlapping directions no matter how long you train it, because when two features overlap, the interference between them goes directly into the output every time. The best it can do is keep the top two features and drop the rest, which is one of the strategies you probably tried in the game. That solution is the same one principal component analysis gives, and the model arrives at it even though nothing in the setup mentions PCA.

With the ReLU, some of that interference can be removed, and how it gets removed depends on which of two layouts the run ends up in. In the default run, F1 and F2 point in nearly opposite directions, so when F1 fires on its own the model's reading for F2 comes out at about −1, and because the ReLU turns every negative number into zero, F2's output is still correct. Hit Retrain a few times and you'll also see runs where five features spread out around the circle instead. In that layout the interference comes out slightly positive, so the model learns a negative bias of around −0.2 that lowers every output a little. That's enough to put the small interference values below zero, where the ReLU sets them to zero, while a feature that's actually on is large enough to stay above zero. The ReLU and the bias together work like a noise gate on a microphone, which silences anything below a threshold and lets louder signals through, and this is how the model stores more features than it has dimensions. Nobody designed this arrangement, since it's simply the one that happens to minimize the loss.

Cell phone networks have used a similar idea in a scheme called CDMA, which many 3G phones relied on. It let dozens of phones share one radio channel by giving each phone its own code, letting all the signals overlap on the channel, and having the receiver match each phone's code and cut everyone else's signal below a threshold. In the toy model the features are the phones, the two-dimensional hidden space is the shared channel, and the ReLU with its bias is the receiver. The comparison has limits, since CDMA uses long codes that Qualcomm's engineers designed on purpose while the toy model has two dimensions and finds its layout by gradient descent, but CDMA was used for the same reason the overlap works here, which is that there were more phones than the channel could carry if each had its own slice, and each phone only transmitted some of the time.

3.Sparsity

The noise gate from the last section only removes small interference. When two overlapping features are active on the same input, the interference between them is large enough to get past the bias, and the output comes out wrong. So superposition only works well if overlapping features are rarely active at the same time, which is the case when features are sparse, meaning each one is active on only a small fraction of inputs.

Language data looks sparse in this sense, since specific concepts like Korean text, HTTP requests, or academic citations each show up in only a small fraction of sentences, and they rarely show up together. (The dictionaries learned from real models are sparse too, but that is partly because they are trained with a penalty that makes them sparse, so it isn't separate evidence.) Superposition depends on this rarity, and the slider below sets how rare the features are.

This is the same model as before, with the importances made nearly flat (they go from 1 down to 0.69), so that no feature gets kept just because it matters more. The slider changes one number, which is how often each feature is active, and it works best if you move it slowly from left to right. Training runs the whole time, so the model keeps re-solving the problem for whatever data the slider gives it.

These changes happen suddenly, in that a layout stays the same across a wide range of slider positions and then rearranges within a few percent of the slider's travel, a bit like water that stays ordinary water all the way down to 1°C and then freezes. Elhage et al. call these phase changes, by analogy with physics, and map out where they happen for different numbers of features and dimensions.^[1]

At the sparse end, the five arrows are close to full length and evenly spaced, so each one has a small and equal amount of interference with its neighbors. Features are rarely active together at this sparsity, so the ReLU and bias remove almost all of that interference, and the model stores five features in two dimensions with very little error.

4.Feature geometry

Nothing in the loss function asks for the regular pentagon at the end of the last section, since all the loss measures is squared reconstruction error. If you add a dimension, regular shapes show up again, which suggests the pentagon wasn't a coincidence. Below is the same kind of model with eight features in three dimensions, shown two ways: the arrows themselves, which you can rotate, and a quantity Elhage et al. call feature dimensionality:

D_i = ‖W_i‖² / Σ_j (Ŵ_i·W_j)²

You can read D_i as how much of a dimension feature i has to itself. A feature with its own axis that nothing else touches gets D=1, a feature that shares an axis with one partner pointing the opposite way gets exactly ½, and a dropped feature gets 0.

The flat steps at clean fractions point to what the model is doing. It spreads its feature directions over the sphere so that they sit as far from each other as possible, even though nothing in the loss mentions geometry, which is close to a question J. J. Thomson asked in 1904 about how electrons would arrange themselves if they all repelled each other.^[6] The best known arrangements for that problem on a sphere are the same shapes on your screen: opposite pairs, triangles, tetrahedra, and square antiprisms. The toy model's loss isn't Thomson's energy, so the two problems don't have to agree in general, but for the cases on this page gradient descent ends up at the same shapes.^[1]

The hidden neurons in this toy have no nonlinearity on them, so you could rotate every arrow by the same angle and the model would work exactly as well. That means the neuron axes in this toy carry no meaning of their own, and only the directions of the features matter. (The idea that features correspond to directions is called the linear representation hypothesis, and Park et al. state it precisely enough to test.^[10]) Real models are different here, because the GPT-2 neurons at the top of the page come after a nonlinearity, which gives the model a reason to line some features up with individual neurons, and when Elhage et al. tested a model like that, they found some neurons that matched a single feature and many that still didn't.^[1]

5.Polysemanticity

This brings us back to the question from the top of the page. Five feature arrows are spread around a circle and a neuron is one axis through that circle, so we can ask what a single neuron in this model responds to.

Below is the model from the top of the page, the same network you fired concepts into, trained at high sparsity into the pentagon from the sparsity section and then frozen. Each panel shows one hidden neuron's weight on every feature. It is also the model shown in the margin on the left, if your screen is wide enough to show it.

This is the same thing you saw with the lobster and larvae neuron at the top of the page, reproduced in a model with 18 parameters. If five features are stored as a pentagon, any axis you draw passes close to several of them, so each neuron reads out a weighted mix of unrelated features. In this toy, polysemanticity follows directly from the geometry rather than from anything going wrong in training. Olah et al. documented polysemantic neurons in vision models before this mechanism was worked out,^[7] and Arora et al. found a similar overlapping structure in word embeddings even earlier.^[12]

This helps explain why labeling what each neuron does has been so hard in language models. It doesn't always fail, and the same vision models have many neurons that respond to one clear thing, like curves or dog heads.^[7] But when features outnumber neurons and are stored this way, the neuron axes aren't the directions the features are stored along. In the toy nothing is lost, since all the features are still there and can be read back almost perfectly along their own directions, which are rotated away from the neuron axes we were reading.

If superposition mixes the features together in a way we understand, it might be possible to undo it.

6.Sparse autoencoders

The reverse problem is to start from the hidden activations h, which are two numbers per input with all the features mixed together, and recover the original features from them. You don't know how many features there are or which directions they point in, and there is no answer key to check against.

The usual approach relies on sparsity, the same property that made superposition possible. If the true features are sparse, then any single activation can be explained using only a few of them, so the sparsest explanation that still reconstructs the data accurately is likely to be the true one. This is the idea behind the sparse autoencoder.^[2][4] It learns a dictionary of candidate directions that is larger than needed (ten here, for at most six real features), rebuilds each activation out of those dictionary entries, which are called atoms, and adds a penalty of strength λ for every atom an explanation uses.

Because this is a toy, we have something nobody has for a real model, which is the true features, so we can check how well the SAE did instead of just looking at its output. The thick colored arrows below are the true features and the pale arrows are the dictionary atoms, training live, and when you move λ the dictionary changes in response. This figure also lets you click any atom to ablate it, which forces it to stay at zero so you can see what the reconstruction loses without it.

The SAE wasn't told there were five features or that they formed a pentagon. It only saw two-dimensional points, and the sparsity penalty was enough to move its atoms onto the true directions. That works here because, for this data, the sparsest accurate description of the hidden activations is the original set of sparse features. That isn't guaranteed in general, and on real models SAEs sometimes merge features or split one feature into several. The idea is also much older than interpretability research. In 1996 Olshausen and Field got receptive fields resembling those in visual cortex out of natural images just by asking for a sparse code,^[8] and at the time it was a claim about how the visual system works.

The two ends of the slider show the same problems people run into with SAEs on real models. With too little penalty the dictionary stays mixed, and with too much, features disappear or break into pieces. On a real model there are no thick colored arrows showing the right answer, so choosing λ relies on proxy metrics, and changes to the method like k-sparse autoencoders, which set the number of active atoms directly, are one current way around it.^[5] It isn't a solved problem, which is part of why a toy where the answer is known is useful for checking the method.

Removing a suspected feature to see what breaks is a basic experiment in interpretability. Here it takes one click and the effect shows up on a meter, while on a large model the same kind of experiment takes much more work.

Below are three atoms from a dictionary trained the same way on GPT-2's residual stream, with their records copied exactly. At the top of the page you couldn't finish the sentence “this neuron detects ___”, so try it again with these.

7.What carries over to real models

Everything on this page used six or eight features and two or three dimensions, which is very far from a real model. The smallest GPT-2 has 768 dimensions per layer, and the largest published dictionaries have tens of millions of features. Even so, the same method has worked at that scale in some cases. Trained on a production model's residual stream, the dictionary included atoms for the Golden Gate Bridge, for backdoors in code, and for sycophantic praise, each a single direction in the residual stream that a person could name.^[3]

A lot about this is still unsettled. SAEs have their own versions of superposition's tradeoffs, since the features they recover split and merge as the dictionary grows,^[2] and some structure isn't a straight-line direction at all. Days of the week and months of the year, for example, come out arranged in circles, which a dictionary of single directions can't recover cleanly.^[11] You have already seen two of these problems in the λ slider, where features died at one end and broke into pieces at the other.

The overall picture still holds up with those caveats. A model has far more things to track than it has dimensions, most of those things are rare, and the model stores them as overlapping directions and uses the ReLU and bias to remove most of the interference. If you read the result one neuron at a time, you get something like the lobster and larvae neuron from the top of the page, but along the right directions much of it becomes readable again. A large part of interpretability research right now is about finding those directions, and the sparse autoencoder you trained above is one of the main tools for doing it.