Weightpedia

What does every single weight in a transformer do?

Circuit analysis tells us what a weight does in service of one task, on one distribution.

Here, we ask the reverse question: pick any single weight in a transformer — can we say what it does, in general?

Check the weights → Paper on arXivcoming soon

Abstract

Understanding what the individual components of a neural network do is a central goal of mechanistic interpretability, and a valuable tool for auditing or editing trained models. In standard dense transformers the question is nearly unanswerable at the level of weights: each weight participates in many computations at once, and the weight basis is not aligned with what the network computes.

Weight-sparse transformers (Gao et al., 2025) offer a path forward — when most weights are trained to zero, the surviving weights align more cleanly with the computation, opening the door to per-weight interpretation. We introduce an automated pipeline that ablates each nonzero weight, measures when it matters, produces a short human-readable account of what it does, and verifies that account on held-out data. Applied at scale, a substantial fraction of the nonzero weights in weight-sparse transformers turn out to be interpretable — several times more than in similar dense models. Weightpedia is the resulting encyclopedia: a browsable ablation profile for every weight with a measurable effect in a weight-sparse transformer trained on Python code.

The framework: select a parameter, set it to zero, run the original and ablated models over a corpus, compute per-token KL divergence, and inspect the most-affected corpus examples.
The framework. Pick a single nonzero weight and set it to zero. Run the original and the ablated model over a large corpus and compare their predictions token by token (KL divergence). The positions where the two models disagree the most reveal when — and why — that weight matters.

1Ablate

Zero out a single nonzero weight in the network, leaving everything else untouched.

2Measure

Compare against the original model on thousands of code prompts: KL divergence at every token position.

3Profile

Weights with consistent effects get a card: affected tokens, activating contexts, and probability shifts.