Mechanistic interpretability
In short
Mechanistic interpretability is research that tries to reconstruct a neural network's computation as a human-readable algorithm. That account is often written as pseudocode or called a circuit, and researchers still disagree about how strict the label is.
- Also called
- MI
- Topic
- Safety, ethics and risk
- Level
- Technical
- First described
- Chris Olah coined the term. It was first used in public in the Distill Circuits thread, posts by OpenAI researchers that ran from March 2020 to April 20212.
Aoki and colleagues describe the aim as reconstructing a neural network as a human-understandable algorithm, often as pseudocode or as a circuit, with each step tied to particular parts of the model1. On a narrower technical use, Saphra and Wiegreffe say the work looks for causal mechanisms: how intermediate representations turn an input into an output2.
One case they discuss is an induction head. Attention heads can let a language model predict "B" after a repeating pattern such as "ABABA", by finding an earlier "A" and the token that followed it2. The narrow definition requires that kind of causal story. It sets aside methods that only correlate an activation with an input or output, and tests that use input-output behaviour alone2. Aoki and colleagues say the methods in the literature run from observation of internals through to interventions on them1.
The name is not used in only one way. Saphra and Wiegreffe separate the narrow causal definition from a broader one that covers any description of activations or weights, and from uses that label a research community rather than a method2. Aoki and colleagues also say "mechanistic" is still ambiguous, and that the line with older interpretability work is not settled1. Chris Olah coined the term to separate circuit-style analysis from saliency methods, which score how much each part of an input matters2.
In definitions Saphra and Wiegreffe quote, the goal is to reverse-engineer a network into human-understandable mechanisms, often by finding circuits: groups of components that implement a behaviour2. The work became prominent with large language models and has drawn attention for safety and reliability1. A still stricter proposal, a complete causal path from input to output, has not been widely adopted2.
Example
Asked how a model continues "ABABA", this kind of study looks for the attention heads that find an earlier "A" and the token after it, rather than only noting which input tokens correlate with the answer2.
Why it matters
Aoki and colleagues say the field is followed partly for what it might show about the safety and reliability of large models1. On the narrow technical standard, a correlation inside the network is not yet an explanation: the account has to be causal2.
Questions people ask
Is mechanistic interpretability the same as explainable AI?
Not on the narrow definition. Older interpretability work includes saliency scores and tests that use only inputs and outputs. The narrow technical sense studies causal mechanisms in internal representations. The same label is also used more loosely, including for any look at a model's internals, and Aoki and colleagues say the boundary is still unclear12.
Can mechanistic interpretability prove that an AI is safe?
The papers cited here do not present it as a safety proof. Aoki and colleagues say safety and reliability are reasons for the interest, and that further work is needed to understand complex systems1. A complete causal path from input to output is a stricter standard that has not been widely adopted2.
What is a circuit in mechanistic interpretability?
Saphra and Wiegreffe describe a circuit as a group of components that together implement a behaviour2. Aoki and colleagues also use the word for the human-readable algorithm itself, often written as pseudocode, with each step tied to a part of the model1.