The Ghost in the Machine: How to Read a Neural Network From the Inside

The Hook: A Bridge That Wouldn't Stop Being a Bridge

In May 2024, for 24 hours, you could talk to a version of Claude that was profoundly and incurably obsessed with the Golden Gate Bridge. Ask it how to spend ten dollars, and it recommended a drive across the bridge. Ask for a love story, and it told the tale of a car crossing its beloved bridge. Ask what it imagined it looked like, and it answered: like the Golden Gate Bridge. This was no prompt trick, no "pretend that." Researchers at Anthropic had found, deep inside the model, a single, precisely identified direction in activation space – a "feature" responsible for the concept Golden Gate Bridge – and artificially turned its value all the way up. The result was a model that literally wove the bridge into every thought.

This is more than a gimmick. It is the most vivid proof yet that the internal representations of large language models are not an impenetrable soup of numbers, but structures that can be found, named, and deliberately manipulated. It is a window into one of the most important and fastest-growing disciplines in AI research: mechanistic interpretability – the attempt to understand a neural network not merely by its behavior, but by its inner mechanisms, the way a biologist understands an organ or a reverse engineer understands a compiled program.

Why should this interest you, beyond the fascination? Because we are building and deploying systems whose decisions we cannot explain – and doing so in domains ranging from medicine and justice to IT security. A language model is not a human-written program with readable source code. It is a grown artifact: hundreds of billions of numbers that an optimization procedure tuned until the behavior was right. No one designed those numbers directly, and no one can read them directly. Mechanistic interpretability is the attempt to open this black box from the inside – and the findings of the last three years are remarkable.

This article takes you the full distance: from the question of why a single neuron can fire for seemingly unrelated things, through the elegant mathematics of superposition by which a network stores more concepts than it has neurons, to the sparse autoencoders that disentangle these intertwined concepts again – and finally to what all this teaches us about software architecture, security, and even the very notion of "understanding."

Part 1: The Core Problem – A Program No One Wrote

Why You Cannot Simply "Read" a Neural Network

When you write a method CalculateDiscount(decimal price) in C#, you can open it and follow what it does. The code is the explanation. A neural network has no such thing. What it has are weights and activations: enormous matrices of floating-point numbers and the intermediate results that arise when an input text flows through the network. A modern language model consists of dozens of layers through which a high-dimensional vector stream – the so-called residual stream – runs. At each layer this stream is modified. At the end, a probability distribution over the next token falls out.

The problem: these intermediate states are not written in a language humans can read. A single neuron has an activation value – say 4.7 – but on its own this value means nothing. All the meaning lies in the interplay of thousands of such values. Mechanistic interpretability asks: Can this interplay be decomposed into understandable building blocks?

The Naive Hope and Its Disappointment: Polysemanticity

The first, obvious hope was: perhaps each neuron corresponds to a concept. Neuron 1,024 fires for "cats," neuron 5,099 for "past tense," and so on. This ideal is called monosemantic neurons – one neuron, one meaning. Had the nature of the network done us this favor, interpretability would be easy: you would simply go through the neurons one by one and name them.

Reality is more stubborn. When you examine real neurons in large language models, you find that most are polysemantic: a single neuron fires for a motley collection of seemingly unrelated concepts. A famous early example was a neuron that responded simultaneously to academic citations, English dialogue, HTTP requests, and Korean text. There is no meaningful human term that captures this mixture. The neuron is not a clean building block but a junction where several meanings overlap.

This polysemanticity was for years the central obstacle of interpretability. As long as the smallest computational unit – the neuron – itself carries no clear meaning, the network cannot be cleanly decomposed. The decisive question therefore became: Why are neurons polysemantic? Is this an accident, a defect – or is there a principle behind it?

Part 2: Superposition – More Concepts Than Neurons

The Central Idea

The answer, worked out by a team around Nelson Elhage at Anthropic in the influential 2022 paper Toy Models of Superposition, is surprising and elegant. It is this: polysemanticity is not a defect, but the visible consequence of a clever compression strategy the network learns on its own. That strategy is called superposition.

The core claim of the superposition hypothesis: A neural network represents more features than it has dimensions (neurons) by encoding each feature as its own direction in the high-dimensional activation space – and by making these directions not exactly perpendicular to each other, but only almost perpendicular.

To understand this, you need two concepts. First, the linear representation hypothesis: the well-supported observation that networks represent concepts as directions in activation space. "More cat" means: further in the direction of the cat vector. An activation is then a weighted sum of many such feature vectors. Second, the concept of orthogonality: in a space with n dimensions there are exactly n mutually perpendicular directions. If every feature needed its own perfectly perpendicular direction, a network with 512 neurons could store only 512 features.

The Mathematical Trick: Almost-Orthogonal Directions

Here comes the beautiful piece of mathematics. If you insist that feature directions be exactly perpendicular, you are limited to n features. But if you allow small deviations – directions that are almost, but not quite, perpendicular to each other – then suddenly exponentially many directions fit into the same space. This is the content of the Johnson-Lindenstrauss lemma, a classic result of high-dimensional geometry: in a space of dimension d, you can accommodate not just d but a number of vectors growing exponentially in d, all of which are approximately perpendicular to one another.

This is counterintuitive, because our intuition is stuck in three-dimensional space, where "almost perpendicular" and "perpendicular" are barely distinguishable. In very high-dimensional spaces it is different: randomly chosen directions there are almost always already approximately orthogonal. High dimensions are "roomier" than intuition suggests.

The network exploits exactly this. It packs thousands or millions of features into a space with only thousands of neurons by assigning them almost-orthogonal directions. The price is interference: because the directions are not perfectly perpendicular, each feature "leaks" a little into the others – a noise. And it is precisely this superimposing of many feature directions onto the same physical neurons that we see at the surface as polysemanticity: a neuron happens to lie near several feature directions and therefore fires for several concepts.

Why the Trade Pays Off: Sparsity

That leaves the question: why does the network accept the interference? The answer lies in a property of the real world that Toy Models of Superposition worked out precisely: sparsity. The vast majority of features are inactive for any given input. In a sentence about quantum physics, the feature "cooking recipe" does not appear, nor does "Renaissance painting" or "SQL syntax." Of the millions of possible concepts, only a tiny selection is active at any moment.

If features are sparse, the probability that two interfering features fire simultaneously and disrupt each other is low. So the interference usually stays harmless. The gain – exponentially more storable concepts – far outweighs the rare damage. The Toy Models paper showed this cleanly in experiments: with dense (non-sparse) data, a mini-network learns no superposition and stores only as many features as it has dimensions. The sparser the data, the more aggressively the network switches into superposition mode and squeezes ever more features into the same space. Superposition is therefore no accident, but a learned, rational response to a sparse world.

I am of the opinion that this is one of the most beautiful insights of modern AI research, because it explains two things at once: why networks are so bewilderingly opaque (they compress aggressively) and why there is nonetheless hope of decoding them (the compression follows a clear mathematical principle).

Part 3: Sparse Autoencoders – Recovering the Dictionary

The Task: Disentangle, Not Guess

If the true building blocks – the features – are smeared across the neurons, then it is pointless to examine individual neurons. You need a procedure that goes the other way: it should decompose the few thousand physical neuron activations back into the many thousand original, clean features. You want to undo the superposition.

The tool that has prevailed for this is the sparse autoencoder (SAE), a variant of the classic technique of dictionary learning. The idea is strikingly direct. An autoencoder is a small neural network with two parts: an encoder takes the activation of a layer of the large model and projects it into a much larger space – instead of, say, 512 dimensions, perhaps 16,000 or millions. A decoder then tries to reconstruct the original activation from this inflated representation.

The decisive trick lies in two conditions under which the training takes place:

1. Overcomplete: The middle space is much larger than the input. This gives the SAE enough "drawers" to redistribute the features entangled in superposition onto one dimension each.
2. Sparse: A penalty term in training enforces that for each individual input only very few of these many dimensions may be active simultaneously.

This second condition is the lever. It forces the SAE to reproduce exactly the structure we assume the large model possesses anyway: many possible features, of which only a handful fire per input. If this succeeds, each of the many dimensions in the inflated space ideally corresponds to one clean, monosemantic feature.

The Breakthrough: Towards Monosemanticity

In 2023, Anthropic showed in the paper Towards Monosemanticity that this recipe works. On a small one-layer transformer, the researchers trained sparse autoencoders and extracted thousands of features that turned out to be genuinely clean and individually interpretable – for instance a feature for DNA sequences, one for Arabic script, one for legal language. Where polysemantic chaos had reigned before, nameable building blocks now lay ready. Human raters could unambiguously assign a large fraction of these SAE features to a concept, while the same people could not meaningfully name the original neurons.

The Scaling Leap: Scaling Monosemanticity

The open question was: does this also work on a real, large, production model – or only in the lab with toy networks? In May 2024 Anthropic answered it with Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. The team trained sparse autoencoders on the middle residual stream of Claude 3 Sonnet, a then-current production model, and extracted up to 34 million features.

These features were not superficial, but abstract and multimodal. The famous Golden Gate Bridge feature activated not only on the English word, but on descriptions of the bridge in Japanese, Korean, and Russian – and even on images of the bridge. A single direction in activation space had thus captured the concept of the bridge, independent of the concrete linguistic or visual form. Other features encoded subtle, safety-relevant concepts: programming bugs in code, gender bias, or – most remarkably – the concept of "caution when handling dangerous, e.g. biological or weaponizable knowledge."

From Reading to Writing: Feature Steering

The truly convincing test of an explanation is intervention: if we claim that a particular direction is the concept "Golden Gate Bridge," then artificially amplifying this direction should change the model's behavior in a predictable way. This is exactly what the researchers demonstrated with feature steering (also called clamping): turn the activation of the bridge feature up to a multiple of its normal maximum, and you get the "Golden Gate Claude" described at the outset, which weaves the bridge into every answer. This is causal proof, not mere correlation: the identified direction demonstrably causes the associated behavior. This principle – finding an internal feature and deliberately amplifying or dampening it – is the basis of the hope of one day making models targeted-safer without retraining them expensively.

Part 4: From Features to Circuits – The "Biology" of a Language Model

The Next Step: Not Just Building Blocks, but Wiring

Finding individual features is like having catalogued the vocabulary of a language. But meaning arises from sentences – from the interplay. The next step of interpretability therefore asks not "What features are there?" but "How does the model wire them into circuits that solve a concrete task?"

In 2025 Anthropic delivered a milestone with two companion papers – Circuit Tracing and On the Biology of a Large Language Model. The researchers developed attribution graphs: maps that trace how information flows from the input tokens through the internal features to the output. Technically, they replace the opaque computational layers of the model with interpretable, sparsely active features (via so-called cross-layer transcoders) and then track which feature triggers which other. The deliberately chosen term "biology" signals the stance: one studies the model like an organism – observe, form hypotheses, test through targeted interventions (perturbations).

Two Surprising Findings

Two results from On the Biology of a Large Language Model are especially illuminating because they contradict widespread intuitions:

The model plans ahead. When composing a rhyme, the attribution graphs showed, the model does not simply pick word by word and stumble at the line's end into a rhyme by chance. Instead it activates possible rhyme words for the end of the line early on and then works backward to build a line that naturally leads to this target word. This refutes the often-heard caricature that a language model is "merely" a prediction machine guessing one word after another – at least on the mechanistic level it plans beyond the next token.

A shared language of thought. On multilingual tasks the model uses partly language-independent, abstract features – a "conceptual intermediate space" in which the meaning lives independent of the concrete language, before it is formulated again in English, German, or Chinese. This matches exactly the multimodal Golden Gate feature and supports the idea that these models develop a kind of abstract "language of thought."

The Honest Caveat: Limits and Open Questions

To preserve the scientific standard, the sober qualification belongs here – and the researchers themselves stress it. Attribution graphs are simplified, local approximations of what goes on in the model, not a complete depiction. Sparse autoencoders do not capture the features perfectly; there is a reconstruction error, and it is an open research question whether they find the "true" features of the model or only a usable approximation. There is moreover a debate about so-called dark matter of interpretability – portions of the activation that so far cannot be assigned to any clean feature.

And the sheer scale remains crushing: tracing a single short prompt through a model like Claude 3.5 Haiku cost the researchers hours of manual analysis of a single attribution graph. Today we can understand some circuits of some models on some tasks – but the complete "script" of a large model is far from written. Anthropic open-sourced its circuit-tracing tools in 2025, explicitly to spread this laborious work across many shoulders.

Part 5: Why It Matters – Safety, Trust, and Control

Interpretability as a Safety Tool

The practical motive behind this research is not academic curiosity alone, but safety. We increasingly deploy language models in situations where errors or manipulation are costly. As long as we can only check their behavior, we are blind to the question of why a model does something – and therefore also to whether it does the right thing for the "right" reasons or only by accident.

Mechanistic interpretability promises concrete levers here. If you can identify a feature for "deceptive behavior" or "dangerous biological knowledge," you might be able to monitor whether it becomes active – a kind of internal lie detector or early-warning system. If you can causally amplify or dampen features (feature steering), this opens a path to adjusting behavior deliberately without retraining the entire model. And whoever understands the internal circuits can find vulnerabilities – for instance, trace which internal paths a "jailbreak" uses to bypass the safeguards.

The Bridge to IT Security: Defense in Depth

Here it is worth stepping out of AI research into the world of security in which you work. The interpretability agenda follows at its core the same principle as good security architecture: you cannot secure what you do not understand. A black box you can only check by its outputs is, from a security standpoint, the same as a closed binary blob you trust blindly. Reverse-engineering a neural network is conceptually related to reverse-engineering malware: in both cases there is a functionality whose source code is missing, and you must infer the mechanism from observable behavior and internal structure.

Just like the logic described in Harvest Now, Decrypt Later: Post-Quantum Cryptography and the Race Against the Quantum Computer – understanding threats before they take effect, and building resilience at the source rather than only repairing after the fact – interpretability is an investment in anticipatory rather than reactive security. And as with the compliance requirements from NIS2 and What It Really Means for Mid-Market IT Consulting Firms in Germany, the question "Can you explain how this system reached its decision?" is increasingly being asked not only technically but regulatorily. Explainability migrates from optional to obligatory.

Part 6: The Bridge Into Your World – What Architects Can Learn

Mechanistic interpretability is not only an AI topic. It is a lesson in principles that recur in every complex system – including the architectures you build.

1. Observability Before Trust

The whole discipline is, at bottom, observability taken to the extreme. In software architecture we have learned that a distributed system without tracing, logging, and metrics is a nightmare in the event of failure: you see that something is going wrong, but not why. Attribution graphs are nothing other than distributed tracing for a neural network – they trace the causal path of a "request" through the internal "services" (features). The lesson is universal: a system whose internal state you cannot make visible, you can neither debug nor secure nor truly trust.

2. Representation as a Computed Result, Not a Memory Cell

The linear representation hypothesis says: meaning resides not in a single storage location (a neuron) but in a direction that emerges from the interplay of many components. A concept is an emergent, computed pattern, not a hard-wired variable. This is astonishingly close to the thinking behind The Logbook of Truth: Understanding Event Sourcing and CQRS, where the state of the system is likewise not a stored value but a result computed from many events. In both worlds it holds: do not confuse the physical storage form with the logical meaning – the meaning lives in the structure, not in the individual slot.

3. Compression Has a Price You Must Know

Superposition is an ingenious compression – and it illustrates a hard truth of every optimization: there is no free lunch. The network gains capacity and pays with interference and opacity. Every aggressive optimization in a system – an overloaded cache, an algorithm "clever" to the point of unreadability, or a database normalized to the last detail – buys efficiency with diminished comprehensibility. Whoever builds systems should make this trade deliberately, not accidentally.

4. An Honest Humility Toward the Self-Built

Perhaps the deepest lesson: we have created an artifact we no longer immediately understand, and must now investigate it with the methods of natural science, as if it were a piece of nature. This is a new condition for engineering. Software was until now the prime example of the fully transparent – every line written by a human. With grown models we leave this comfortable world. The humility that follows is healthy: it recalls that "works in the tests" and "I understand why it works" are two different things – a distinction that grows more important with the complexity of every system.

Part 7: The Philosophical Dimension – What Does "Understanding" Mean?

Mechanistic interpretability touches an old philosophical question in a new, empirical form. If a model possesses an abstract, language-independent feature for "Golden Gate Bridge" that stays the same across languages and even across images – does it then have a concept of the bridge? Or is it only a particularly sophisticated statistical correlation that merely looks like a concept?

The temptation to separate the two sharply is great – but perhaps the separation is less clean than we hope. Our own brain, too, presumably represents concepts as activation patterns in neuron assemblies, and there too are hints of something like superposition. The question "real understanding or mere statistics?" could turn out to be a false dichotomy: perhaps understanding, at a sufficiently deep level, is a certain kind of structured, robust, interveneable representation – and that is exactly what we find in these models.

I am of the opinion that mechanistic interpretability makes its perhaps most underrated contribution here: it transforms a centuries-old philosophical speculation into an experimental question. Instead of arguing about whether a system "really understands," we can now concretely investigate which concepts it represents, how abstract they are, how they are wired, and what happens when we intervene. This does not make the discussion superfluous, but it gives it, for the first time, an empirical foundation. The bridge to the question of consciousness and representation that also underlies human cognition connects this topic with the question of psychological and cognitive structures in Psychological Safety: The Hidden Force Behind High-Performing Teams.

A final, honest clarification against a widespread misconception: the "superposition" of AI research has nothing to do with the quantum superposition discussed in Spooky Action at a Distance: Quantum Entanglement from Einstein to the Quantum Internet. The term is merely a metaphor for the geometric superimposing of feature directions in the classical, high-dimensional vector space. No qubits flow, there is no entanglement, no collapse. The shared name is a linguistic bridge, not a physical one – and it is worth keeping this distinction clean, precisely because the terms sound so seductively similar.

Part 8: Where the Research Stands 2024–2026

The field is conspicuously alive, and the cadence is high. Some of the more recent milestones:

• Toy Models of Superposition (2022): The theoretical foundation – superposition as a rational response to sparse data, with a clean mathematical underpinning.
• Towards Monosemanticity (2023): The first demonstration that sparse autoencoders disentangle superposition in a real (small) transformer and yield interpretable features.
• Scaling Monosemanticity (May 2024): The scaling leap onto a production model (Claude 3 Sonnet) with up to 34 million abstract, multimodal features – and the causal proof through feature steering ("Golden Gate Claude").
• Circuit Tracing & On the Biology of a Large Language Model (2025): The step from individual features to whole circuits via attribution graphs – with surprising findings on forward planning and language-independent "thinking."
• Open-source tools (2025): Anthropic makes circuit-tracing tools publicly available to democratize the labor-intensive analysis; a growing academic community (including SAEs for open models such as Llama) contributes.

The big open questions remain: do sparse autoencoders find the true features or only usable approximations? How can the "dark matter" of unexplained activations be captured? And above all: how do you scale the laborious, often manual circuit analysis far enough that it keeps pace with the size and tempo of model development? Today we know some letters and a few words of the language in which these models "think" – the grammar is not yet written.

The Central Takeaway

Mechanistic interpretability is no esoteric niche topic, but a lesson in three stages:

1. Mathematically it shows that the apparent opacity of neural networks follows a clear principle: superposition, thanks to almost-orthogonal directions, packs exponentially more features into the space than there are neurons – a rational response to a sparse world, whose price is the polysemantic smearing.
2. Methodologically it provides, with sparse autoencoders and attribution graphs, tools that disentangle this intertwining again: from millions of clean, even multimodal features (Scaling Monosemanticity) to traceable circuits whose behavior can be causally altered (Golden Gate Claude).
3. Conceptually it gives everyone who builds systems a precise picture: observability before trust, meaning as computed structure rather than stored value, and the humility that "it works" and "I understand why" are two different things.

Concrete call to action for this week: Take one system – or an AI service – that you trust in your work, and ask the interpretability question: "If this system makes a consequential, wrong decision tomorrow – could I reconstruct why?" Where the honest answer is "no," there is a blind spot. You need not close it today, but you should know it is there – because a non-explainable system is a non-securable system.

Reflection question: If we build machines we can only understand with the methods of natural science – observe, form hypotheses, experiment – rather than simply reading them like classical code: what does this mean for our responsibility as engineers, and where will the boundary between "built" and "grown" run in the future?

Cross-References in the Vault

• The Logbook of Truth: Understanding Event Sourcing and CQRS – Meaning, respectively state, as a computed result rather than a stored value: the same architectural principle in two worlds.
• Harvest Now, Decrypt Later: Post-Quantum Cryptography and the Race Against the Quantum Computer – Anticipatory rather than reactive security; understanding threats before they take effect.
• NIS2 and What It Really Means for Mid-Market IT Consulting Firms in Germany – Explainability as a growing regulatory and compliance requirement.
• Spooky Action at a Distance: Quantum Entanglement from Einstein to the Quantum Internet – Important distinction: AI "superposition" is not quantum superposition; a metaphor, not physics.
• Psychological Safety: The Hidden Force Behind High-Performing Teams – Representation and cognition: the bridge to how concepts arise in (biological as well as artificial) neural systems.

Sources and Further Reading

• Anthropic / Transformer Circuits – Toy Models of Superposition (Elhage et al., 2022): https://transformer-circuits.pub/2022/toy_model/index.html
• Anthropic / Transformer Circuits – Towards Monosemanticity: Decomposing Language Models With Dictionary Learning (2023): https://transformer-circuits.pub/2023/monosemantic-features/index.html
• Anthropic / Transformer Circuits – Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (2024): https://transformer-circuits.pub/2024/scaling-monosemanticity/
• Anthropic – Golden Gate Claude (May 2024): https://www.anthropic.com/news/golden-gate-claude
• Anthropic / Transformer Circuits – On the Biology of a Large Language Model (2025): https://transformer-circuits.pub/2025/attribution-graphs/biology.html
• Anthropic – Open-sourcing circuit-tracing tools (2025): https://www.anthropic.com/research/open-source-circuit-tracing
• Toy Models of Superposition (arXiv version, 2209.10652): https://arxiv.org/abs/2209.10652
• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models (2025): https://arxiv.org/html/2503.05613v3

Created as part of the daily learning workflow. Field of interest: AI. Estimated reading time: ~30 minutes.

← All articles