Attention Is All You Need: The Transformer, Self-Attention, and the Architecture of Modern AI

The Hook: A Sentence That Changed the World

In June 2017, a team of eight researchers at Google uploaded a paper to a preprint server whose title read like a provocation: "Attention Is All You Need." The title was half joke, half manifesto. It claimed that an entire branch of engineering craft — the recurrent neural networks that translation research had been polishing for years, and the convolutional networks that had come from image processing — could simply be thrown away. All that needed to remain, it argued, was a single mechanism called attention.

It is worth remembering how bold this was. At the time, recurrent networks built from so-called LSTM cells were the undisputed royal road for anything involving language: machine translation, speech recognition, text summarization. They carried a built-in, intuitive logic — they read a sentence word by word, left to right, the way a human reads. The proposal to abandon that whole temporal logic and instead let every word look at every other word simultaneously seemed almost perverse.

And yet: hardly any scientific paper of the past two decades has reshaped the world so deeply. The architecture introduced in that paper — the Transformer — is today the foundation of virtually every major AI system you know. Every large language model, from the GPT series to Claude to Gemini and the open models, is at its core a Transformer. The image generators, the protein-folding machines, the code assistants: Transformer, Transformer, Transformer. The "T" in "GPT" literally stands for "Transformer." An architecture originally designed to translate German into English became the universal computational building block of an entire technological era.

This article takes you along the full route: from the problem the Transformer solved, through the surprisingly simple core idea of attention, its mathematical form, the complete blueprint of the architecture, all the way to the question of why this particular design won — and whether the triumphant 2017 title still holds today, in the age of Mamba and state space models. By the end you should know not just what a Transformer is, but why it looks the way it does.

Part 1: The Problem Before the Transformer

The Sequential Bottleneck

To understand why the Transformer was a revolution, you have to understand what it was up against. Until 2017, recurrent neural networks (RNNs) dominated the processing of sequences. An RNN reads a sequence — say, the words of a sentence — strictly in order. It maintains an internal state, a kind of memory, and updates that state with each new word. Word one changes the state, then word two changes it further, then word three, and so on. The state after the last word is a compressed summary of the entire sentence.

This design has a fundamental weakness that cannot be optimized away: it is inherently serial. To process word 100, the network must first have passed through words 1 through 99, because each step depends on the result of the previous one. On modern graphics processors (GPUs), which draw their enormous power precisely from performing thousands of operations simultaneously, this is a disaster. An RNN leaves that parallelism untapped. It forces a machine built for parallel work to compute in single file.

The Problem of Long Memory

A second weakness compounded the first. Information had to travel a long way inside an RNN. If the first word of a sentence matters for the meaning of the last word — think of a German subordinate clause in which the decisive verb only appears at the very end — then that signal must be passed through all the intervening processing steps without being diluted. In practice this succeeded only imperfectly. The notorious problems of vanishing and exploding gradients made it hard to learn dependencies across large distances. LSTM and GRU cells, ingenious variants of the RNN cell with built-in "gates," alleviated the problem but never fully solved it.

Put vividly: in an RNN the path between two words is as long as the number of words that lie between them. The farther apart two related words stand, the harder it becomes to capture their relationship.

The First Spark: Bahdanau and the Attention of 2014

The idea of attention was not invented in 2017. Its birth came in a 2014 paper by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio titled "Neural Machine Translation by Jointly Learning to Align and Translate." These researchers worked on the classic encoder-decoder model for translation. The encoder — an RNN — squeezed the entire source sentence into a single fixed-length vector, the context vector. The decoder — another RNN — was supposed to generate the whole target sentence from that one vector.

This was precisely where a bottleneck sat. The whole sentence, whether five or fifty words long, had to pass through that single bundle of numbers. For long sentences, quality collapsed, because a fixed-size vector simply did not have enough room to store every nuance. Bahdanau and colleagues found an elegant solution: instead of nailing the decoder to a single context vector, they allowed it, for each word it produced, to look back at all the encoder's states and pick out the ones that were relevant at that moment. When translating the word "cat," the decoder could pay targeted "attention" to the part of the source sentence that was about the cat.

This mechanism of weighted looking-back was the first attention. It notably improved the translation of long sentences. But — and this is crucial — it still sat on top of an RNN. Attention was an aid; the serial backbone remained. The real revolution came three years later with a radical thought: what if attention is not the aid but the main thing? What if you drop the RNN entirely?

Part 2: The Core Idea — Attention as Soft Lookup

A Library in Which Every Book Partly Fits

Before we reach formulas, we should grasp the core idea of self-attention in a way that sticks. Imagine you have a question and search a library for answers. In a classic database this works on an all-or-nothing principle: you have a key, and if your query matches it exactly, you get the associated value — otherwise nothing.

Attention does the same thing, only softly. Your query is compared with every key in the library. Instead of a single hit you get a whole landscape of similarities: book A matches 60 percent, book B 30 percent, book C 10 percent. The answer you get back is a weighted blend of all the values, with each book contributing in proportion to how well it fits. No book is entirely ignored, none counts alone.

That is exactly what self-attention does with the words of a sentence. Each word poses a query: "Which other words in this sentence matter for my understanding?" It compares its query with the keys of all words — including itself — and blends their values in proportion to similarity into a new, context-enriched representation of itself.

The Classic Example: What Does "It" Refer To?

Take the sentence: "The animal didn't cross the street because it was too tired." What does the word "it" refer to? The animal or the street? For a human the answer is obvious — only animals can be tired, not streets. Change the sentence to "… because it was too wide," and the reference snaps abruptly to the street.

Self-attention solves this problem by having the word "it" send out a query and find that comparing its query against the key of "animal" yields high similarity, while against "street" it yields low similarity. Consequently, the new representation of "it" is infused above all with the meaning of "animal." The word "it" is literally charged with the sense of "animal." And the beautiful part: this mapping is not hand-coded. It emerges from parameters the network learned by itself during training on millions of sentences.

The decisive difference from the RNN: the word "it" does not have to walk back step by step to the word "animal." It looks directly there in a single step, no matter how far apart the two stand. The path between any two words has constant length — it is always exactly one step long.

Part 3: Scaled Dot-Product Attention — The Mathematics, Calmly

Three Roles for Every Word: Query, Key, Value

Now let us make it concrete, but without fear of the math. Each word enters the mechanism first as a vector — a list of numbers representing the word in what is called the embedding space. From this single vector the Transformer produces three different versions by multiplying it with three learned weight matrices:

• the query vector (Q): "What am I looking for?"
• the key vector (K): "What do I offer others as a search criterion?"
• the value vector (V): "What content do I pass on when someone attends to me?"

So the same word representation plays three roles at once. It asks, it is asked, and it delivers content. Query and key govern who attends to whom; the value determines what is transmitted in the process.

The Formula and What It Means

The famous formula from the paper by Vaswani and colleagues reads:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

Let us go through it piece by piece, because each part has a clear purpose:

First, \(QK^\top\). This is the dot product of each query with each key. The dot product of two vectors measures their similarity: if they point in the same direction it is large; if they are perpendicular it is zero. The result is a table of raw scores — for each pair of words a number saying how well one's query matches the other's key. For a sentence with \(n\) words this produces an \(n \times n\) matrix. It is precisely in this matrix that the price the Transformer pays will later reside — we will come to that.

Second, the division by \(\sqrt{d_k}\). This is the "scaled" part that gives the mechanism its name. \(d_k\) is the dimension of the key vectors, 64 in the original paper. For high-dimensional vectors the dot products become very large in magnitude. Large values drive the subsequent softmax function into extreme regions where its derivative is practically zero — the notorious vanishing gradients that bring learning to a halt. Dividing by the square root of the dimension keeps the values in a well-behaved range. It is a small, technical, but indispensable trick.

Third, the softmax. This function turns each row of the table into a probability distribution: numbers between zero and one that sum to one. These are the actual attention weights. For each word they say: "Distribute your attention 60 percent onto this word, 30 percent onto that one, 10 percent onto a third."

Fourth, the multiplication by \(V\). The attention weights are now used to form a weighted sum of the value vectors. The result, for each word, is a new vector — its context-enriched representation. The word "it" from our example now carries within it a strong share of the "animal" value.

The whole formula is, in the end, nothing more than a soft, differentiable lookup in a table: compare, normalize, blend. Its beauty lies in the fact that it can be expressed entirely as matrix multiplications — and that is precisely what graphics processors do at breathtaking speed and, crucially, for all words simultaneously.

Part 4: Multi-Head Attention — Several Viewpoints at Once

Why One Head Is Not Enough

A single attention operation forces each word to distribute its attention in one way. But language is many-layered. A word stands simultaneously in several relationships: it has a grammatical role (subject, object), a semantic reference (what it points to), a syntactic neighborhood, perhaps a stylistic tone. A single attention distribution cannot cleanly separate these different relationship types at the same time.

The solution is called multi-head attention. Instead of computing attention once, one does it multiple times in parallel — eight times in the original paper. Each of these "heads" has its own independently learned Q, K, and V matrices and operates in its own lower-dimensional subspace. Because the dimension per head is reduced accordingly (with eight heads and a model dimension of 512, each head works with 64 dimensions), the total computational cost stays roughly the same as a single full-size head.

What the Heads Learn

The intriguing thing is what happens in practice: different heads specialize without being told to. Analyses of trained Transformers have shown that some heads focus on the immediately preceding word, others track relationships between pronouns and their referents, still others capture syntactic patterns such as the mapping of verbs to their objects. Each head, in a sense, looks at the same sentence through a different pair of glasses. In the end the results of all heads are concatenated and fused through another learned matrix into a shared representation.

I am of the opinion that this very division into specialized heads is one of the reasons Transformers lend themselves so well to interpretation — research on mechanistic interpretability picks up exactly here when it tries to decode individual heads and their function. It is well established that such specialized heads exist; how clean the division of labor really is in any given case remains a subject of ongoing research.

Part 5: The Complete Blueprint

Attention is the heart of the Transformer, but a heart alone is not yet an organism. To turn the attention mechanism into a working architecture, a few more components are needed. Let us walk through the complete blueprint.

Positional Encoding: Where Order Comes From

Here lies a subtlety. Attention treats a sentence like a set of words, not an ordered sequence. When every word looks at every other, the computation initially encodes nowhere which word came first. "The dog bites the man" and "The man bites the dog" would yield the same attention scores — an obvious problem, because word order carries meaning.

The solution is positional encoding. Before the words enter the first attention block, each word vector has a second vector added to it that encodes its position in the sentence. In the original paper this is done with an elegant construction of sine and cosine functions of different wavelengths. You can picture it as a set of hands rotating at different speeds — together they give each position a unique "fingerprint." This way the model does know which word sits where, without the processing itself having to become serial. Later variants refined this, for instance through the now widely used rotary position embeddings (RoPE), but the basic principle remains: position is fed into the representation as an additional signal.

Feed-Forward Networks, Residuals, and Normalization

After the attention, each Transformer block contains a small feed-forward network applied separately to each position: two linear layers with a nonlinear activation in between. Whereas attention exchanges information between words, this network processes the information within each individual word further. I am of the opinion that one can aptly understand the Transformer as an interplay of two motions: attention mixes horizontally across the words, the feed-forward network thinks vertically at each position.

Two more techniques are added, less spectacular but indispensable for training deep networks. Residual connections (skip connections) route a block's input around it and add it to its output. This gives the signal and the gradients a "highway" through the entire network and makes it possible to stack very many blocks on top of one another without training collapsing. Layer normalization keeps the numerical values in a stable range at each step. Together these are the silent heroes that make a tower of dozens of Transformer blocks learnable at all.

Encoder, Decoder, and the Three Shapes

The original Transformer was meant for translation and therefore had two halves: an encoder that reads in the source sentence and turns it into a rich representation, and a decoder that generates the target sentence from it word by word. The decoder uses a special variant, masked attention: when producing a word, it may only look at words already produced, not into the future — otherwise it would cheat during training by reading off the answer.

From this basic construction three families have developed that are worth knowing:

Remarkably, the family that dominates today — large language models — settled on the simplest of these shapes: pure decoders that do nothing but predict the next word over and over again.

Part 6: Why the Transformer Won

Parallelism and Short Paths

Let us return to the two weaknesses of RNNs — serial and long-stretched — and see how the Transformer abolishes both in one stroke.

First, parallelism. Because in self-attention all words look at all words simultaneously, there is no forced order in the computation. All positions of a sentence can be processed in one go. This fits perfectly with graphics processors, built for massively parallel matrix multiplications. This is precisely what made it practical in the first place to train models on data volumes and at sizes that would have been unthinkable with RNNs.

Second, the short path. In an RNN the distance between two words grows with their separation in the sentence. In self-attention it is constant: each word reaches every other in a single step. Long-range dependencies — a reference spanning a hundred words — are in principle just as easy to learn as neighborly ones.

The following table, in its essentials from the original paper, sums up the comparison (\(n\) = sequence length, \(d\) = dimension of the representation):

You can see the balance of forces immediately: self-attention needs zero serial steps and has a constant path — ideal for parallel machines and long-range references. The price stands in the first column: the compute grows with \(n^2\), the square of the sequence length. That number will occupy us again in Part 8.

An Unexpected Gift: Scalability

There was an effect whose full significance even the Transformer's inventors could not foresee in 2017. The Transformer turned out to be extraordinarily scalable. Make it bigger — more blocks, more dimensions, more parameters — and feed it more data and more compute, and its capabilities improve in an astonishingly regular, predictable way. This regularity was later measured precisely in the neural scaling laws. The Transformer was thus not merely a better architecture but one that lent itself as a vehicle for an unprecedented race in size. Without the Transformer's parallelizability, one simply could not have trained the giant models of the present.

Part 7: The Children of the Transformer

The years after 2017 were a Cambrian explosion. From the one blueprint an entire family tree arose.

BERT (2018, Google) took only the encoder half and trained it with a simple but ingenious task: randomly hide some words of a text and have the model guess them. Because the model may see the context from both sides in doing so, it learns a deep, bidirectional understanding of language. BERT became the workhorse for search, classification, and all the tasks that are about understanding rather than generating.

The GPT series (from 2018, OpenAI) went the other way: only the decoder, trained on the simplest conceivable task — predict the next word. What sounds modest at first turned out to be surprisingly powerful. A model that becomes good enough at predicting the next word must, in order to do so, implicitly acquire grammar, facts, patterns of reasoning, and a wealth of world knowledge. As size grew, capabilities appeared that no one had explicitly programmed: translating, arithmetic, programming, following instructions. Today's chat and reasoning models are the direct descendants of this line.

And the Transformer did not stop at language. The Vision Transformer cut images into small patches and treated them like words — with excellent results. In biology, a Transformer-related attention architecture helped crack the protein-folding problem. Transformers today process audio, video, time series, molecular structures. The architecture proved remarkably domain-agnostic: anything that can be construed as a sequence of building blocks with internal relationships is fodder for a Transformer.

Part 8: The Price of Attention — And Whether the Title Still Holds

The Quadratic Achilles' Heel

Recall the \(n \times n\) matrix from Part 3. When every word looks at every other, the number of comparisons grows with the square of the sequence length. Double the length of the text and the effort quadruples. For a sentence this is irrelevant; for a book, a codebase, or an hours-long video stream it quickly becomes crushing — both in compute and, often more painfully, in the memory required for that enormous matrix. This is why "context length" — how much text a model can consider at once — was long a hard-fought measure.

FlashAttention: The Same Computation, Organized More Cleverly

An important breakthrough came in 2022 with FlashAttention (Tri Dao and colleagues). The insight was not to change the mathematics but how it runs on the hardware. On a graphics processor the actual arithmetic is often not the bottleneck, but the shuffling of data back and forth between the large, slow memory and the small, fast one. FlashAttention computes attention in tiles that fit into the fast memory and avoids ever laying down the full \(n \times n\) matrix in slow memory. The result is the same exact attention, but with a memory requirement that grows linearly rather than quadratically with length, and with considerably higher speed. Later versions — FlashAttention-2 and -3 — pushed this further, among other things through better adaptation to new GPU generations. FlashAttention has dramatically enlarged the practically usable context lengths and is today built into almost every serious training run.

Mamba and State Space Models: Was Attention Not All You Needed After All?

More interesting still is another line of attack that questions the basic assumption itself. In late 2023, Albert Gu and Tri Dao introduced Mamba, a so-called selective state space model (SSM). The idea in a sense returns to the RNN notion: instead of letting every word look at every other, Mamba maintains a compressed state that travels along the sequence — but with a clever, input-dependent mechanism that decides which information to keep and which to forget. The clincher: the effort grows only linearly with sequence length, not quadratically. For very long sequences Mamba promises significantly faster processing and lower memory use.

This makes the provocation of the 2017 title an open question today. Was attention really "all you need"? Current research points to a more nuanced picture. First, the follow-up work Mamba-2 (2024) revealed a deep theoretical connection: under the umbrella of state space duality, attention and state space models turn out to be two sides of one coin — a certain form of linear attention is mathematically equivalent to a state space model. The supposed rivals are relatives. Second, in practice it is increasingly not the either-or that prevails but the both-and: hybrid models that mix cheap state space layers with a few expensive attention layers. Pure state space models, after all, have weaknesses of their own — they struggle to copy long passages verbatim or to reach back deliberately to a distant detail; precisely what attention excels at.

I am of the opinion that the most honest state of affairs is this: attention is not everything, but it is indispensable. It has proven so fundamental that even its challengers ultimately tend to build it back in in some form. The 2017 title may have been an overstatement — but one that has proven remarkably durable.

The Central Takeaway

If you take a single thing from this article, let it be this: the Transformer was a revolution because it turned a serial problem into a parallel one. The old recurrent networks forced the processing of language through a narrow bottleneck of step-by-step computation. The Transformer tore down that bottleneck by placing a simple but powerful question at the center — "which other elements are relevant to this element right now?" — and answering that question for all elements simultaneously. From this one idea, thought through consistently and surrounded with the right components, grew the foundation of an entire technological era.

The practical takeaway, especially for you as an engineer: the next time you use a language model, remember that "context" is not a magical property but a computation with a price. The quadratic nature of attention explains why long inputs are expensive, why context windows remain a hard-fought measure, and why the choice of what you put into the context matters so much. Whoever understands that every token is compared with every other makes better decisions about prompt length, chunking, and architecture — and at the same time sees through the race for ever more efficient alternatives that will shape the coming years of AI.

And there is a more general lesson in it too: sometimes the greatest progress consists not in refining an existing tool but in dropping one of its deeply rooted assumptions. For years no one had seriously doubted that language must be read in order. The courage to question exactly that was the real breakthrough.

A Question to Reflect On

The Transformer owes its success in large part to the fact that it fit perfectly the hardware of its time — massively parallel graphics processors. The superior recurrent networks of an earlier day failed not only on their mathematics but because they could not exploit the available computing machinery. How many of our "best" ideas in computer science are in truth the best for the hardware that happens to be available — and which idea discarded today might return once the machines beneath it change?

Cross-References in the Vault

This article forms, in a sense, the architectural foundation on which several earlier posts rest. Anyone who wants to understand what actually goes on inside the specialized attention heads and internal representations of a Transformer will find the continuation in The Ghost in the Machine: How to Read a Neural Network From the Inside. The question of why merely making this architecture larger so reliably yields better capabilities is treated by How Big Is Big Enough? Scaling Laws, Chinchilla, and the Measurement of AI. Whether such a system, which only shifts word forms according to learned patterns, really understands is the philosophical flip side explored by The Chinese Room: Searle and the Question of Whether Machines Can Understand. A fascinating parallel from neuroscience — the brain as a predictive machine — is drawn by The Predictive Brain: Predictive Processing and the Illusion of Perception. And anyone who wants to know within which regulatory framework these models will be operated in Europe should read The Pyramid of Risk: How the EU AI Act Tames Artificial Intelligence – and Why It Concerns the Whole World.

Sources

• Vaswani, A. et al. (2017): Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS). https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf (Preprint: https://arxiv.org/abs/1706.03762)
• Bahdanau, D., Cho, K., Bengio, Y. (2014): Neural Machine Translation by Jointly Learning to Align and Translate. https://arxiv.org/abs/1409.0473
• Devlin, J. et al. (2018): BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805
• Dao, T. et al. (2022): FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. https://arxiv.org/abs/2205.14135
• Gu, A., Dao, T. (2023): Mamba: Linear-Time Sequence Modeling with Selective State Spaces. https://arxiv.org/abs/2312.00752
• Dao, T., Gu, A. (2024): Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. https://arxiv.org/abs/2405.21060

← All articles