The State That Must Remember Everything: Mamba, State Space Models, and the Alternative to the Transformer

The Hook: The Bill That Grows With Every Token

There is one number that comes up in every discussion of large language models, and almost nobody reads it as an architectural decision: the context length. 128,000 tokens. 200,000. A million. It gets sold like a product feature — "our model can read an entire book" — but what it really is, is a statement about cost.

The reason lies at the heart of the Transformer, in self-attention. Every token compares itself against every other token. At 1,000 tokens that is a million pairwise comparisons; at 10,000 tokens, a hundred million; at 100,000 tokens, ten billion. Compute grows quadratically with sequence length. That is the sentence everyone knows.

The second sentence is the one that actually hurts in production and gets said far less often: when generating text, it is not only compute time that grows, but memory — linearly and relentlessly. To write token number 50,001, an autoregressive model must keep the key and value vectors of all 50,000 preceding tokens on hand. This KV cache is not some intermediate buffer you can optimize away; it is the model's memory. It sits in the most expensive memory in the data center, it grows with every word written, and it grows per concurrent user. Anyone who wants to understand why inference at long context lengths is so disproportionately expensive should not look at the FLOPs but at the gigabytes.

And now the question that carries this article: does it have to be this way?

There is an older answer, and it is no. Recurrent neural networks — LSTMs, GRUs — never had this problem at all. An RNN maintains a fixed-size state. It reads a token, updates the state, forgets the token. Memory per generated token: constant. Compute per token: constant. Total cost for a sequence of length L: linear, not quadratic.

The catch is well known: RNNs were worse. They did not parallelize well during training, their gradients vanished across long distances, and in 2017 the Transformer displaced them for exactly that reason.

What has happened since 2020 is therefore one of the most interesting stories in AI architecture research: a group around Albert Gu and Tri Dao revived the recurrent idea not by making RNNs better, but by replacing them with a mathematical object borrowed from control theory — the linear state space model. Out of that came S4, Mamba, state space duality, and most recently, in March 2026, Mamba-3. And then — the part I consider the actually instructive one — practice did not choose either side. It chose the mixture.

Part 1: Two Kinds of Memory

Before we get to the mathematics, one clean distinction is worth drawing, because it explains everything that follows.

The Transformer has a growing archive. It stores every token uncompressed and can afterwards access any token exactly. That is an associative memory with perfect recall. The price: the archive grows, and every access must search all of it.

The recurrent model has a summary. It compresses the entire past into a vector of fixed dimension. Whatever does not appear in that summary is irretrievably lost. The price: information loss. The gain: constant cost.

This distinction is not an implementation question but an information-theoretic one. A state holding d real numbers can carry only a bounded amount of information. If 100,000 tokens flow into it and d stays at 128, compression must happen. The entire research history that follows is the attempt to make that compression as good as possible — and, in the end, the recognition of where it cannot in principle be good enough.

Part 2: The State Space Model — A Loan From Control Theory

The Equation

The linear state space model is roughly as old as modern control theory and describes a continuous system with two equations:

x'(t) = A x(t) + B u(t)      (how the state evolves)
y(t)  = C x(t) + D u(t)      (what is observed from outside)

u(t) is the input signal, x(t) the hidden state, y(t) the output. A is the state matrix: it alone determines how the state changes over time when left alone — and therefore how long it remembers anything.

For a language model you need the discrete variant. You choose a step size Δ and obtain a recurrence of the form

h_t = Ā h_{t-1} + B̄ u_t
y_t = C h_t

This looks like an RNN. The decisive difference: there is no nonlinearity between time steps. That sounds like a weakness, and it is the central trick I will come back to in Part 2.3.

HiPPO: Which A Actually Remembers

The first important paper came in 2020. HiPPO: Recurrent Memory with Optimal Polynomial Projections (Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher Ré; arXiv:2008.07669, NeurIPS 2020) asked the question: if a fixed-size state is to summarize an ever-lengthening past — what does the optimal summary look like?

The authors' answer is elegant: treat the problem as online function approximation. The past signal is a function; the state stores its coefficients with respect to a polynomial basis. Given a specification of how important each point in the past is (a "measure"), HiPPO produces the optimal update rule — and with it one particular, derived matrix A.

The result is the point: the matrix A is neither learned nor guessed, but deduced from an approximation principle. The HiPPO-LegS variant (Legendre polynomials with a scaled measure) "remembers all history, avoiding priors on the timescale" and has provably bounded gradients — precisely the property whose absence ruined LSTMs over long distances. Empirically, HiPPO-LegS reached 98.3% on Permuted MNIST, then the standard long-memory test, and outperformed RNN and neural ODE baselines by 25 to 40 percentage points under shifted timescales.

So it was clear: a linear state can remember things for a very long time, if you pick the right matrix. What was unclear was how to compute with it.

S4: The Same Operation, Two Faces

Here is the payoff of linearity. Because no nonlinearity sits between the time steps, the recurrence can be unrolled — and what falls out is a convolution:

y = K * u      with    K = (C B̄, C Ā B̄, C Ā² B̄, …, C Ā^(L-1) B̄)

Which means: the same model has two exactly equivalent computational forms.

This is a property a nonlinear RNN can never have, and it solves exactly the problem RNNs failed on: you train in parallel like a CNN and infer sequentially like an RNN. Same model, two modes.

The difficulty was actually computing the kernel K: the naive form requires powers of Ā and is, as the paper puts it, "prohibitive" in computation and memory. Efficiently Modeling Long Sequences with Structured State Spaces (Albert Gu, Karan Goel, Christopher Ré; arXiv:2111.00396, ICLR 2022) solved this with a new parameterization: A is written as a diagonal matrix plus a low-rank correction ("conditioning A with a low-rank correction"), so that it becomes stably diagonalizable and the kernel computation collapses into the well-studied evaluation of a Cauchy kernel.

S4's results were, for 2021, spectacular:

• 91% accuracy on sequential CIFAR-10 with no data augmentation and no auxiliary losses — an image read as a pixel sequence, on par with a larger 2-D ResNet.
• 60× faster generation than comparable Transformers on image and language modeling.
• State of the art on every task of the Long Range Arena — the benchmark from Long Range Arena: A Benchmark for Efficient Transformers (Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler; arXiv:2011.04006, ICLR 2021), which tests sequences from 1,000 to 16,000 tokens across text, images, and mathematical expressions.
• And the sentence that electrified the field: S4 solved Path-X, the length-16k task, "that all prior work fails on." Ten evaluated efficient Transformer variants had previously performed at chance level there.

And Then Came Language

There was only one problem, and it was a big one: on language, S4 was not good. On synthetic long-range tasks, on audio, on genomic data — convincing. On text — clearly behind Transformers of the same size.

That was not a detail but a puzzle. And the explanation that arrived in 2023 is why this article does not end with S4.

Part 3: Mamba and Selection

The Diagnosis

Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Albert Gu, Tri Dao; arXiv:2312.00752, submitted 1 December 2023) names the weakness precisely: all subquadratic architectures — linear attention, gated convolutions, recurrent models, SSMs — fail at the same point, namely their "inability to perform content-based reasoning."

What does that mean concretely? S4 is a time-invariant system. The matrices Ā, B̄, C and the step size Δ are fixed numbers after training. The model processes every token with exactly the same dynamics. It is a filter — a very good, mathematically derived filter, but one that cannot distinguish whether a filler word or a proper name is coming in.

For audio that is fine: an audio signal is largely uniform in information density. For language it is fatal. Language consists of a few highly informative tokens in a sea of grammar. A model that cannot decide "this I remember, this I discard" is forced to squander its finite state uniformly.

The Solution and Its Price

Mamba's first idea fits in one sentence: make the SSM parameters functions of the input. Δ, B, and C are computed per token from the token itself. This creates a selection mechanism: "allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token."

The step size Δ thereby becomes the accelerator and brake pedal of memory. A large Δ means: write this input strongly into the state, overwrite what was there. A Δ near zero means: leave the state untouched, ignore the token. The model learns to decide for itself what it keeps. The kinship with LSTM and GRU gating is obvious and is drawn explicitly by the authors — except that here a mathematically derived state model is being gated, rather than a hand-built cell state.

And here lies the price, and it is steep. As soon as the parameters depend on the input, the system is no longer time-invariant — and the convolutional form ceases to exist. The entire trick from Part 2.3, parallel training via FFT, is gone. The paper says so unmistakably: "even though this change prevents the use of efficient convolutions." Mamba throws away precisely the property that made S4 trainable in the first place.

The Hardware-Aware Way Out

The answer to that is the paper's second contribution, and I consider it the underrated one: a "hardware-aware parallel algorithm in recurrent mode." Three elements:

1. Parallel scan instead of convolution. A linear recurrence is an associative operation. Associative operations can be evaluated in parallel as a prefix sum (scan) in logarithmic depth — even when the coefficients differ at every step. Parallelism is therefore preserved, just by a different route.
2. Kernel fusion. Discretization, scan, and multiplication by C are placed in a single GPU kernel. The large state tensor is never written to slow HBM main memory but stays in fast SRAM.
3. Rematerialization. For the backward pass, intermediate states are not stored but recomputed — compute time traded against memory bandwidth.

This is the same line of reasoning that made FlashAttention famous, and it comes from one of the same two authors: the bottleneck of modern GPUs is not arithmetic but memory movement. Whoever reduces the number of HBM accesses wins, even if they compute more.

The paper's result: Mamba achieves "5× higher throughput than Transformers" at inference, scales linearly in sequence length, and — the most remarkable sentence — "its performance improves on real data up to million-length sequences." On language modeling, Mamba-3B outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and in downstream evaluation. Architecturally, Mamba is radically tidied up: a homogeneous block, "without attention or even MLP blocks."

Part 4: The Duality — Transformers Are State Space Models

Half a year later came the paper I consider the most intellectually beautiful of the series: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Tri Dao, Albert Gu; arXiv:2405.21060, ICML 2024).

The claim in the title is not marketing hyperbole but a mathematical result. Both families can be written as multiplication by a structured semiseparable matrix. Attention computes that matrix explicitly — hence the quadratic cost. An SSM computes the same matrix operation via a decomposition that never materializes the full matrix — hence linear cost. They are two orderings of computation over the same object, and which one is cheaper depends on whether the sequence or the state dimension is larger.

This perspective is called state space duality (SSD), and it has two consequences.

Theoretically, it dissolves the strange separation the literature had worked under for years. Linear attention, gated linear attention, selective SSMs, RetNet — these are not competing inventions but points in a space whose coordinates can be named.

Practically, it yields Mamba-2: a "refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling." The mechanism is that the recurrence can be formulated blockwise as a matrix multiplication — and matrix multiplication is exactly the operation tensor cores were built for. Mamba-1's scan did not use them.

I am of the opinion that this is the most important methodological lesson of the whole topic, and it is not AI-specific: when two design families look irreconcilable, the most rewarding question is not which one wins, but what common object contains both as special cases. The answer usually delivers both — better theory and faster implementations.

Part 5: The Hard Limits — Where the State Is an Illusion

It would be tempting at this point to tell the story as a replacement. It is not, and 2024 delivered two precise counter-findings. Both matter more than any benchmark table, because they make structural claims.

Copying: Fixed State Size Is an Information Bottleneck

Repeat After Me: Transformers are Better than State Space Models at Copying (Samy Jelassi, David Brandfonbrener, Sham M. Kakade, Eran Malach; arXiv:2402.01032, 2024) takes the simplest conceivable task: repeat a string from the context verbatim.

The result is a proof, not a measurement. A two-layer Transformer can copy strings of exponential length, because attention can reach directly back to the position where the character stood. Models with a fixed latent state — the authors call them "generalized state space models" — are by contrast "fundamentally limited by their fixed-size latent state." Anyone who must reproduce n bits exactly needs n bits of state. That is not a training question and not a hyperparameter question; it is information theory. Empirically the authors confirm the finding on synthetic tasks and then on pretrained models, where Transformers "dramatically outperform" state space models at copying and retrieving information from context.

That is the hard limit for a vast class of practically important tasks: finding quotations, reproducing code snippets exactly, picking up an ID from a log file, retrieval-augmented generation. Everywhere exact recall rather than summarization is required, the growing archive is in principle superior to the compressed state.

State Tracking: Recurrence Alone Is Not Enough

The second finding is even more surprising. The Illusion of State in State-Space Models (William Merrill, Jackson Petty, Ashish Sabharwal; arXiv:2404.08819, ICML 2024) tests the obvious expectation that a recurrent model should at least beat the Transformer at state tracking. Transformers are known to be unable to express certain sequential computations; SSMs are architecturally close to RNNs; so they ought to manage it.

"Surprisingly, the answer is no." The analysis shows that the expressive power of SSMs is "limited very similarly to transformers": they cannot express computation outside the complexity class TC⁰. Concretely, they cannot solve permutation composition — and it follows that they cannot reliably track chess moves in certain notations, cannot evaluate code, and cannot keep entities consistent across a long narrative. Experiments with Mamba-style models confirm this. The title states the result: despite its recurrent formulation, the "state" in an SSM is an illusion.

The deeper reason is linearity — exactly what was the trick in Part 2.3. A linear recurrence with no nonlinearity between steps can be collapsed into a flat, parallelizable form — and what can be evaluated flat and in parallel cannot have genuine sequential depth. The parallelizability that makes training possible is the same property that bounds the expressive power. That is not a bug but a conservation law.

Part 6: The Synthesis — Why Practice Went Hybrid

If attention can do exact recall and Mamba can do cheap long contexts, then the obvious answer is not either/or but a division of labor. That is exactly what happened, and it happened fast.

Jamba (Opher Lieber, Barak Lenz, Hofit Bata et al., AI21 Labs; arXiv:2403.19887, 2024) was the first large model of this kind: "a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture" that interleaves blocks of both families and additionally adds experts in some layers. The result of the particular configuration they implemented is telling: a model that "fits in a single 80GB GPU," offering "high throughput and small memory footprint compared to vanilla Transformers," with strong results "for up to 256K tokens context length." The authors stress that the mixing ratios are not arbitrary: "some of them are crucial in large scale modeling."

Nemotron-H (NVIDIA; arXiv:2504.03624, 2025) carried this into a product family — 8B and 56B/47B parameters — and states the design goal in economic terms: "As inference-time scaling becomes critical for enhanced reasoning capabilities, it is increasingly becoming important to build models that are efficient to infer." The intervention is exactly the expected one: "we replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers that perform constant computation and require constant memory per generated token." The outcome, per the abstract: better or on-par accuracy versus Qwen-2.5-7B/72B and Llama-3.1-8B/70B, while being up to 3× faster at inference. The additionally compressed variant Nemotron-H-47B-Base, produced by pruning and distillation, achieves accuracy similar to the 56B model while being 20% faster.

The structure behind both cases is the same: the majority of layers are Mamba layers, running cheaply and with constant memory along the sequence. A minority of attention layers remains as a "recall organ," for precisely the cases Jelassi et al. provably isolated. The KV cache shrinks by the factor by which attention layers were removed — and that is the entire economics of the design.

Part 7: Mamba-3 (2026) — What the Third Round Added

The most recent station is Mamba-3: Improved Sequence Modeling using State Space Principles (Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, Albert Gu; arXiv:2603.15569, submitted 16 March 2026). The abstract is remarkably self-critical about its own model family: "many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice."

The three changes address exactly the findings from Part 5:

1. A more expressive recurrence derived from discretization. Mamba-1 and Mamba-2 correspond, the paper argues, to an exponential-Euler discretization. Mamba-3 uses a trapezoidal variant whose state update has three terms rather than two — it incorporates the previous input alongside the current one: h_t = α_t h_{t-1} + β_t B_{t-1} x_{t-1} + γ_t B_t x_t. This can, the authors say, be expanded to reveal an implicit convolution over the SSM input, and can replace the short causal convolution used so far. Numerical integration as an architectural decision — that is the core of the "state space principles" in the title.
2. A complex-valued state update. Per the paper it is "equivalent to a data-dependent rotary embedding" and enables state tracking that real-valued variants cannot deliver. The reported effect is the direct counter-finding to Merrill et al.: on synthetic arithmetic state-tracking tasks, the rotation-like Mamba-3 variant solves the tasks near perfectly, while Mamba-3 without that component and Mamba-2 perform no better than random guessing.
3. A MIMO formulation (multi-input, multi-output). The state update switches from an outer-product form to a matrix-multiplication form. The purpose is hardware economics: during decoding a GPU is memory-bandwidth-bound, not compute-bound — so FLOPs per step can be increased (up to fourfold relative to Mamba-2 at fixed state size, per the paper) without decode latency per token rising noticeably. More model quality at zero cost in waiting time.

The reported numbers at 1.5 billion parameters: Mamba-3 improves average downstream accuracy by 0.6 percentage points over the next-best model (Gated DeltaNet), the MIMO variant adds a further 1.2 points, for 1.8 points in total. And the result that best captures the direction of travel: Mamba-3 reaches perplexity comparable to Mamba-2 with half the state size.

An honest framing belongs here: these are results at 1.5 billion parameters, not 70 billion, and they come from the architecture's own authors. I am of the opinion that architecture comparisons of this kind should be treated as robust only once independent groups have reproduced them at production-relevant scale — the story of S4 on audio versus S4 on language is precisely the warning against inferring generality from good numbers on one slice.

Part 8: The Map

Two things stand out. First: the "state tracking" row is equally bounded in all three columns — that is Merrill's result, and it is why chain-of-thought and external tools are not an embarrassed workaround but an architectural patch over a structural gap. Second: the hybrid column is not a compromise that does a little less of everything, but competitive in every practically important row. That is rare, and it is why the production models ended up there.

The Central Takeaway

1. The bottleneck is not compute but memory. Quadratic attention is the known problem; the linearly growing KV cache is the more expensive one. Mamba's promise is less "compute faster" than "constant memory per token" — and that is precisely why it is an inference argument, not a training argument.
2. Compression versus recall is the real trade. A fixed-size state is a summary; a KV cache is an archive. Jelassi et al. proved that no amount of training data erases that difference. Choosing an architecture is choosing whether exact recall is a requirement.
3. Parallelizability and expressive power are linked — inversely. The linearity that enables parallel training is the same property that confines SSMs to TC⁰. That is a conservation law, not an implementation flaw, and it affects Transformers just as much.
4. When two design families look irreconcilable, look for the common object. State space duality showed that attention and SSMs are two orderings of computation over the same matrix operation. From that followed both better theory and a 2–8× faster kernel.
5. Practice almost never picks the pure solution. Jamba and Nemotron-H replace the majority of attention layers, not all of them. The interesting design variable is not "A or B" but the mixing ratio — and that is determined empirically, not derived.

A concrete prompt to act on this week: take one component of your architecture that carries state over time — a cache, a session, an event-projection snapshot, a monitoring aggregate — and ask exactly one question: is this an archive or a summary?

If it grows with the number of events processed, it is an archive, and its cost is unbounded; then a bound belongs in the documentation, even if it does not bite today. If it is of fixed size, it is a summary, and then there are questions it cannot answer in principle — namely any question requiring exact recall of a single earlier event. Write that limit down explicitly, once. In my experience the most common production failure in such components is not the wrong type, but a summary from which somebody eventually started expecting archive answers.

A question to reflect on: Mamba did not revive the old recurrent idea through more compute, but through a derivation imported from an entirely different field — and the limit it then ran into was not technical but information-theoretic. Which assumption in your own architecture do you treat as a technical constraint that, on closer inspection, might be a conservation law — and which do you treat as a law of nature, although it was only a design decision made by someone who framed the problem differently than you do?

Cross-References in the Vault

• Attention Is All You Need: The Transformer, Self-Attention, and the Architecture of Modern AI – the architecture whose quadratic cost this article views from the other side; state space duality shows both as the same object.
• How Big Is Big Enough? Scaling Laws, Chinchilla, and the Measurement of AI – the economic frame: scaling laws say how much compute you need, architectures like Mamba say how expensive it gets per token.
• The Council of Specialists: Mixture of Experts and How AI Models Grow Without Getting Costlier – the second efficiency axis; Jamba combines both, Mamba layers and MoE, in one model.
• The Needle in a Billion Haystacks: Approximate Nearest Neighbor Search, HNSW, and the Architecture of Vector Databases – the external answer to exactly the recall weakness Jelassi et al. prove: outsource retrieval instead of enlarging the state.
• Answers Without Memory: Bloom Filters, HyperLogLog, and the Art of Calculated Error – the same line of reasoning without neural networks: fixed memory traded against bounded information loss.
• The Error That Learns Backward: Backpropagation, the Chain Rule, and the Algorithm Behind Deep Learning – why vanishing gradients over long sequences were the original RNN problem that HiPPO's bounded gradients address.
• The Ghost in the Machine: How to Read a Neural Network From the Inside – the open follow-on question: interpretability methods were developed largely for attention circuits, not for compressed states.

Sources and Further Reading

• Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher Ré – HiPPO: Recurrent Memory with Optimal Polynomial Projections, arXiv:2008.07669 (NeurIPS 2020): https://arxiv.org/abs/2008.07669
• Albert Gu, Karan Goel, Christopher Ré – Efficiently Modeling Long Sequences with Structured State Spaces (S4), arXiv:2111.00396 (ICLR 2022): https://arxiv.org/abs/2111.00396
• Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler – Long Range Arena: A Benchmark for Efficient Transformers, arXiv:2011.04006 (ICLR 2021): https://arxiv.org/abs/2011.04006
• Albert Gu, Tri Dao – Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv:2312.00752, 1 December 2023 (v2: 31 May 2024): https://arxiv.org/abs/2312.00752
• Tri Dao, Albert Gu – Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, arXiv:2405.21060, ICML 2024: https://arxiv.org/abs/2405.21060
• Samy Jelassi, David Brandfonbrener, Sham M. Kakade, Eran Malach – Repeat After Me: Transformers are Better than State Space Models at Copying, arXiv:2402.01032, 2024: https://arxiv.org/abs/2402.01032
• William Merrill, Jackson Petty, Ashish Sabharwal – The Illusion of State in State-Space Models, arXiv:2404.08819, ICML 2024: https://arxiv.org/abs/2404.08819
• Opher Lieber, Barak Lenz, Hofit Bata et al. (AI21 Labs) – Jamba: A Hybrid Transformer-Mamba Language Model, arXiv:2403.19887, 2024: https://arxiv.org/abs/2403.19887
• NVIDIA – Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models, arXiv:2504.03624, 2025: https://arxiv.org/abs/2504.03624
• Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, Albert Gu – Mamba-3: Improved Sequence Modeling using State Space Principles, arXiv:2603.15569, 16 March 2026: https://arxiv.org/abs/2603.15569
• Reference implementation and kernels: https://github.com/state-spaces/mamba

Created as part of the daily learning workflow. Area of interest: AI. Estimated reading time: ~30 minutes.

← All articles