Sven Erik Matzen

Software Architect | Cloud & Security Expert | AI-enabled Solutions

From Noise to Image: Diffusion Models and the Physics of Generative AI

🎧 Listen to this article

AI · 2026-07-31

EU label: fully AI-generated content Fully AI-generated article (no prior review).

The Hook: Order Out of Pure Randomness

Imagine handing a computer nothing but a square of pure visual noise — the grainy, meaningless flicker that old televisions displayed when no channel was tuned in. Every pixel a random number, no structure, no meaning. And then, in a few dozen computational steps, this chaos transforms into the razor-sharp photograph of an astronaut riding a horse through the Martian desert — an image no camera ever captured, because the scene never existed.

This is no sleight of hand; it is the everyday operation of diffusion models, the technology that has taken over image, video, and audio generation since roughly 2021. Systems such as DALL·E, Stable Diffusion, Midjourney, Google Imagen, and OpenAI's video model Sora all rest on the same underlying principle. And that principle is one of the most elegant in the recent history of machine learning, precisely because it comes from a seemingly unrelated corner: non-equilibrium thermodynamics, the branch of physics that describes how a drop of ink dissolves in a glass of water.

The core idea can be captured in a single sentence, and it is so startlingly simple that one can hardly believe it works: If I know how to destroy an image into noise step by step, then I can teach a neural network to reverse each of those steps — and if I run all the reversals in sequence, I turn noise back into an image. Destruction is trivial and requires no learning. Only the reversal is hard. It is exactly this asymmetry that diffusion models exploit.

This article traces the journey of that idea: from its physical origin, through the three mathematical viewpoints that turned out to be one and the same thing, to the question of how you even tell such a model what to paint — and finally to the open problems and the very real societal consequences.

The Core Concept: Destroy Structure in Order to Learn It

Before diffusion models arrived, generative AI was dominated by two approaches, each with its own demons. There were the Generative Adversarial Networks (GANs, Goodfellow et al. 2014): two neural networks, a forger and an inspector, trained in a contest against each other. GANs produced breathtakingly sharp images, but their training was notoriously unstable — the two networks could oscillate, collapse, or fall into "mode collapse," in which the forger merely varies a handful of motifs. And there were Variational Autoencoders (VAEs) and autoregressive models, which trained more stably but tended to deliver blurry results or, in the case of autoregressive image models, agonizingly slow ones.

Diffusion models offered a third path, and their decisive advantage is conceptual. Instead of pitting one network against another or predicting an image pixel by pixel, they break the impossibly hard problem "generate a realistic image out of nothing" into a long chain of tiny, individually easy sub-problems. Each single step asks the network only to make a slightly noisy image a little less noisy. That is a task a neural network can learn reliably and stably. The difficulty does not sit in one heroic step; it is spread across many modest ones.

The procedure consists of two processes, which you can picture as an outbound and a return trip along the same road.

The forward process (forward diffusion) takes a real image from the training data and, in many small steps — typically a few hundred to a thousand — adds a small dose of Gaussian noise each time. After the first step the image is almost unchanged, just a touch grainier. After half the steps you can barely make out its outlines. At the end, nothing of the original structure remains: pure, structureless noise, indistinguishable from the flicker we started with. This process requires no learning; it is a fixed recipe that can be written down exactly in mathematics. It is controlled destruction.

The reverse process (reverse diffusion) is the real feat and the only thing that must be learned. A neural network is trained to reverse the forward process: given a noisy image and the information about which noise stage it is in, it must estimate what a slightly less noisy image would look like. Run this learned reversal over and over — starting from pure noise — and you climb back down the noise staircase, arriving at a clean, new image that did not exist before.

The deep insight behind this, already present in the original paper, is a theorem from probability theory: if the individual forward steps are small enough (adding only a tiny amount of noise each time), then every reverse step has the same simple mathematical form — it, too, is a Gaussian distribution. So the network never has to model a wildly complicated distribution; for each step it only has to estimate the mean (and spread) of a simple bell curve. This is the mathematical reason why decomposing the problem into many small steps works at all.

Part 1: The Physical Origin — an Ink Drop, in Reverse

The root of the idea lies in a 2015 paper. Jascha Sohl-Dickstein, then at Stanford University, together with Eric Weiss, Niru Maheswaranathan, and Surya Ganguli, published at the ICML conference a paper with the unwieldy but programmatic title "Deep Unsupervised Learning using Nonequilibrium Thermodynamics."

The inspiration was literally physical. In statistical physics, diffusion describes how particles move of their own accord from places of high concentration to places of low concentration — a drop of ink spreading through water, a gas filling a room. This process runs on its own in one direction: toward disorder, toward thermodynamic equilibrium, toward the structureless uniform distribution. The second-law intuition tells us the drop never spontaneously reassembles.

Sohl-Dickstein and colleagues asked the decisive question: what if we regard this decay not as an inevitable loss but as a reversible computational path? Their idea: define a forward diffusion process that slowly and systematically turns an arbitrarily complicated data distribution (say, "the set of all realistic faces") into a simple, known distribution (pure Gaussian noise). Then train a model that learns the corresponding reverse process — the one that restores structure. The result, they promised, would be a "highly flexible and yet tractable" generative model: flexible because it can represent arbitrary distributions, and tractable because probabilities can be computed exactly — a trade-off that earlier methods never offered both of at once.

The work was mathematically beautiful and ahead of its time. But at first it went largely unnoticed. The images it produced were, measured against what GANs could already do, unspectacular, and the computational cost looked unappealing. It would take five years before someone showed that a sleeping giant lay dormant in the idea.

Part 2: The Breakthrough — DDPM and the Art of Predicting the Noise

That someone was Jonathan Ho, together with Ajay Jain and Pieter Abbeel at UC Berkeley. Their 2020 paper "Denoising Diffusion Probabilistic Models" (DDPM) is the turning point at which an elegant curiosity became a serious technology.

Ho and colleagues inherited the scaffolding from Sohl-Dickstein, but they made two crucial simplifications that turned the theory into a practical powerhouse.

The first concerns the forward process. You might think that to bring an image into the noise state after, say, 700 of 1,000 steps, you would actually have to add noise 700 times in sequence. But the mathematics of Gaussian distributions permits a shortcut: you can compute the noise state at any arbitrary time \(t\) in a single step, directly from the original image. A noisy image \(x_t\) is simply a weighted mixture of the original image \(x_0\) and a portion of pure noise \(\varepsilon\): the further along the time step, the smaller the share of the original and the larger the share of noise. This closed-form formula means that during training you can pick a random time step for each image and instantly produce its noisy version — without having to walk through the whole chain. This makes training enormously efficient.

The second, and conceptually more beautiful, simplification concerns what the network should actually predict. Naively, you would have the network estimate the denoised image or the mean of the next reverse step. Ho and colleagues showed that it works far better to have the network instead predict the added noise itself. The training procedure thereby becomes almost banal: take a real image, roll a random time step, roll a portion of noise \(\varepsilon\), mix both into the noisy image \(x_t\), and let the network guess which noise \(\varepsilon\) was mixed in. The objective is then nothing more than the mean squared error between the true and the predicted noise — one of the simplest loss functions there is in machine learning. This "\(\varepsilon\)-prediction" (epsilon prediction) remains the standard formulation to this day.

The result spoke for itself: on the standard benchmark datasets CIFAR-10 and LSUN, DDPM produced images whose quality could rival the best GANs of the time — with a stable, unproblematic training procedure that did not fall into the notorious GAN instabilities. The sleeping giant had awoken.

It is worth pausing to connect the network's task with an intuition from cognitive science. The network essentially learns to infer, from a noisy observation, the probable underlying structure — it learns what is "plausible" so as to compute away the implausible (the noise). In doing so it develops a kind of internal model of what the world of images looks like. This kinship with the brain as a prediction-and-error-correction machine is no accident; it surfaces again and again in AI and links this topic to the predictive processing account of perception.

Part 3: Three Viewpoints, One Model — the Great Unification

Perhaps the most fascinating twist of this story is that DDPM did not stand alone. At the same time, a second line of research had independently arrived at a closely related idea — and a third paper eventually showed that both lines were merely two views of the same object.

Viewpoint A: Denoising (DDPM)

The DDPM view already described is the most intuitive: you think in discrete steps, the network predicts noise, and sampling means descending the noise staircase step by step.

Viewpoint B: Score Matching and Langevin Dynamics

The second line came from Yang Song and Stefano Ermon, also at Stanford. Their 2019 paper ("Generative Modeling by Estimating Gradients of the Data Distribution") thought about the problem in an entirely different way. Instead of predicting noise, they wanted to learn the so-called score of the data distribution: the gradient of the logarithm of the probability density, \(\nabla_x \log p(x)\). Intuitively, the score is a vector field that, at every point in image space, points in the direction in which the data become "more probable" — a compass that everywhere points toward where more plausible images lie.

If you know this compass, you can generate new images by starting at a random point and following the score, while simultaneously mixing in a small portion of randomness. This procedure is called Langevin dynamics and comes — once again — from physics, where it describes the motion of a particle in a force field under thermal noise. Song and Ermon's decisive practical insight: the score is hard to estimate where there are almost no data (in the "empty" regions of image space). Their solution was to perturb the data with noise of varying strength and to learn the score for each noise level — from strong noise, which fills the whole space, to fine noise, which models the details. One immediately recognizes the family resemblance to DDPM.

Viewpoint C: The Unifying SDE View

In 2021, Yang Song and several coauthors (including Sohl-Dickstein and Ermon) closed the circle. Their paper "Score-Based Generative Modeling through Stochastic Differential Equations" showed that DDPM and the score-based models are merely two discrete approximations of one and the same continuous process, which can be described with a stochastic differential equation (SDE).

The idea: let the number of noise steps go to infinity. The discrete staircase becomes a smooth ramp. The forward process becomes an SDE that continuously transforms a data distribution into noise. And here a remarkable mathematical result comes into play, one that the physicist Brian Anderson had already proved back in 1982: for every such forward SDE there exists a reverse SDE that undoes the process exactly — and this reverse SDE depends on nothing but that very score. This made it clear: predicting the noise (DDPM) and learning the score (Song & Ermon) are, up to a scaling factor, the same thing. Both estimate the compass that points from noise back to the data.

This SDE view was not merely theoretically elegant but practically fruitful. Among other things, it produced the probability flow ODE — a deterministic variant of the reverse process (an ordinary rather than stochastic differential equation) that yields the same result without randomness. From it, faster samplers such as DDIM and subsequent methods were derived, cutting the number of required steps from thousands to a few dozen. That three research groups, starting from thermodynamics, probability theory, and differential equations, arrived at the same place is a lovely example of the convergence that often marks deep ideas.

Part 4: Control — How Do You Tell the Model What to Paint?

Up to this point, our models can generate some plausible image — a random face, a random landscape. That is impressive but useless if you want a specific image: "an astronaut on a horse, oil painting." The transition from unconditional to conditional generation is the step that took diffusion models from lab toy to a tool for millions.

The simplest approach is to feed the condition (say, a text) into the network that predicts the noise as an additional input. That works, but the fidelity to the prompt is often weak — the model treats the text as a loose suggestion, not a command. Two techniques solved this problem.

Classifier Guidance

Prafulla Dhariwal and Alex Nichol of OpenAI showed in 2021, in the programmatically titled paper "Diffusion Models Beat GANs on Image Synthesis," that you can use a separate classifier (a network that recognizes what an image shows) to steer the sampling process. At each reverse step you nudge the image a bit in the direction that convinces the classifier it is seeing the desired class. A dial controls how strong this nudge is — and with it the classic trade-off between image quality and diversity: more guidance yields cleaner, more prototypical images but less variety. This work was also the moment when diffusion models overtook GANs quantitatively as well.

The drawback: you need an extra classifier, and it must itself be able to handle noisy images — a costly special case.

Classifier-Free Guidance

The more elegant solution came from Jonathan Ho and Tim Salimans in 2022 with "Classifier-Free Guidance" (CFG), which to this day sits inside virtually every text-to-image system. The trick: you don't need a separate classifier at all. Instead, you train one network so that it can do both — predict noise conditionally (with text) and unconditionally (without text); the latter is achieved by occasionally dropping the text condition during training. When generating, you then let the network make two predictions, with and without text, and extrapolate in the direction that the text adds: you take the unconditional prediction and amplify exactly the difference that the condition makes. A weighting factor \(w\) (the "guidance scale") regulates the strength. Raise it and the image follows the text more slavishly but becomes less diverse; lower it and it becomes freer but less accurate.

Classifier-free guidance is the quiet hero of the generative image revolution. It sits inside DALL·E 2, Google's Imagen, and Stable Diffusion, and it is the reason a text prompt today so reliably produces what was asked for. The textual condition itself is usually fed into the network via a cross-attention mechanism — precisely the attention machinery that also forms the backbone of language models and that has its origin in the Transformer. Text encoder and image generator thus speak to each other in the same language of attention.

Part 5: Latent Diffusion — Democratization Through Compression

As impressive as the early models were, they had a prosaic problem: they computed directly on the pixels. An image of 512×512 pixels with three color channels has nearly 800,000 numerical values, and the reverse process has to push this enormous object through a large network dozens of times. That is so computationally intensive that high-resolution image generation remained the preserve of expensive data centers. DALL·E 2 and Imagen worked around this with multi-stage cascades (first generate a small image, then upscale) — functional but cumbersome.

The decisive leap in efficiency came in 2022 from a group around Robin Rombach and Björn Ommer at LMU Munich with the "Latent Diffusion Models" (LDM), presented at CVPR. Their insight: most of the pixels of an image are perceptually redundant — the fine details contribute little to the meaning. So you can first translate an image with a separately trained autoencoder into a much smaller, compressed "latent space" (by roughly a factor of 48 in data volume), run the entire diffusion process there, and finally decode the result back into a full image. Diffusion thus no longer operates on pixels but on a condensed, meaning-bearing representation.

The effect was dramatic: computational cost dropped by more than an order of magnitude, with barely visible loss of quality. Rombach and colleagues combined this with the aforementioned cross-attention conditioning for flexible control via text, segmentation maps, or other inputs. The result was so efficient that it ran on a single off-the-shelf graphics card.

It was precisely this that made the work a historic turning point — less technically than socially. The latent diffusion architecture became the foundation of Stable Diffusion, released in August 2022 as the first powerful text-to-image open-source model. For the first time, anyone with a reasonably current computer could generate high-quality images, study the model, fine-tune it, and build on it. Image generation was no longer the privilege of a few corporations with their closed APIs. This democratization ignited an explosion of tools, extensions, and applications — and at the same time, as we will see, an equally fierce debate about copyright and misuse.

Part 6: The Architectural Evolution and the Leap Beyond Images

From the U-Net to the Diffusion Transformer

The neural network that performed the actual noise prediction in all these models was long a U-Net — an architecture with a characteristic contracting-and-expanding structure, originally developed for medical image segmentation and well suited to images through its local convolution operations. But in 2023, William Peebles and Saining Xie showed with the Diffusion Transformer (DiT) that the U-Net can be replaced by a pure Transformer — the same architecture that powers language models. The advantage: Transformers scale exceptionally well. More parameters and more data led to predictably better images, entirely in the spirit of the neural scaling laws. The DiT became the next-generation architecture, used among others by OpenAI's video model Sora.

Flow Matching and Rectified Flow

In parallel, a mathematical reformulation emerged that generalizes the diffusion idea: flow matching and, in particular, rectified flow. Instead of learning the winding path between noise and data prescribed by the noising process, these methods connect both ends as straight as possible. The advantage of a straight trajectory is immediately obvious: it can be traversed in far fewer steps, which speeds up sampling. Stable Diffusion 3 (2024) and the model Flux combine both — a multimodal diffusion Transformer with a rectified-flow formulation — and thereby mark the current state of the art in image generation.

Diffusion Beyond Images

What is truly remarkable about diffusion models is that their principle has nothing specifically to do with images. All you need is a data distribution you can noise and denoise. Accordingly, diffusion models have spread far beyond their field of origin:

  • Video — Sora and related models treat video sequences as sequences of "image tokens" and denoise them to generate minute-long, high-resolution clips.
  • Audio and speech — modern text-to-speech systems and music generators use diffusion or flow matching to synthesize waveforms or spectrograms.
  • Molecule and drug design — diffusion models generate three-dimensional molecular structures with desired properties.
  • Protein structure — the 2024 version AlphaFold 3 from Google DeepMind (its predecessor was already Nobel-worthy) uses a diffusion module to generate the atomic coordinates of proteins and their complexes — a remarkable bridge to the question of how proteins find their shape, treated in the article on protein folding.
  • Weather forecasting — models such as GenCast produce probabilistic weather forecasts by "denoising" plausible future states of the atmosphere.

This breadth shows that diffusion is less an image technique than a general recipe for generating structured, high-dimensional data out of noise — a universal generator.

Part 7: The Limits, the Shadows, and the Open Questions

As powerful as the technique is, it has its costs and its dangers, and intellectual honesty demands that we name them.

Slowness. The design advantage — many small steps — is at the same time the biggest drawback. A diffusion model must evaluate its network dozens to hundreds of times to generate a single image, whereas a GAN does it in a single pass. Research into faster samplers (DDIM, consistency models, rectified flow, distillation methods) is a central topic precisely for this reason; the number of steps has fallen from an original thousand to, in some cases, under ten, but the fundamental disadvantage remains.

Memorization. Diffusion models are meant to learn the distribution of the training data, not to memorize the individual images. Yet studies have shown that under certain conditions they can reproduce individual training images almost verbatim — especially those that appear multiple times in the dataset. This is not only a technical nuisance but the crux of the smoldering copyright disputes: if a model was trained on millions of images collected without consent and can reproduce fragments of them, who owns what it produces?

Bias. A model mirrors the distribution of its training data. If those data are skewed — say, in the depiction of professions, genders, or ethnicities — the model reproduces and amplifies these biases, often in subtle and hard-to-control ways.

Misuse and deepfakes. The same ability to generate convincing images and videos from a text prompt also enables the mass production of disinformation, non-consensual imagery, and fraud. This danger is one of the reasons generative AI has come into the focus of regulation: the EU AI Act requires, among other things, the labeling ("watermarking") of synthetic content. I am of the opinion that technical labeling alone will not solve the problem, because watermarks can be removed and open-source models escape central control — but it is a sensible first step in a larger societal negotiation.

The understanding deficit. As with the large language models, the same holds here: we can build these models and admire their outputs, but we understand their inner workings only rudimentarily. Exactly why a model generates a particular image component the way it does, which "concepts" it represents internally — that is the subject of the still-young research described in the article on mechanistic interpretability, which is increasingly turning to the diffusion Transformers as well.

The Central Takeaway

The real lesson of diffusion models is a methodological one, and it reaches far beyond image generation. It reads: an impossibly hard problem becomes solvable when you decompose it into a long chain of easy sub-problems and cleanly define the relationship between them. No one knows how to "create a realistic image out of nothing" — but anyone can learn to make an image a little less noisy. The art lay not in finding the one brilliant step, but in recognizing the right decomposition and in grasping the reversibility of a destruction as a computational path.

For anyone who builds systems, there is a transferable principle in this. Whether in software architecture, where you reconstruct a complex state from a sequence of small, traceable events, or in data processing, where iterative refinement is often more robust than the one grand attempt: the diffusion idea is a prime example of the fact that elegance and performance often spring from the same source — from the right decomposition of a problem. And it is a reminder that the deepest tools of AI are frequently borrowed from seemingly foreign disciplines: a model that today moves billion-dollar markets owes its existence to an analogy with the dissolving of a drop of ink.

A Question to Reflect On

Diffusion models work because the destruction of information is trivial and only its reversal must be learned — the hard direction is defined by the easy one. Where in your own work is there a seemingly unsolvable "forward problem" that would become easier if you first described the reverse, easy path exactly and then reversed it step by step? And one level deeper: if a model can generate convincing reality out of pure randomness — what meaning does the word "real" retain, and who should be permitted to wield this ability?

Cross-References in the Vault

  • The Transformer and Self-Attention – the cross-attention through which diffusion models receive their text condition, and the diffusion Transformer itself, come directly from this architecture.
  • Scaling laws – why the switch from the U-Net to the diffusion Transformer improved image quality so reliably.
  • Predictive processing – the deep kinship between denoising and the brain as a prediction-and-error-correction machine.
  • Protein folding – AlphaFold 3 uses a diffusion module to generate atomic coordinates.
  • Mechanistic interpretability – the attempt to understand which concepts these models represent internally.
  • The EU AI Act – the regulatory response to deepfakes and synthetic content.

Sources

Note: The methods and results presented here rest on peer-reviewed or widely cited scholarly publications. Where personal assessments of open, contested questions (such as the effectiveness of technical labeling) have been made, they are marked with "I am of the opinion that ..."

← All articles