Diffusion models

TL;DR

• Diffusion models generate data (images, audio, video) by learning to reverse a noise-adding process: start with pure Gaussian noise, iteratively denoise over 20-50 steps until a clean image emerges.
• The forward process adds noise gradually over T timesteps. The reverse process trains a neural network (usually a U-Net or DiT) to predict and remove the noise at each step.
• Latent diffusion (Stable Diffusion, DALL-E 3) runs denoising in a compressed latent space instead of pixel space, cutting compute by 10-100x while maintaining quality.
• Classifier-free guidance (CFG) controls how closely the output follows the text prompt. Higher CFG means more prompt-faithful but less diverse outputs.
• Diffusion Transformers (DiT) replace U-Net with transformer blocks and now power the best models: Sora, Stable Diffusion 3, and Flux.

---

The problem it solves

Before diffusion models, the two dominant generative architectures were GANs (Generative Adversarial Networks) and VAEs (Variational Autoencoders). Both had fundamental failure modes that limited their practical use.

GANs train two networks against each other: a generator creates fake images, and a discriminator tries to identify them. This adversarial setup is a minimax game that is notoriously difficult to stabilize. The generator can discover a small number of convincing-looking outputs that reliably fool the discriminator, and then stop improving. This is mode collapse: a GAN trained on dog photos might only produce three dog poses that exploit discriminator weaknesses, ignoring the full diversity of the training distribution.

The training dynamics compound the problem. If the discriminator becomes too accurate, gradients flowing back to the generator vanish and training stalls. If neither network settles, they oscillate in a feedback loop with no convergence. Getting both networks to train together requires careful architecture choices and hyperparameter sensitivity that made GAN-based systems fragile in practice.

VAEs had the opposite problem: they were stable to train but produced blurry outputs. Compressing an image through a continuous latent bottleneck and decoding back forces the model to average over possible reconstructions, producing the soft, washed-out look of overcompressed images. No amount of training fully removes this blur because the continuous bottleneck is a structural constraint.

The 2020 DDPM paper (Ho, Jain, and Abbeel at Berkeley) showed that reformulating generation as a denoising regression problem produced image quality competitive with GANs while training with the reliability of a VAE. The field did not look back.

---

What is it?

A diffusion model is a generative model that learns to reverse a gradual noise-adding process: given a data sample corrupted over T timesteps until only random noise remains, train a neural network to undo that corruption one step at a time.

Think of it as restoring an old photograph that was gradually buried under layers of TV static. You receive a photograph covered in 1,000 layers of static and must reconstruct the original. Each restoration step removes a tiny layer of static. You do not need to understand what the whole image is at step 1; you just need to know what "slightly less static" should look like. After enough restoration steps, the original photograph emerges.

At inference time, start with pure random static, apply the restoration model 20-50 times, and a new image forms. The model does not retrieve that image from a database; it constructs it through the iterative denoising process.

The key insight is the asymmetry: the forward process is fixed and mathematical (add noise according to a schedule), while the reverse process is learned (train a network to predict and subtract noise). This asymmetry gives diffusion models GAN-level output quality with VAE-level training stability.

---

How it works

The forward process: adding noise

At each timestep t, a small amount of Gaussian noise is mixed into the previous image: x_t = √(αt) · x_{t-1} + √(1−αt) · ε, where ε ~ N(0, I) is freshly sampled noise. The coefficient αt decreases from near-1 at t=1 to near-0 at t=1000, so each step discards a little more signal and adds a little more noise.

A crucial mathematical property lets you jump to any timestep directly without running all earlier steps: x_t = √(ᾱt) · x₀ + √(1 − ᾱt) · ε, where ᾱt is the cumulative product of all αs for s ≤ t. This makes training efficient: sample a random t, corrupt the original image in one operation, and train the network to reverse that single step.

The noise schedule controls how fast noise accumulates. DDPM used a linear schedule, adding noise at a constant rate over 1,000 steps. The cosine schedule from Nichol and Dhariwal (2021) adds noise more slowly near t=0 and t=T, preserving more structure at low noise levels and producing noticeably better visual quality in practice.

The reverse process: learning to denoise

The training objective is: given a noisy image x_t and timestep t, predict the noise ε that was added. The loss is MSE: L = E[‖ε − ε_θ(x_t, t)‖²]. There is no adversarial component and no mode collapse; this is a standard regression problem with a stable gradient signal.

The standard architecture is a U-Net with downsampling encoder layers, a bottleneck, and upsampling decoder layers with skip connections. Skip connections bridge the encoder and decoder at each resolution level, letting the network recover fine spatial details that were compressed away in the downsampling path. The timestep t is embedded as a sinusoidal vector and injected into every layer via adaptive group normalization, telling the network how noisy the input currently is.