What are Diffusion Models?

Lilian Weng's blog post "What are Diffusion Models?" provides an overview of diffusion-based generative models. The post recounts that several such models have been proposed with similar underlying ideas, including diffusion probabilistic models (Sohl-Dickstein et al., 2015), noise-conditioned score network (NCSN; Yang & Ermon, 2019), and denoising diffusion probabilistic models (DDPM; Ho et al. 2020).

Diffusion models are described as inspired by non-equilibrium thermodynamics. They define a Markov chain of diffusion steps that slowly add random noise to data and then learn to reverse the diffusion process to construct desired data samples from the noise. Unlike VAE or flow models, the post states, diffusion models are learned with a fixed procedure and the latent variable has high dimensionality, the same as the original data.

The post covers the forward diffusion process, in which Gaussian noise is added over T steps controlled by a variance schedule, and the reverse diffusion process, which learns a model to approximate conditional probabilities. It discusses connections to stochastic gradient Langevin dynamics and noise-conditioned score networks, parameterization choices for training loss, variance schedules and reverse process variance, conditioned generation through classifier guidance and classifier-free guidance, and techniques to speed up sampling.

The post carries update notes: a 2021-09-19 note recommending a blog post on score-based generative modeling by Yang Song; a 2022-08-27 note adding classifier-free guidance, GLIDE, unCLIP and Imagen; a 2022-08-31 note adding latent diffusion model; and a 2024-04-13 note adding progressive distillation, consistency models, and the Model Architecture section.