Latent space
In short
A latent space is a smaller set of numbers a model uses to stand in for something it does not observe directly, such as the hidden code behind a picture, so it can work on that shorter code.
- Topic
- How AI models work
- Level
- Some background helps
- First described
- Diederik Kingma and Max Welling, December 2013, in "Auto-Encoding Variational Bayes", treat each observation as drawn from an unobserved continuous code z and compare models by the dimensionality of that latent space1.
In Kingma and Welling's 2013 paper on auto-encoding variational Bayes, the data you see are the result of a hidden step1. A continuous variable z is drawn from a prior, and the observation x is then drawn from a distribution that depends on z1. The true z is unknown. They treat it as a latent representation, or code: the encoder's job is to guess a distribution over codes that could have produced x, and the decoder's job is to turn a code back into a distribution over x1.
They use the phrase latent space for the set of those codes. One figure compares the same method at different dimensionalities of latent space, and a later figure shows 2-D, 5-D, 10-D and 20-D versions1. In the worked example the prior is a centred Gaussian, equal in every direction, with no extra parameters of its own1. A very low-dimensional latent space can also be used as a map: their encoder lays high-dimensional images, such as handwritten digits, onto a 2-D latent space1.
A latent space is not the pixels or the words themselves. It is the space the hidden variable lives in1. It is also not one embedding of a single word. An embedding is a vector the model stores for an item. A latent space is the whole set of hidden codes a generator draws from and decodes1.
The same pattern is why some image models do their slow work away from pixels. Rombach, Blattmann, Lorenz, Esser and colleagues note that diffusion models usually denoise directly in pixel space, which makes training and inference expensive2. They first train an autoencoder whose representation is lower-dimensional but, they write, perceptually equivalent to the image2. Diffusion then runs in that latent space, and the decoder turns the result back into pixels2. They name the class latent diffusion models, and report that this makes training cheaper and speeds up inference with almost no reduction in synthesis quality2.
That claim is about the models in their paper, whose code they released as latent diffusion2. It is not a finding about every image app. What carries across is the three-step shape: compress into a latent code, do the generative work there, decode.
Example
A latent diffusion model does not spend the whole generation editing pixels. It encodes the picture into the autoencoder's smaller latent code, runs diffusion there, and decodes back to an image2.
Why it matters
It is how a generator can work on a short code instead of every pixel. That split is what the 2021 latent diffusion models used to make high-resolution synthesis cheaper2.
Questions people ask
What is a latent space in plain terms?
It is the set of hidden codes a model uses in place of the raw data. Kingma and Welling draw a code z from a prior, then decode it into the thing you actually see1. In their figures that space can be as small as two numbers1.
Is a latent space the same as an embedding?
No. An embedding is a vector attached to one item, such as a word. A latent space is the whole set of hidden codes. In a variational autoencoder the encoder outputs a distribution over that space, and the decoder reads a code back out1.
Why do some image models use a latent space?
Diffusion in pixel space is slow. Rombach and colleagues train an autoencoder first, run diffusion in its lower-dimensional latent space, and decode. They report cheaper training and faster inference with almost no loss of synthesis quality2.