Anomaly detection live
trained on normal cats only. Same idea spots defects on a factory line.
MAAI901 · Deep Learning for Visual Data
Autoencoders & VAEsteaching a network to imagine cats
Danish · MAAI901 · Tuesday 20 October 2026
scroll, swipe or use the arrow keys ↓
Part 1 · From classification to generation
Discriminative vs generative
analogya pet-show judge can’t paint a cat
CNN?
cat
p(label | image)p(image)
knows a cat when it sees one. Cannot draw one.
Part 2 · Autoencoders
Where autoencoders come from
- 1986Rumelhart, Hinton & Williams. The commonly cited origin: train a network to rebuild its input, so a hidden layer has to learn a code. (The backpropagation paper, and the PDP chapter the same year.)
- 2006Hinton & Salakhutdinov, Science. Deep autoencoders that compress images, after a long quiet period.
- 2008Vincent, Larochelle, Bengio & Manzagol. Denoising autoencoders: corrupt the input, still ask for the clean original.
Part 2 · Autoencoders
Every face is a point
illustrationdescribe a face with two numbers (after Deepia's VAE video)
smile 0.80 · hair 0.30
a network gets no names for its axes: it invents its own, for cats
two numbers per face; drawing the face back from a point is a decoder's job
Part 2 · Autoencoders
Encoder and decoder
same encoder, different ending
Part 2 · Autoencoders
Undercomplete: a cat is 12,288 numbers
analogypacking two weeks of clothes into one carry-on
Like the faces chart, but the network has to find its own two numbers. First, what does it see?
Part 2 · Autoencoders
Overcomplete: room to copy
analogybringing the whole textbook into the exam
features learned: none
it’s just copying
Photocopier Cat
12,288 numbers → 2 numbers
loss 0.0 (summed squared error)
undercomplete illustration: the output is the input
Part 2 · Autoencoders
Making overcomplete work: add a rule
Big codes can copy, so each type adds one rule about z.
A big code is allowed, but a penalty keeps most neurons off. Each one that fires has to earn its place.
Same noisy cat, two models. The denoising AE was trained to predict the clean original. The plain AE was not.
Jiggle the pixels. The bars show how far z moves. The contractive model is punished if a small jiggle moves the code a lot.
plot twist: you've been watching one all along
Every model in this deck is already convolutional. dae_128 is a convolutional denoising autoencoder.
Part 2 · Autoencoders
Reconstruction loss
analogya test where the question sheet is also the answer key
labels used
0
human annotators harmed: 0
Part 2 · Autoencoders
Watch it learn
analogya ball rolling downhill, one small nudge at a time
a replay of ae_2's training: the loss falls, the cats sharpen, and the latent space organises itself
Part 2 · Autoencoders
What autoencoders are good for
analogya security guard who only knows the regulars
12,288 → 64
Live ae_2, ae_16, ae_64 and ae_256. A smaller code is more compression, and a blurrier cat.
Noise is added here. dae_128 was trained to rebuild the clean cat.
ae_2's code is a feature. These are the training cats with the nearest codes. With only 2 numbers they share broad traits like lighting and pose, not the exact cat. Hand z to another model and you can throw the decoder away.
Part 3 · Why autoencoders can't generate
Holes in the latent space
analogyan old map: the cities are drawn, the rest is monsters
nearest training cat
in the data: 0 / in the gaps: 0
the decoder never saw anything from the gaps
Part 4 · Variational autoencoders
Where VAEs come from
- 2013Kingma & Welling, “Auto-Encoding Variational Bayes”. Posted in December 2013, then at ICLR 2014. Encoder outputs a distribution, and the reparameterization trick lets you train it.
- 2014Rezende, Mohamed & Wierstra, “Stochastic Backpropagation and Approximate Inference in Deep Generative Models”, ICML 2014. Independent, and almost the same idea, at the same time.
two papers, one trick: keep the randomness where a gradient can get past it
Part 4 · Variational autoencoders
Probability in one minute
analogya table of grades and ages: column totals, and one row rescaled
joint: how often a cat and a code go together.
, marginals: add up the other variable, .
code → cat: the decoder.
cat → code: the encoder (the ‘posterior’).
Doing this exactly is impossible, so the VAE guesses it with a simple bell curve. That guess is the ‘variational’ part.
drag across the heatmap: the horizontal slice is p(x | z), the vertical slice is p(z | x)
Part 4 · Variational autoencoders
Two worlds
analogya translator between cat photos and a language we invented
all cat photos: messy, unknown
a round blob we chose (the prior)
decoder: blob → cat
encoder: cat → blob
or tap anywhere in p(z) · pick a cat to encode it:
generate = sample the round world, decode into the strange one. The encoder only helps us train.
Part 4 · Variational autoencoders
The VAE’s loss: the ELBO
analogyKL is a leash tying every cloud to the middle of the park
Loss = how badly it rebuilt the cat + how far its cloud drifted from the round blob
ELBO (Evidence Lower Bound) = the official name for this combined score. Loss = −ELBO.
reconstruction: how far the rebuilt cat is from the input
= Kullback–Leibler divergence: how different two distributions are (0 = identical). Here per dimension, 0 only when
Part 4 · Variational autoencoders
Encode a cloud, not a point
analogynot an exact address, a neighbourhood
encoder
decoder
Part 4 · Variational autoencoders
The reparameterization trick
analogyroll the dice first, then stretch it by σ (sigma) and move it by μ (mu)
Can we backprop through a random sample? No: the gradient stops at the dice.
the encoder really outputs log σ² (log of sigma squared), for numerical stability
Part 4 · Variational autoencoders
Watch a VAE learn
KL warm-up lets the cats spread out first; then KL pulls every cloud into one round N(0, 1)
Part 4 · Variational autoencoders
Turning the KL dial
analogya tug of war: rebuild each cat vs pack everyone together
Reconstruction
KL
KL = Kullback–Leibler divergence
Part 4 · Variational autoencoders
Generating cats
either way, the plain AE’s gaps were never trained
grids: bigger AE and VAE trained overnight (10M weights, 100 epochs), samples precomputed. The map: the small live VAE.
Part 4 · Variational autoencoders
Interpolation
A
B
pixels
plain AE
VAE
bigger AE and VAE (10M weights), walks precomputed
pixels stack two ghost cats · the plain AE keeps one cat but glitches between codes · the VAE morphs smoothly, but softer
Part 5 · Limitations, alternatives and wrap-up
Why VAE cats are blurry
the averaging problem
MSE's best guess is the average of the sharp cats that fit, and the KL term also caps how much detail the code can hold
Part 5 · Limitations, alternatives and wrap-up
GAN as an alternative?
analogya forger and a detective
- No reconstruction loss, no averaging: the discriminator punishes blur.
- The catch: unstable training, and mode collapse (the forger repeats one trick).
Part 5 · Limitations, alternatives and wrap-up
VAE vs GAN vs diffusion
| VAE | GAN | Diffusion |
|---|
Gary · illustration
Not a GAN photograph. This repo has no GAN checkpoint, so Gary is our VAE decoding one code nine times with a little jitter. A real GAN sample would be sharper. This only shows mode collapse: one picture, over and over.
Part 5 · Limitations, alternatives and wrap-up
Variants: conditional VAE, β-VAE, VQ-VAE
Conditional VAE
Feed in a label too, then ask for a tabby. A plain VAE can’t take requests: you get a random cat.
β-VAE
A bigger KL weight: a tidier, more disentangled latent space, where each axis means one thing. The cost: blurrier cats.
VQ-VAE illustration
A fixed dictionary of codes instead of a cloud, like paint-by-numbers. A close cousin (dVAE) powered the first DALL-E.
→
Stable Diffusion
Its VAE squeezes a 512×512 image into a latent 48× smaller. Diffusion denoises there, and a VAE decoder paints the final image.
→
Part 5 · Limitations, alternatives and wrap-up
Measure it, then use it well
analogyFID compares two crowds, not two faces
How do we measure it?
Reconstruction: PSNR on the 64 held-out cats, computed live. Higher is better.
Generation has no answer key, so compare clouds: FID runs real and generated images through a pretrained network and measures how far apart the two clouds of features are. Lower is better. This deck shows reconstruction, not FID.
Where it’s used
- Anomaly detection: factory defects, odd medical scans
- Denoising and compression
- Synthetic data to grow a small dataset
- Latent space for Stable Diffusion
Use it responsibly
- It learns the data’s bias. Ours saw only front-facing cats, so a side-on cat looks “abnormal”. With people, that is a fairness problem.
- Faces that never existed: deepfakes, consent for face data, label synthetic images.
a model you can’t measure is a model you can’t trust
Part 5 · Limitations, alternatives and wrap-up
Wrap-up
- 1CNNsJudges: they tell images apart, but can’t paint.
- 2AutoencodersSqueeze a cat into a small code z and rebuild it. No labels needed.
- 3Under vs overcompleteToo much room, and it just copies.
- 4The holesA plain autoencoder’s map has gaps, so random codes give monsters.
- 5VAEsA cloud per cat plus a KL leash: no gaps, so you can sample new cats.
- 6NextGANs and diffusion trade blur for sharpness.
Part 5 · Limitations, alternatives and wrap-up
Questions?
Questions?
Cat faces: fferlito/Cat-faces-dataset on GitHub
AFHQ: Choi et al., StarGAN v2 (CVPR 2020)
Built with PyTorch and ONNX Runtime Web
References
3Blue1Brown, Neural networks series (YouTube)
Deepia, Variational Autoencoders | Generative AI Animated (YouTube)
Kingma & Welling, Auto-Encoding Variational Bayes (2013)
autoencoders.danishali.dev · github.com/Danish12196/vae-presentation