Variational Auto Encoder (VAE) by Hand ✍️
Calculating AI by Hand: 13 of 28
Library › Calculating AI by Hand ✍️
Variational Auto Encoder (VAE) by Hand ✍️
Reinforcement Learning with Human Feedback (RLHF) by Hand ✍️
A Variational Auto Encoder (VAE) learns the structure (mean and variance) of hidden features and generates new data from the learned structure.
In contrast, GANs only learn to generate new data to fool a discriminator; they may not necessarily know the underlying structure of the data.
The International Conference on Learning Representations (ICLR) this year announced its first ever "Test of Time Award" to recognizes the VAE paper, published 10 years ago.
"𝘈𝘶𝘵𝘰-𝘌𝘯𝘤𝘰𝘥𝘪𝘯𝘨 𝘝𝘢𝘳𝘪𝘢𝘵𝘪𝘰𝘯𝘢𝘭 𝘉𝘢𝘺𝘦𝘴" by Diederik Kingma and Max Welling.
How does a VAE work?
Setup
Step 1 of 11: Given
Three training examples X1, X2, X3
Copy training examples to the bottom
The purpose is to train the network to reconstruct the training examples.
Since each target is a training example itself, we use the Greek word "auto" which means "self." This crucial step is what makes an autoencoder "auto."
Encoder
Step 2 of 11: Layer 1 + ReLU
Multiply inputs with weights and biases
Apply ReLU, crossing out negative values (-1 -> 0)
Step 3 of 11: Mean and Variance
Multiply features with two sets of weights and biases
🟩 The first set predicts the means (𝜇) of latent distributions
🟪 The second set predicts the standard deviation (𝜎) of latent distributions