Autoencoder by Hand ✍️

Calculating AI by Hand: 12 of 28

Jan 22, 2024

∙ Paid

Library › Calculating AI by Hand ✍️

The autoencoder model is the basis for training foundational models from a ton of data. We are talking about tens of billions of training examples, like a good portion of the Internet.

With that much data, it is not economically feasible to hire humans to label all of those data to tell a model what its targets are. Thus, people came up with many clever ideas to derive training targets from the training examples themselves [auto]matically.

The most straightforward idea is to just use the training data itself as the targets. This hands-on exercise demonstrates this idea.

Then, people tried hiding some parts of the training data and using those missing parts as the targets. This is called masking, which is how LLMs are trained these days.

Then, people tried pairing up text and images and using each other as targets. This is called "constrative" learning. This is the C in the famous CLIP model from OpenAI, which is the basis of all the multimodal foundational models.

Let's start with the basics, AutoEncoder.

Network Architecture

🟨 Encoder

Linear(4,3)
ReLU
Linear(3,2)
ReLU

🟦 Decoder

Linear(2,3)
ReLU
Linear(3,4)

Setup

Step 1 of 7: Given

Four training examples X1, X2, X3, X4

Step 2 of 7: Auto (copy to targets)

Copy training examples to Targets (Y')
The purpose is to train the network to reconstruct the training examples.
Since each target is a training example itself, we use the Greek word "auto" which means "self." This crucial step is what makes an autoencoder "auto."

Encoder

Step 3 of 7: Layer 1 + ReLU

Multiply inputs with weights and biases
Apply ReLU, crossing out negative values (-1 -> 0)

AI by Hand ✍️