CLIP by Hand ✍️

Calculating AI by Hand: 21 of 28

Library › Calculating AI by Hand ✍️

1. Matrix Multiplication by Hand ✍️

2. Multi Layer Perceptron (MLP) by Hand ✍️

3. Backpropagation by Hand ✍️

4. SVM by Hand ✍️

5. Batch Normalization by Hand ✍️

6. Dropout by Hand ✍️

7. Recurrent Neural Network (RNN) by Hand ✍️

8. LSTM by Hand ✍️

9. Deep RNN by Hand ✍️

10. Self Attention by Hand ✍️

11. Transformer by Hand ✍️

12. Autoencoder by Hand ✍️

13. Variational Auto Encoder (VAE) by Hand ✍️

14. Sparse Auto Encoder (SAE) by Hand ✍️

15. Generative Adversarial Network (GAN) by Hand ✍️

16. Sampling a Sentence by Hand ✍️

17. Residual Network by Hand ✍️

18. U-Net by Hand ✍️

19. Discrete Fourier Transform by Hand ✍️

20. Graph Convolutional Network (GCN) by Hand ✍️

21. CLIP by Hand ✍️

22. Vector Database by Hand ✍️

23. Mixture of Experts (MoE) by Hand ✍️

24. Switch Transformer by Hand ✍️

25. Mamba's S6 by Hand ✍️

26. Sora's Diffusion Transformer (DiT) by Hand ✍️

27. BitNet by Hand ✍️

28. Reinforcement Learning with Human Feedback (RLHF) by Hand ✍️

The CLIP (Contrastive Language-Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today.

How does CLIP work?

Goal: Learn a shared embedding space for text and image

Setup

Step 1 of 13: Given

• A mini batch of 3 text-image pairs

• OpenAI used 400 million text-image pairs to train its original CLIP model.

Embeddings

Step 2 of 13: Text → Vectors

• Look up word embedding vectors using word2vec.

---

Step 3 of 13: Image → Vectors

• Divide the image into two patches.

• Flatten each patch