BitNet by Hand ✍️

Calculating AI by Hand: 27 of 28

Library › Calculating AI by Hand ✍️

1. Matrix Multiplication by Hand ✍️

2. Multi Layer Perceptron (MLP) by Hand ✍️

3. Backpropagation by Hand ✍️

4. SVM by Hand ✍️

5. Batch Normalization by Hand ✍️

6. Dropout by Hand ✍️

7. Recurrent Neural Network (RNN) by Hand ✍️

8. LSTM by Hand ✍️

9. Deep RNN by Hand ✍️

10. Self Attention by Hand ✍️

11. Transformer by Hand ✍️

12. Autoencoder by Hand ✍️

13. Variational Auto Encoder (VAE) by Hand ✍️

14. Sparse Auto Encoder (SAE) by Hand ✍️

15. Generative Adversarial Network (GAN) by Hand ✍️

16. Sampling a Sentence by Hand ✍️

17. Residual Network by Hand ✍️

18. U-Net by Hand ✍️

19. Discrete Fourier Transform by Hand ✍️

20. Graph Convolutional Network (GCN) by Hand ✍️

21. CLIP by Hand ✍️

22. Vector Database by Hand ✍️

23. Mixture of Experts (MoE) by Hand ✍️

24. Switch Transformer by Hand ✍️

25. Mamba's S6 by Hand ✍️

26. Sora's Diffusion Transformer (DiT) by Hand ✍️

27. BitNet by Hand ✍️

28. Reinforcement Learning with Human Feedback (RLHF) by Hand ✍️

BitNet is an extreme form of quantization: a linear layer's weights are restricted to just three values, [-1, 0, 1]. Because three choices take about 1.58 bits to encode, it is called a 1.58-bit model.

The payoff comes at inference: multiplying by weights that are only -1, 0, or +1 turns into simple addition and subtraction, far cheaper than full multiplication.

How does a BitNet work?

Setup

Step 1 of 17: Given

• Inputs and full-precision weights for a linear layer, shown side by side for Training and Inference.

Linear Layer

Step 2 of 17: Multiplication

• A regular linear layer multiplies the inputs by the full-precision weights.

---

Step 3 of 17: Activation

• Apply the ReLU activation function.