This week, Mohamed Mekkouri from Hugging Face published a blog post to share their success in fine-tuning LLMs to 1.58bit.
Sorry, I cannot get how to calculate the mean at the beginning of the Bit linear layer.
Ah. It's 0.05. I made a mistake. I will fix it soon.
Ah OK! :-)
It's fixed now.
The mean is the sum of all six numbers and divide it by 6.
Thanks for pointing out! I'm glad that you actually tried!
How do you calculate the layernorm? I use z = x - \mu / \sqrt{ \sigma^2 + \epsilon } but got different result
can you do swin transformers by hand?
Sorry, I cannot get how to calculate the mean at the beginning of the Bit linear layer.
Ah. It's 0.05. I made a mistake. I will fix it soon.
Ah OK! :-)
It's fixed now.
The mean is the sum of all six numbers and divide it by 6.
Thanks for pointing out! I'm glad that you actually tried!
How do you calculate the layernorm? I use z = x - \mu / \sqrt{ \sigma^2 + \epsilon } but got different result
can you do swin transformers by hand?