u/choHZ
-
785points
-
183points
choHZ
choHZ
Yeah, I agree W8A16 is pretty great — as obviously 8 < 11 — and with an efficient quantization method you’d run faster than BF16, which is already faster than DF11.
Though a) it still fails in some cases; and b) even though it’s on par with most accuracy-like benchmarks, it sometimes changes model behavior quite drastically (like right questions get wrong answers and vice versa), which can be a bit unexpected for users. So DF11 still has its place.
A true killer for DF11, imho, is if FP8 training becomes the standard; which will likely happen in the next 4 year ish according to this plot: https://www.reddit.com/r/MachineLearning/comments/1kzbphg/d_chart_shows_that_fp8_for_training_becoming_more/
choHZ
We are currently updating the code and will push more models once they are finalized (so that we don't have to reupload on HF everytime we optimize the code). LLMs with mainstream architecture are easy cases, VLMs will require some extra consideration on what components to keep in DF11 and so, but should also be doable.
choHZ
Usually I'd stop at the main post and just chat with people in the replies, but I think this one deserves a slightly extended tl;dr — especially for this sub. The core technique may be unfamiliar to many ML folks (myself included at first), but it's definitely interesting to those of us from a CS-related background, as it is leveraging a lot of classic tech.
Background: Data Formats
Formats like FP16, BF16, FP32 all break down into three parts: sign, exponent (range), and mantissa (precision). Here's a quick look:
- FP16: 1-bit sign, 5-bit exponent, 10-bit mantissa
- FP32: 1-bit sign, 8-bit exponent, 23-bit mantissa
- BF16: 1-bit sign, 8-bit exponent, 7-bit mantissa
So BF16 matches FP32’s range (8-bit exponent) but sacrifices precision (7-bit vs 10-bit in FP16). This makes BF16 ideal for model training, as conventional wisdoms suggest NN doesn’t care too much about slightly noisy weights during training, yet value overflow/underflow is critical as it can easily cause issues like gradient vanishing.
Our first key observation is after training, the landed exponent values are super sparse. That is, most of the 8-bit combinations just never show up at all. So we chop up the model (in our case, by blocks), and compress each one losslessly.
Background: Huffman Coding on GPUs
Huffman coding is simple: assign shorter bit patterns to more frequent values. As an exaggeration, if `1001` and `0010` show up often, just we can use `10` and `01` for them and save 2 bits each.
But decoding Huffman codes is sequential by nature — you have to walk a binary tree node by node as showned here, and there is no way to skip and parallelize as the latter bit depedends on the former ones.
So, how to speed it up? Just like anyone who mastered LeetCode Grind 75, the go-to is to HashMap it. Like in this case, if we just have `{10: 121}`, we don’t need to walk this tree and everything is in sweet O(1).
And this is more-or-less what we did. We concatenate exponent bits for a block into a long "string," Huffman-encode that into 32 bits, then chop it into 4x 8-bit tables for lookup because a 32 bits one is too large to form a single LUT. But that raises many tricky edge cases:
- What if the concated exponent string doesn’t compress into 32 bits at first?
- What if a compressed exponent straddles two 8-bit boundaries? An error here not only breaks the lossless guarantee, it is catastrophic as a flip in exponent can result in a vast numerical difference.
- How do we parallelize the whole thing and make it GPU-friendly?
We leave the juicy bits to the paper.
This auto delete is wrong. Not only there is no link in this post, I also provided extensive commentary about the research.