Loading...
As we saw in the notebook, the last piece we are missing is how to determine the optimal s and z. To obtain the scale and the zero point, we need to look at the extreme values r min should mapped q, min and r max should map to q max, and we get the following two equation. Since we have two unknowns s and z, we can solve this equation. If we subtract the first equation from the second one, we can get the scale. So this equation minus this one will give us the scale. And for the zero point. Since we always determine s, we just need for example, to use the first equation and replace s by the value we got before to get the zero point. And at the end we end up with this specific formula. We also need to round the value and to cast it to the correct d-type since we saw that z has the same d-type as the quantized value. If you want to have a look at the details of how we derived the scale and the zero point, I invite you to pause the video and take a screenshot at the following slides. So this one is for the scales derivation, and this one is for So this one is for the scales derivation, and this one is for the zero point derivation. As you saw previously, we make z as the same the d-type as the quantized tensor. For example, as an integer and this is not the same d-type as the scale. The goal behind this choice is to represent zero in the original range as an integer in the quantized range. So thanks to that, when you quantize the value zero, it will take the value z in the quantized range and what is great is that if you'd dequantize the value z, it will become zero again. Now let's have a quick look at how we calculate the scale and the zero point on this example that you saw in the previous slides. So first, we need to get the maximum and minimum range of the original tensor. So we have -184 and 728.6. So this is the maximum value and this is the minimum value. And for the range of the quantized value since we are quantize it in torch.int8. In it the minimum value is -128 and the maximum value is 127. So if you take the formula we learned before, you get that the scale is equal to 3.58 and the zero point is equal to -77. The last case we need to figure out is what happens when the zero point is out of range. For example, since we need to cast z to the quantized datatype, such as int8, what should we do when z is out of range? So if z_min less than q_min, we set z equal to q_min, and if z is superior to q_max, we set that z to be equal to q_max. So this way we don't have overflow and underflow. Now we have everything to code how to get the scale and the zero point. Let's do that. And don't worry about the slide. Will code it directly in the notebook. Now, let's get the scale and the zero point. Let's first start with the scale. As you saw in the formula we need r_max , r_min. Q_max and q_min. We already saw how to get the q_max in the q_min. So, I'll just copy-paste the code. So q_mean will be equal to the minimum value of the torch.int8 information. And the same for q_max where we have max here. And as you saw in the example q_min should be equal to minus 128 and q_max should be equal to 127 Let's have a look. And we indeed have the same results. Now we need to get r_min and r_max, to get the minimum value of the tensor. We can just use the min methods. And we also need to call item to get the value and not the tensor. As you can see here we have the tensor. But we need to call also item to only get the value. We do the same thing for r_max, but this time we can get the maximum value by calling max. Now we have everything to get the scale. As we said earlier, the scale is equal to (r_max-r_min) /(q_max-q_min). And if you remember the example we just saw before, we have the right scale around 3.58. As you can see here. Now let's get the zero point. To get the zero point. We just use the formula. So zero_point=q_min- (r_min/scale) And let's have a look at the zero point. We have -76.5 around. So we need to run and cast it to int. And we get that the zero point is equal to -77. As we saw before, if the zero point was inferior to q_min we will set it to q_min. And if the zero point was superior q_max we will set it to q_max. Now let's define the general function to get the scale and the zero point. We'll call it "get q scale and zero point." This function takes two arguments: the tensor and the type. And we'll set it to torch.int8 by default. As we saw before, we need to define the q_min and the q_max. Then we need to define the r_min and the max of the tensor. We then define the scale and the zero point. For the zero point. As we saw in the slide. There are three cases. Indicates the zero point is less than q_mean. We set the zero point to be equal to q_min. Is the zero point is superior to q_max. We set it to q_max. And the last case is we just run it and cast it to an integer. And we just return the scale and the zero point. Now let's test this general function with the test tensor we define earlier. You can see that indeed we get the same scale and the same zero point as the one we saw in the lecture and before. Now using these new scales and new zero point let's quantize r tensor. So we will call the linear q with scale and zero point function by passing the new scale and the new zero point. So the quantized tensor is equals to this function where we pass this time the test tensor. But with the new scale and the new zero point. And as we did earlier. Also, we also need to dequantize r tensor to compare with the original tensor. So we call the linear the dequantization function where we pass the quantized tensor and the new scale and the new zero points. To have a summary of what we just did. Let's call the plot quantization error function with the test tensor, the quantized tensor and the dequantized tensor. And as you can see this time, the original tensor and the dequantized tensor are very similar, and the quantization error tensor looks also way much better. Now let's also have a look at the quantization error to see if it has decreased a lot or not. So if you remember well, to get the quantization error, you subtract the dequantized tensor and the test tensor. We take the square and you do the min. And this time, as you can see, compared with the quantization error of around 170. Now we only have a quantization error of around one. Now let's put everything inside a linear quantization function that will only take a tensor and will return to you the quantized tensor, the scale and the zero point. So we defined the linear quantization function. It takes as input a tensor and a d-type that we will set to torch.int8 by default. In this function, we will use the two function that we coded before. So the get q scales and zero point to get the scales and the zero point. So we just call that function and we just pass the tensor. And also the d-type. Then after getting the scale and the zero point we can perform the quantization of the tensor. So we will get the quantized tensor. If we use the linear q scale and zero point function we coded before where we passed the tensor and the scale, the zero point, and the d-type. We just return the quantized sensor, the scale and the zero point. Now let's play with this linear quantizer on a random matrix. So we'll define a tensor. Which will take random values. And it will be of size 4x4. As you can see, we do have a random tensor of size 4x4. And we can just call the linear quantization function on our random tensor to get the quantized tensor, the scale and the zero point. Let's have a look at the quantized tensor. As you can see, the tensor was quantized and we also have the following values for the scales and the zero points to have the summary of the quantization process, let's also dequantize the tensor by calling the linear dequantization and by passing the quantized tensor, the scale and the zero point. And we can use the plot quantization error function to have the summary of the quantization process. We passed the random tensor, the quantized tensor, and the dequantized tensor. Oh, and as you can see, the original tensor here is pretty much the same as the dequantized tensor, and the quantization errror tensor is very small. And we can also print the quantization error. Which is also pretty low. And now I invite you to pause the video and try to play with this quantization with your own inputs and see how it performs. In the next lesson, we will dive deeper into linear quantization by learning its symmetric variants. And we will also look into quantization granularity, such as per tensor, per channel and group quantization. Finally, we will also look at how to perform inference with quantized models.
Quantization in Depth
- Introduction
- Overview
- Quantize and De-quantize a Tensor
- Get the Scale and Zero Point
- Symmetric vs Asymmetric Mode
- Finer Granularity for more Precision
- Per Channel Quantization
- Per Group Quantization
- Quantizing Weights & Activations for Inference
- Custom Build an 8-Bit Quantizer
- Replace PyTorch layers with Quantized Layers
- Quantize any Open Source PyTorch Model
- Load your Quantized Weights from HuggingFace Hub
- Weights Packing
- Packing 2-bit Weights
- Unpacking 2-Bit Weights
- Beyond Linear Quantization
- Conclusion