[Question] Understanding function(QunatLinear) scaling and weight quantization for C/FPGA implementation

[L-IKEA]

Hi, thanks so much for your positive reply on Bluesky! I truly appreciate your willingness to help.

As you suggested, I’m opening this issue with my questions and code examples about how the functions of Brevitas (especially "brevitas.nn.QuantLinear", for this case) works under the hood, since I'm aiming to reimplement it in C for deployment on an FPGA.

I’ve been working on deploying a PyTorch-based quantized neural network to an FPGA platform. During this process, I have been using the Brevitas library to perform quantization-aware training, particularly leveraging "brevitas.nn.QuantLinear" for fully connected layers.

To implement the trained model in hardware, I am converting all parameters from the ".pth" file into C header files. However, I have encountered a few challenges in understanding the exact internal quantization mechanism of "QuantLinear" function, especially regarding how the quantized weights and activations are computed, and how I can retrieve all necessary parameters (such as scale) to replicate the behavior in C.

For instance, in the following layer:

self.input_to_hidden_f = qnn.QuantLinear(

    model_input_size + 1, 

    num_neurons, 

    bias=False, 

    weight_quant=WeightQuant, 

    weight_bit_width=self.Weight_Quant_Bit, 

    output_quant=ActQuant, 

    return_quant_tensor=True

)

I examined the runtime behavior by printing the values in forward():

    def forward(self, input_01, input_02, input_03, input_04, input_05, input_06, input_07, input_08, input_09, input_10, input_11, input_12, input_13, input_14, input_15, input_16, input_17, input_18, input_19, input_20, input_21, input_22, input_23, input_24, integral_01, integral_02, integral_03, integral_04, integral_05, integral_06, integral_07, integral_08, integral_09, integral_10, integral_11, integral_12, integral_13, integral_14, integral_15, integral_16, integral_17, integral_18, integral_19, integral_20, integral_21, integral_22, integral_23):
              
        # Forward path
        combined_0101 = torch.cat((integral_01, input_01), dim=-1)
        print("1 identity:", combined_0101[0][0], combined_0101[0][0].type)
        
        forward_01_in = self.input_to_hidden_f(combined_0101)
        # qnn.QuantLinear(model_input_size + 1, num_neurons, bias=False, weight_quant=WeightQuant, weight_bit_width=self.Weight_Quant_Bit, output_quant = ActQuant,return_quant_tensor=True)
        print("Weight:", self.input_to_hidden_f.weight)
        print("Weight Type:", self.input_to_hidden_f.weight.dtype)
        print("Quantized Weight:", self.input_to_hidden_f.quant_weight().tensor)
        print("Weight Scale:", f"{model.input_to_hidden_f.quant_weight().scale:.6f}")
        
        forward_01_in_unquant = torch.matmul(combined_0101, self.input_to_hidden_f.weight.T)
        print("1-2 MatMul:", forward_01_in_unquant)
        
        print("2 fc linear:", forward_01_in[0][0][0], forward_01_in[0][0][0].type, forward_01_in[0][0][0].shape)
        
        forward_01_in = self.activation_01(forward_01_in)
        print("3 activation LReLU:", forward_01_in[0][0][0], forward_01_in[0][0][0].type, forward_01_in[0][0][0].shape)
        
        forward_01_in = self.identity_01(forward_01_in)
        print("4 identity:", forward_01_in[0][0][0], forward_01_in[0][0][0].type, forward_01_in[0][0][0].shape)

The quantized weight scale printed as 0.007812, and the quantized weights appeared to be per-tensor scaled to int8 values. However, when I load the .pth file, this scale value does not appear to be directly accessible. Instead, I only find:

input_to_hidden_f.output_quant.fused_activation_quant_proxy.tensor_quant.scaling_impl.value:

6.531342506408691

This value seems unrelated to the weight scale. Therefore, I would appreciate it if you could help me understand:

Where is the quantized weight scale (e.g., 0.007812) stored or how is it derived?
Is there a way to retrieve it directly from the saved .pth model?

How exactly does QuantLinear compute its output at runtime?
I understand it is something like:
output = quantize(quant_input @ quant_weight.T)
But I would like to know the precise order of operations, and whether input and weights are quantized before or after matrix multiplication, and when scaling is applied.

What are the best practices for replicating this quantization behavior exactly in C?
If there are recommended approaches or documentation (especially regarding fixed-point scaling and integer arithmetic used internally), I would be very grateful.

Understanding these details is crucial for me to ensure bit-accurate hardware implementation, especially in the context of low-bit quantization (e.g., 8-bit).

Thank you very much for your time and for your work on Brevitas — it has been an incredibly helpful library in bridging PyTorch and FPGA design.

If any part of my question was unclear due to my limited experience with this library or the field in general, I sincerely apologize. I would greatly appreciate any clarification or guidance you could provide.

Thank you again for your time and support.

[Giuseppe5]

The concept of scale in Brevitas is a bit overloaded.

The scale that gets printed with model.input_to_hidden_f.quant_weight().scale is the combination of two elements, one is the range of values to cover during quantization, that can be computed through statistics (e.g., max(abs(weights))/2) or a learned parameter, or a constant, etc. Let's call this pre-scale; the second is the bit-width, and in the case of signed integer quantization with narrow-range quantization, the value is given by 2^(bit_width-1)-1. Let's call this threshold.

These two elements are combined such that scale = pre-scale / threshold.

In the state dict we only store the pre-scale.

With respect to the second question, Brevitas' computation can be simplified as:
weights = de-quantize(quantize(weights)
input = de-quantize(quantize(input)
output = torch.nn.functional.linear(input, weights)

The idea is that we apply fake-quantization, with the matmul happening in higher precision. The advantage is that we can simulate a lot of different data-types without relying on HW support.
The disadvantage is that usually it's difficult to achieve bit-precise execution. Usually the discrepancies are small enough that they can be safely ignored but it's important to consider that there could be present, mostly because of floating point arithmetic and approximations.

Let me know if something is not clear

[L-IKEA]

Hi again, and thank you so much for your detailed explanation
It helped me understand the concept of scale in Brevitas much better.

That said, I’d like to follow up with a more specific question regarding how the weight scale (e.g., 0.007812) is actually determined.

From debugging, I printed the quantized weights and their corresponding scale using the following code:

forward_01_in = self.input_to_hidden_f(combined_0101) print("Weight:", self.input_to_hidden_f.weight) print("Quantized Weight:", self.input_to_hidden_f.quant_weight().tensor) print("Weight Scale:", f"{model.input_to_hidden_f.quant_weight().scale:.6f}")

And the output looked like this:
Weight: tensor([[-0.3878, -0.5117], ...]) Quantized Weight: tensor([[-0.3906, -0.5078], ...]) Weight Scale: 0.007812

And from your advice, I got that

quantized_weight = round(weight / scale) * scale

This matches correctly with the printed quantized weights.

Now, I’m trying to understand where exactly the value 0.007812 comes from.

As you mentioned earlier, the final scale is computed as:

scale = pre_scale / threshold

Where threshold = 2^(bit_width - 1) - 1 (i.e., 127 for int8).

I couldn't find the expected pre_scale value,
the only value that contains the name 'scale' was as the below:
static const float input_to_hidden_f__output_quant__fused_activation_quant_proxy__tensor_quant__scaling_impl__value = {6.5313425};

But this was not the expected value, am I right?

So I tried computing pre_scale using:

pre_scale = max(abs(weight)) / 2 = 0.785674 / 2 = 0.392837 scale = pre_scale / 127 = 0.003093

But that doesn’t match the actual scale 0.007812, so it seems the scale was not computed from simple statistics like max(abs(weight))/2.

So here's the thing: How exactly was the value 0.007812 computed in this case?
I assume it must have come from a learned parameter or a constant, but I can’t find it in the .pth file directly.
Is there a reliable way to retrieve the final scale from the saved model?

My goal is to extract these values to replicate the exact quantization behavior in C for FPGA deployment, so understanding how this scale is defined and stored is critical.

Thank you again for your support and guidance!

[L-IKEA]

model_params.json

The concept of scale in Brevitas is a bit overloaded.

The scale that gets printed with model.input_to_hidden_f.quant_weight().scale is the combination of two elements, one is the range of values to cover during quantization, that can be computed through statistics (e.g., max(abs(weights))/2) or a learned parameter, or a constant, etc. Let's call this pre-scale; the second is the bit-width, and in the case of signed integer quantization with narrow-range quantization, the value is given by 2^(bit_width-1)-1. Let's call this threshold.

These two elements are combined such that scale = pre-scale / threshold.

In the state dict we only store the pre-scale.

With respect to the second question, Brevitas' computation can be simplified as: weights = de-quantize(quantize(weights) input = de-quantize(quantize(input) output = torch.nn.functional.linear(input, weights)

The idea is that we apply fake-quantization, with the matmul happening in higher precision. The advantage is that we can simulate a lot of different data-types without relying on HW support. The disadvantage is that usually it's difficult to achieve bit-precise execution. Usually the discrepancies are small enough that they can be safely ignored but it's important to consider that there could be present, mostly because of floating point arithmetic and approximations.

Let me know if something is not clear

+) There was an mistake.

From this .pth file (I made it to .json file)

pre-scale = 6.53134250640869
threshold = 127 (because bit-width is 8bit)

so real scale should be about 0.0514.... .

This still doesn't match with the valeu above: 0.007812... .

[Giuseppe5]

Would it be possible to make a simple python script to reproduce this? So it's easier to understand where your numbers are coming from

[L-IKEA]

Of course! For reference, I’ve attached the full version of the original code that was used. ((I used this code for testing so you'll be able to take a look at the outputs for the first layer.)) This should make it easier to follow the calculations and understand where each number comes from.

Thanks again for your interest!

NEQ_b_test.zip

[Giuseppe5]

Sorry, this code is a bit too dense for me to go through at the moment.

Maybe this reply could help with your question about scale and thresholds:
#1303 (comment)

If it does not, I will try to make some time to run your code but I am not sure when it could be.

Thanks for your understanding.

[nickfraser]

Closed for inactivity. Please reopen if this is still an issue.