feat: add HardSwish activation function

[surajkarki66]

Reason for this PR

This change adds support for QuantHardSwish, a quantized version of the HardSwish activation function, to Brevitas. Currently, Brevitas lacks support for quantized HardSwish, limiting its applicability to networks that use this activation. I am currently doing my research with brevitas and FINN for quantization-aware training and deploying object detection algorithms on FPGA. Currently, I am using the quantized ReLU activation function for YOLOv8, but I want to try Hardswish, which is better than ReLU in some scenarios.

This addition enables quantization-aware training (QAT) and post-training quantization (PTQ) for networks using HardSwish activation. This addition could potentially enhance the existing quantized activation functions within the Brevitas ecosystem.

Changes Made in this PR

1. New QuantHardSwish Layer (quant_activation.py)

   • Implemented QuantHardSwish class inheriting from QuantNLAL
   • Uses (nn.Hardswish) as the underlying activation implementation
   • Defaults to Uint8ActPerTensorFloat quantizer for unsigned output

2. Graph Quantization Support (quantize.py)

   • Added nn.Hardswish to UNSIGNED_ACT_TUPLE for proper unsigned handling
   • Added mapping in QUANT_ACT_MAP to enable automatic layer conversion during graph quantization

3. FlexML Target Support (flexml.py)

   • Added HardSwish to FLEXML_QUANT_ACT_MAP with Uint8ActPerTensorFixedPoint quantizer
   • Enables export compatibility with FlexML target

4. QuantTensor Functional Support (torch_handler.py)

   • Implemented hardswish_qt_handler for F.hardswish operations on QuantTensor
   • Enables functional-style usage with proper quantization metadata propagation

5. Comprehensive Test Suite (test_act.py)

   • Added 5 test cases covering initialization, forward pass, output characteristics, and training/eval modes

6. Public API Export (init.py)

   • Exported QuantHardSwish in the public API

Design Decisions:

• Unsigned Quantization: HardSwish output range is [0, x] for positive x, making unsigned quantization (Uint8) more appropriate than signed quantization
• passthrough_act=True: This flag preserves quantization metadata through the activation, allowing proper propagation of scale/zero-point in quantized networks
• Uint8ActPerTensorFloat vs FixedPoint: Used floating-point scaling for general use (more flexible), and fixed-point for FlexML (hardware-oriented target)

Implementation Approach:

• Followed the existing pattern established by other quantized activations (QuantReLU, QuantSigmoid, QuantHardTanh)
• Leveraged QuantNLAL (Quantized Non-Linear Activation Layer) base class for consistency
• Used quant_invariant_handler for functional support, ensuring quantization-invariant operations

Testing Summary

Tests Performed:

• All 11 activation tests pass (6 existing + 5 new QuantHardSwish tests)
• Created comprehensive test suite in (test_act.py)
  • test_module_init_default: Verifies default initialization
  • test_module_init_const_scaling: Tests constant scaling configuration
  • test_forward_pass: Validates forward pass with realistic input shapes
  • test_output_non_negative: Confirms output characteristics
  • test_training_eval_modes: Ensures proper behavior in both modes
• Tests run locally and passing

Risk Highlight

• This PR includes code from another work (please detail).
• This PR contains API-breaking changes.
• This PR depends on work in another PR (please provide links/details).
• This PR introduces new dependencies (please detail).
• There are coverage gaps not covered by tests.
• Documentation updates required in subsequent PR.

Checklist

• Code comments added to any hard-to-understand areas, if applicable.
• Changes generate no new warnings.
• Updated any relevant tests, if applicable.
• No conflicts with destination dev branch.
• I reviewed my own code changes.
• Initial CI/CD passing.
• 1+ reviews given, and any review issues addressed and approved.
• Post-review full CI/CD passing.

[Giuseppe5]

Hey, thanks for looking into this.
The reason why we do not provide a default implementation for HardSwish is because it's a non linear function that (unlike ReLU), does not necessarily have a "standard" way to be handled across different different hardware types.

I would still be happy to accept a baseline implementation like this, but let me think about it.

In the meantime, why did you decide to make it UnsignedInt8?
HardSwish seems to have a (small) negative y for ~-3 < x < 0.
With this quantization decision, you are completely cutting off those values.

Another reason why we don't provide a default quantizer is related to this. HardSwish requires a signed quantizer, but most of the negative range is "wasted". To overcome this, the most intuitive solution would be to have asymmetric quantization with a zero-point that allows you to fully use the quantization range.

Considering that asym quantization tends also to be fairly hardware specific, we decided to leave this activation out, since it can be relatively easily implemented out of source.

Also, I don't think setting passthrough_act=True is correct. That works mostly for ReLU, which is piecewise-linear functions that does not change the scale/zero-point of the underlying activation tensor. Hardswish has a (small) non-linear part which can affect the scale computation of the output tensor.

Looking forward to hearing back from you while we think about whether or not to add this feature.

In any case, many thanks for the contribution.

[nickfraser]

FlexML Target Support (flexml.py)

Added HardSwish to FLEXML_QUANT_ACT_MAP with Uint8ActPerTensorFixedPoint quantizer
Enables export compatibility with FlexML target

Also, please remove HardSwish from the FlexML target. FlexML refers to a specific software stack and we don't want to add anything there that isn't supported by FlexML.

Since you are working with FINN, you may be interested to hear that we are planning to have a programmatic quantization flow for FINN in the next release.

[surajkarki66]

HI @Giuseppe5

Changes Made

• Fixed quantization from unsigned to signed
  Changed from Uint8ActPerTensorFloat to Int8ActPerTensorFloat to properly handle HardSwish's small negative outputs (~−3 < x < 0).

• Removed passthrough_act=True
  Changed to passthrough_act=False since HardSwish is non-linear and affects the output scale/zero-point.

• Updated tests
  Tests now validate both positive and negative output behavior.

---

Current State

The implementation provides a baseline using Int8ActPerTensorFloat (symmetric signed quantization).
I fully acknowledge this wastes a significant portion of the negative range, as you mentioned.

---

Regarding Asymmetric Quantization

I understand your concern about hardware-specific requirements.
I can offer two paths forward:

Option A – Baseline only

Keep the current symmetric signed implementation as a starting point that users can customize.
The documentation warns about the inefficiency and suggests alternatives.

Option B – Add asymmetric variant

Add an example/alternative using ShiftedUint8ActPerTensorFloat for asymmetric quantization with a learned zero-point.
Documentation will clarify that this is hardware-dependent and should be validated for specific targets.

---

Next Steps

Which approach would you prefer?
I'm happy to adjust based on what would be most useful for the Brevitas community while being clear about the hardware-specific considerations.

Thanks again for your guidance!

[surajkarki66]

I have one more question. For my research purposes, I also want to implement a quantized version of MaxPool2d and AdaptiveMaxPool2d. Would you be able to add this? Are you interested?

I wanted to do this for my quantized yolov8-obb also.

[nickfraser]

I also want to implement a quantized version of MaxPool2d and AdaptiveMaxPool2d. Would you be able to add this? Are you interested?

Can you tell us more about your QuantMaxPool use-case? MaxPool is quantization-invariant, so it's non-obvious to use why this would be needed.

Our current position is that this is non-necessary / interesting, but we are open to being convinced.

Please note, that Brevitas is designed to be completely compatible with out-of-source extensions, (for example, everything in brevitas_examples is considered to be an out-of-source extension), so whether or not we have such a layer should no affect your ability to add it out-of-source.

If you find something that is difficult to do out-of-source, please raise this to our attention.

Option A – Baseline only

Keep the current symmetric signed implementation as a starting point that users can customize.
The documentation warns about the inefficiency and suggests alternatives.
Option B – Add asymmetric variant

Add an example/alternative using ShiftedUint8ActPerTensorFloat for asymmetric quantization with a learned zero-point.
Documentation will clarify that this is hardware-dependent and should be validated for specific targets.

I prefer option A. @Giuseppe5, do you agree / disagree?

[surajkarki66]

Thank you for the quick response. Yes, MaxPool is quantization-invariant, so it usually does not make sense to quantize it. I also appreciate your reminder about the brevitas_examples.

I have a one question:

• I have implemented a YOLO-like object detection algorithm using QAT, but I am also interested in PTQ with Brevitas. As far as I know, PTQ is currently available only for ImageNet models. Is it possible to apply PTQ to detection algorithms as well, or would that require implementing additional features? I just want to understand the current state so I can start working on it if it’s feasible.

Also, thank you for considering my pull request for review. I hope it will be merged.

[Giuseppe5]

As far as I know, PTQ is currently available only for ImageNet models

This is not correct. We have implemented PTQ pipelines as examples for torchvision, stable diffusion, and LLM models.

For what concerns the PTQ algorithms themselves, there is nothing preventing their application on any model with any task.

The most important part is understanding the concepts behind how each algorithm works (most of them require having a calibration dataset and iterating through it once per model or once per layer, but please take this as general directive and look at our existing pipelines for examples).

The main reason why we do not have a PTQ pipeline for object detection is that it takes time, there is no general benchmark (unlike imagenet for torchvision), and there are multiple libraries we would need to add and study to make sure everything works as intended.

We are not saying it is not important but it is not currently a priority.

[surajkarki66]

Ok, I will start implementing PTQ for object detection algorithms.

[surajkarki66]

@Giuseppe5
Sorry, I accidently created another pull request. Now, I closed that as well. You can review changes here.

[Giuseppe5]

Overall this looks good to me, except for the two small changes above.

Please check our contribution guideline to see how to sign your commits:
https://github.com/Xilinx/brevitas/blob/master/CONTRIBUTING.md

Once all these is handled, we can merge, unless @nickfraser has some other comments

Thanks for the contribution.

[surajkarki66]

Hi @Giuseppe5 and @nickfraser ,

I have don the necessary changes. Thank you for the guidance.

[surajkarki66]

Ok

[surajkarki66]

Hi @Giuseppe5 and @nickfraser,
Sorry for bothering.

I removed unused function.

[Giuseppe5]

Just two remaining changes, mostly related to comments.

If you want, I can also do that myself.

[surajkarki66]

Hi @Giuseppe5 and @nickfraser,
Done

[surajkarki66]

Hi @nickfraser and @Giuseppe5
I hope you are doing well.
Is it possible to merge now ?

[surajkarki66]

Done

[surajkarki66]

One question to @nickfraser
I have checked your mobilenetv2 PTQ for FINN compilation. I found it really helpfull. Now, I am planning to do exact same strategy for YOLO models. Hopefully everything will works.

[surajkarki66]

Hello @nickfraser and @Giuseppe5