FP8 Depthwise Convolutions

[bowCine89]

Hello,

There is a limitation that still holds from the beggining of the fp8 introduction (see the limitations section in the release notes):

There are no optimized FP8 Convolutions for Group Convolutions and Depthwise Convolutions. Therefore, INT8 is still recommended for ConvNets containing these convolution ops.

Is this something that will be addressed at some point ? If not, could we have a hint at the reason why ?
Also would it be technically sound and feasible to leverage the CUTLASS framework to do that and integrate it as a tensorrt plugin (for a custom node) ?

Best,

[asfiyab-nvidia]

Assigning to @nvpohanh

[nvpohanh]

Is this something that will be addressed at some point ? If not, could we have a hint at the reason why ?

Yes, this is on the roadmap. Although in general, depthwise convs cannot utilize TensorCores well, so we recommend not using depthwise convs to better utilize GPU's computational power. The reason is because TensorCores require specific gemm size alignments but depthwise convs do not satisfy those requirements.

Also would it be technically sound and feasible to leverage the CUTLASS framework to do that and integrate it as a tensorrt plugin (for a custom node) ?

It should be possible.

[bowCine89]

Duly noted regarding the efficiency.

Yes, this is on the roadmap.

May I dare to ask for a very rough time window ?

[nvpohanh]

@bowCine89 Which GPU are you targetting for? Thanks

[bowCine89]

We are targeting the Jetson Orin and Jetson Thor.

[nvpohanh]

Orin doesn't support FP8 data type. Only Thor does. I will check internally about Thor FP8 depthwise conv support roadmap

[asfiyab-nvidia]

Closing as the query has been addressed. @bowCine89 please re-open if needed