Why NVFP4 Inference (50 PFLOPS) Outperforms Training (35 PFLOPS) on Rubin GPU？

[Yogaht]

According to NVIDIA's official blog Inside the NVIDIA Rubin Platform: Six New Chips, One AI Supercomputer, the 3rd Gen Transformer Engine is equipped with "hardware-accelerated adaptive compression designed to boost NVFP4 performance while preserving accuracy", which enables up to 50 PetaFLOPS of NVFP4 inference capability and 35 PetaFLOPS for training.

1. Could you please tell me the technical mechanism of the "hardware-accelerated adaptive compression" in the 3rd Gen Transformer Engine?
2. What are the key factors that cause the NVFP4 performance gap between training (35 PFLOPS) and inference (50 PFLOPS)?

[ptrendx]

We cannot comment on the details of the unreleased hardware beyond what was disclosed in the official blog.
As for the difference in training and inference throughput, please keep in mind that even when using the same datatype (like nvfp4 in this case), there are differences in the recipes required. For example, inference in nvfp4 may use predetermined tensor-wide scaling factors obtained via calibration. In contrast, during training we need to compute that scaling factor every time a tensor is quantized. This limits the fusions that can be performed during training. Additionally, there are some operations in the nvfp4 training recipe, like Random Hadamard Transformations, which only apply to the backward pass, and therefore are not used in the inference at all. All of this means that, while the GEMM performance is the same, the actual end-to-end performance will differ between workloads.