add soft fp8 feat(matmul and grouped matmul ops)

[LinyuanLi0046]

This MR introduces Soft FP8 support by adding software-based FP8 computation paths with weight dequantization (FP8 → BF16).

• Implemented Soft FP8 MatMul operation with integrated weight dequantization.
• Implemented Soft FP8 Grouped MatMul operation with integrated weight dequantization.
• The implementation has been tested with the Qwen3-MoE-30B-A3B-FP8 and DeepSeekR1-0528 model.

[gemini-code-assist]

Summary of Changes

Hello @LinyuanLi0046, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the system's capability by integrating software-driven support for 8-bit floating-point (FP8) operations. It specifically targets matrix multiplication (MatMul) and grouped matrix multiplication, incorporating a crucial step of dequantizing FP8 weights to BFloat16 (BF16) during the computation. This approach aims to balance performance efficiency with numerical accuracy, particularly benefiting large language models and has been verified with the Qwen3-MoE-30B-A3B-FP8 model.

Highlights

• Soft FP8 Support: Introduced software-based FP8 computation paths for matrix multiplication.
• Weight Dequantization: Implemented on-the-fly dequantization of FP8 weights to BF16 for improved precision during computation.
• MatMul and Grouped MatMul: Added dedicated operations for both standard and grouped matrix multiplications using the new FP8 pipeline.
• Qwen3-MoE-30B-A3B-FP8 Model Validation: The new FP8 functionality has been tested and validated with the Qwen3-MoE-30B-A3B-FP8 model.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces Soft FP8 support for MatMul and Grouped MatMul operations, including weight dequantization from FP8 to BF16. The changes involve adding new host-side and kernel-side C++ files, updating CMakeLists.txt to include these new components, and registering the new operations in pytorch_extensions.cpp and sgl_kenel_npu_ops.h. The implementation appears to be comprehensive, covering the necessary data type conversions, memory management, and kernel launching mechanisms for the AscendC platform. The introduction of catlass_op_utils.cpp centralizes utility functions for data type conversions and tensor creation, which is a good practice for maintainability. The kernel implementations for both MatMul and Grouped MatMul leverage custom block MMAD policies and tile operations for efficient FP8 dequantization and computation. Overall, the changes are well-structured and align with the stated objective of adding Soft FP8 support.