Skip to content

CUDA: topk-moe: add optional parameter for gpt-oss #16649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

CUDA: topk-moe: add optional parameter for gpt-oss #16649

wants to merge 2 commits into from
+162 −62

Conversation

am17an
Copy link
Collaborator

@am17an am17an commented Oct 18, 2025 •
edited
Loading

While looking at this kernel I realized that it is relatively easy to add it for gpt-oss, which does the softmax after the top-k.

Performance on a 4090:

Model Test t/s master t/s cuda_gpt_oss_opt Speedup
gpt-oss 20B MXFP4 MoE tg32 170.99 177.68 1.04
gpt-oss 20B MXFP4 MoE tg64 168.75 175.36 1.04
gpt-oss 20B MXFP4 MoE tg128 167.01 173.33 1.04

@am17an am17an requested a review from slaren as a code owner October 18, 2025 11:24
@github-actions github-actions bot added testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Oct 18, 2025
jeffbolznv added a commit to jeffbolznv/llama.cpp that referenced this pull request Oct 18, 2025
@jeffbolznv
vulkan: Update topk_moe fusion to handle gpt's late softmax
0111a34 
Based on ggml-org#16649.
@jeffbolznv jeffbolznv mentioned this pull request Oct 18, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@slaren slaren Awaiting requested review from slaren slaren is a code owner

@JohannesGaessler JohannesGaessler Awaiting requested review from JohannesGaessler

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning Nvidia GPU Issues specific to Nvidia GPUs testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@am17an