[Bug]: Can't run GPT-OSS models on DGX Spark

[kalelegs]

System Info

• System: Nvidia DGX Spark
• CPU : ARM
• GPU architecture Blackwell GB10
• unified Memory -128Gb

Who can help?

@Tracin

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Follow the quick start recipe on TensorRT-LLM Official Docs

docker run --rm -it \
    --ipc=host \
    --gpus all \
    -p 8001:8000 \
    -v ~/.cache/model:/root/.cache:rw \
    --name tensorrt_llm \
    nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc0 \
    /bin/bash

Inside the container run -

# using  cutlass backend
EXTRA_LLM_API_FILE=/tmp/cutlass-config.yml

cat << EOF > ${EXTRA_LLM_API_FILE}
enable_attention_dp: true
cuda_graph_config:
  enable_padding: true
  max_batch_size: 720
moe_config:
    backend: CUTLASS
stream_interval: 20
num_postprocess_workers: 4
attention_dp_config:
    enable_balance: true
    batching_wait_iters: 50
    timeout_iters: 1
EOF


# start the server
trtllm-serve openai/gpt-oss-20b \
    --host 0.0.0.0 \
    --port 8000 \
    --backend pytorch \
    --max_batch_size 720 \
    --max_num_tokens 16384 \
    --kv_cache_free_gpu_memory_fraction 0.9 \
    --tp_size 1 \
    --ep_size 1 \
    --trust_remote_code \
    --extra_llm_api_options ${EXTRA_LLM_API_FILE}

The container errors our saying

RuntimeError: [TensorRT-LLM][ERROR] Assertion failed: The attention sinks is only supported on SM90.

Expected behavior

The server should start and model should be useable.

actual behavior

The server fails to start

additional notes

Also tried with TRTLLM backend instead of cutlas

cat << EOF > ${EXTRA_LLM_API_FILE}
enable_attention_dp: false
cuda_graph_config:
  enable_padding: true
  max_batch_size: 720
moe_config:
    backend: TRTLLM
stream_interval: 20
num_postprocess_workers: 4
EOF

it also failed with error different NotImplementedError: TRTLLMGenFusedMoE does not support SM120 and above.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[SimengLiu-nv]

cc @farazkh80

[farazkh80]

Hey @kalelegs the v1.2.0 TensorRT-LLM image doesn't include the gpt-oss support for dgx spark.

You could use spark-single-gpu-dev node for testing https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release?version=spark-single-gpu-dev

More details in https://build.nvidia.com/spark/trt-llm/instructions

[aikitoria]

Is there a plan for these sm120 kernels to be made available in the open source version rather than a specific build?

[farazkh80]

They are already merged to main, next RC release should have them.

You could also try building from source.

[aikitoria]

Good to know, thanks!

[aikitoria]

Hmm @farazkh80 what am I missing? I've compiled it from main, but the result is this on following the "low latency" sample:

[10/19/2025-02:51:30] [TRT-LLM] [I] Fallback to regular model init: Traceback (most recent call last):
  File "/code/tensorrt_llm/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 211, in load
    model = AutoModelForCausalLM.from_config(config_copy)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/models/modeling_auto.py", line 48, in from_config
    model = cls(config)
            ^^^^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/models/modeling_utils.py", line 328, in __call__
    obj = type.__call__(cls, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/models/modeling_gpt_oss.py", line 599, in __init__
    Transformer(model_config),
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/models/modeling_utils.py", line 225, in __call__
    obj = type.__call__(cls, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/models/modeling_gpt_oss.py", line 513, in __init__
    TransformerBlock(
  File "/code/tensorrt_llm/tensorrt_llm/_torch/models/modeling_gpt_oss.py", line 346, in __init__
    self.mlp = MLPBlock(config,
               ^^^^^^^^^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/models/modeling_gpt_oss.py", line 189, in __init__
    self.experts = create_moe(**moe_params)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/modules/fused_moe/create_moe.py", line 101, in create_moe
    return moe_cls(
           ^^^^^^^^
  File "/code/tensorrt_llm/tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py", line 90, in __init__
    raise NotImplementedError(
NotImplementedError: TRTLLMGenFusedMoE does not support SM120 and above.

Edit: The "high throughput" sample appears to work!

[farazkh80]

You need to use the cutlass backend for MoE

moe_config:
    backend: CUTLASS

[karljang]

With #7937 merged, all kernels needed by GPTOSS now support SM121 for DGX Spark~
If you see any other issue, please feel free to open a new one~