[Bug]: Empty output issue from invalid channel establishment after parsing, for gpt-oss-120b. Often caused by garbage, repeated output tokens.

[harrisonlimh]

System Info

Using nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc6 on B200x8 machine

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Start TensorRT-LLM official docker container

docker run --privileged -itd --ipc=host \
-v /tmp:/tmp --gpus=all --network=host \
--name=${CONTAIER_NAME} \
--entrypoint="bash" \
nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc6

2. Download weights from HF

HF_MODEL_PATH="openai/gpt-oss-120b"
hf download ${HF_MODEL_PATH} --local-dir=/tmp/${HF_MODEL_PATH}
HF_MODEL_PATH="nvidia/gpt-oss-120b-Eagle3"
hf download ${HF_MODEL_PATH} --local-dir=/tmp/${HF_MODEL_PATH}

3. Create runtime config and start the server

cat > gpt-oss-120b-eagle3-trtllm-bs-128-xgrammar.yaml  << 'EOF'
trust_remote_code: true
kv_cache_config:
  dtype: fp8
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.95
speculative_config:
  allow_advanced_sampling: true
  decoding_type: Eagle
  max_draft_len: 3
  speculative_model_dir: /tmp/nvidia/gpt-oss-120b-Eagle3
cuda_graph_config:
  enable_padding: true
  max_batch_size: 128
max_batch_size: 128
moe_config:
  backend: TRTLLM
enable_attention_dp: false
enable_min_latency: true
guided_decoding_backend: xgrammar
num_postprocess_workers: 256
EOF

config_file_name=gpt-oss-120b-eagle3-trtllm-bs-128-xgrammar.yaml 
port=30000
tp=8
ep=4

TRTLLM_ENABLE_PDL=1
trtllm-serve /tmp/openai/gpt-oss-120b --host 0.0.0.0 --port ${port} --max_batch_size 128  --tp_size ${tp} --ep_size ${ep} --trust_remote_code --extra_llm_api_options ${config_file_name} --max_num_tokens 131072 --max_seq_len 131072

4. Send the same chat completions requests for multiple times and examine output tokens with custom logging here; logging.warning(f"...processed_tokens: {self.parser.tokens}, decoded_messages: {"" if not self.parser.tokens else self.encoding.decode(tokens)}.

Run #1
Server side log for run#1. Output is empty as it failed to establish one of the valid channels (e.g. <|channel|>final<|message|>, 200005,17196,200008)

[01/13/2026-00:44:59] [TRT-LLM] [E] process_token_batch(iter=9) - processed_tokens: [200005, 35644, 200008, 1844, 10648, 316, 5230, 472, 1991], decoded_messages: <|channel|>analysis<|message|>User wants to pick as many

$ curl http://0.0.0.0:30000/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer input_any_token_here" -d @- <<EOF
{
"model": "openai/gpt-oss-120b",
"messages": [
{
"role": "user",
"content": "Pick as many lines as you can from these poem lines:\nAh! but those tears are pearl which thy love sheds,\nWhich, like a jewel hung in ghastly night,\nThou gavest me thine, not to give back again.\nThat I in thy abundance am sufficed\nWhereon the stars in secret influence comment;\nFor thee and for myself no quiet find.\nThat beauty still may live in thine or thee.\nBut since she prick'd thee out for women's pleasure,\nWho with his fear is put besides his part,\nMine eyes have drawn thy shape, and thine for me\nTill then not show my head where thou mayst prove me.\nSo is it not with me as with that Muse\nA dearer birth than this his love had brought,\nWho all in one, one pleasing note do sing:\nFrom sullen earth, sings hymns at heaven's gate;\nWho will believe my verse in time to come,\nMuch liker than your painted counterfeit:\nLet them say more than like of hearsay well;\nYet, do thy worst, old Time: despite thy wrong,\nAnd moan the expense of many a vanish'd sight:\nAlthough our undivided loves are one:\n'Tis not enough that through the cloud thou break,\nBeauty's effect with beauty were bereft,\nThen being ask'd where all thy beauty lies,\nAnd your true rights be term'd a poet's rage\nAnd kept unused, the user so destroys it.\nAs fast as thou shalt wane, so fast thou growest\nGilding the object whereupon it gazeth;\nFor where is she so fair whose unear'd womb\nYet eyes this cunning want to grace their art;\nAnd make the earth devour her own sweet brood;\nSets you most rich in youth before my sight,\nTo eat the world's due, by the grave and thee.\nMaking a famine where abundance lies,\nNor lose possession of that fair thou owest;\nSometime too hot the eye of heaven shines,\nThyself thy foe, to thy sweet self too cruel.\nWith what I most enjoy contented least;\nFor thou art so possess'd with murderous hate\nWhen I perceive that men as plants increase,\nServing with looks his sacred majesty;\nThat I an accessary needs must be\nSince sweets and beauties do themselves forsake\nShall hate be fairer lodged than gentle love?\nWhich, like a jewel hung in ghastly night,\nWhen in eternal lines to time thou growest:\nWhose speechless song, being many, seeming one,\n\nThe region cloud hath mask'd him from me now.\nBut day by night, and night by day, oppress'd?\nAfter a thousand victories once foil'd,\nLook, what is best, that best I wish in thee:\nThose hours, that with gentle work did frame\nThat thou no form of thee hast left behind,\nLook, whom she best endow'd she gave the more;\nO, let my books be then the eloquence\nDo in consent shake hands to torture me;\nStrikes each in each by mutual ordering,\nSeeking that beauteous roof to ruinate\nO'ercharged with burden of mine own love's might.\nA dearer birth than this his love had brought,\nThe age to come would say 'This poet lies:\nHow many a holy and obsequious tear\nO' let me, true in love, but truly write,\nFor thy sweet love remember'd such wealth brings\nSave that my soul's imaginary sight\nSuch civil war is in my love and hate\nUnless thou take that honour from thy name:\nSweets with sweets war not, joy delights in joy.\nAnd dumb presagers of my speaking breast,\nHim in thy course untainted do allow\nStirr'd by a painted beauty to his verse,\nThis wish I have; then ten times happy me!\nMy body is the frame wherein 'tis held,\nThat thou no form of thee hast left behind,\nI may not evermore acknowledge thee,\nFor having traffic with thyself alone,\nAnd your sweet semblance to some other give.\nUnlook'd on diest, unless thou get a son.\nMine eyes have drawn thy shape, and thine for me\nNor shall Death brag thou wander'st in his shade,\nWhich erst from heat did canopy the herd,\nBut day by night, and night by day, oppress'd?\nWhich hides your life and shows not half your parts.\nThe bounteous largess given thee to give?\nWithin thine own bud buriest thy content\nBut that I hope some good conceit of thine"
}
],
"temperature": 0.0,
"max_tokens": 10,
"stream": true
}
EOF
data: {"id":"chatcmpl-8b3d9f6482684889bdf4a763e680c040","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-37dc79f3c47c41479696695370ba4d11","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-6bda144bb04f4e86b4add7aa8f94e1db","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"reasoning":"User"}}]}

data: {"id":"chatcmpl-b4f8715bce5c4d12ac37042735c31c2f","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-1ed48da8902c4c6e899b07043a89a982","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"reasoning":" wants"}}]}

data: {"id":"chatcmpl-016603335dac4d0683728b2832efe365","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-7c89edfbcd5c47369c48ac799205602e","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"reasoning":" to"}}]}

data: {"id":"chatcmpl-35c5cc6f09bd4e79a99b03937c50a7ca","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"reasoning":" pick"}}]}

data: {"id":"chatcmpl-a4ca1f35498b459b9381c796f2df5fd0","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-934a369f2df543fe88a9b0180d34e04b","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"reasoning":" as"}}]}

data: {"id":"chatcmpl-3fb7227fb6dd44a1bc6f709256660a87","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"reasoning":" many"}}]}

data: {"id":"chatcmpl-4c7d53102d524ca8973936a110cebe76","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{},"finish_reason":"length"}]}

data: {"id":"chatcmpl-885e7ac409d148c58bc2909bf6ad8d9d","object":"chat.completion.chunk","created":1768265099,"model":"openai/gpt-oss-120b","choices":[],"usage":{"prompt_tokens":1003,"total_tokens":1013,"completion_tokens":10}}

data: [DONE]

---

Run #2
Server side log. Output is empty as it failed to establish one of the valid channels (e.g. <|channel|>final<|message|>, 200005,17196,200008)

[01/13/2026-00:45:01] [TRT-LLM] [E] process_token_batch(iter=8) - processed_tokens: [279, 279, 279, 19371, 279, 19371, 279, 279], decoded_messages: 





Then

Then

Client observes no output as expected

(e2e_eval_env) harrisonlim_google_com@harrisonlim-dev-b200-uw2-20251202-000238:~/bench/scripts$ curl http://0.0.0.0:30000/v1/chat/completions   -H "Content-Type: application/json"   -H "Authorization: Bearer input_any_token_here"   -d @- <<EOF
{
  "model": "openai/gpt-oss-120b",
  "messages": [
    {
      "role": "user",
      "content": "Pick as many lines as you can from these poem lines:\nAh! but those tears are pearl which thy love sheds,\nWhich, like a jewel hung in ghastly night,\nThou gavest me thine, not to give back again.\nThat I in thy abundance am sufficed\nWhereon the stars in secret influence comment;\nFor thee and for myself no quiet find.\nThat beauty still may live in thine or thee.\nBut since she prick'd thee out for women's pleasure,\nWho with his fear is put besides his part,\nMine eyes have drawn thy shape, and thine for me\nTill then not show my head where thou mayst prove me.\nSo is it not with me as with that Muse\nA dearer birth than this his love had brought,\nWho all in one, one pleasing note do sing:\nFrom sullen earth, sings hymns at heaven's gate;\nWho will believe my verse in time to come,\nMuch liker than your painted counterfeit:\nLet them say more than like of hearsay well;\nYet, do thy worst, old Time: despite thy wrong,\nAnd moan the expense of many a vanish'd sight:\nAlthough our undivided loves are one:\n'Tis not enough that through the cloud thou break,\nBeauty's effect with beauty were bereft,\nThen being ask'd where all thy beauty lies,\nAnd your true rights be term'd a poet's rage\nAnd kept unused, the user so destroys it.\nAs fast as thou shalt wane, so fast thou growest\nGilding the object whereupon it gazeth;\nFor where is she so fair whose unear'd womb\nYet eyes this cunning want to grace their art;\nAnd make the earth devour her own sweet brood;\nSets you most rich in youth before my sight,\nTo eat the world's due, by the grave and thee.\nMaking a famine where abundance lies,\nNor lose possession of that fair thou owest;\nSometime too hot the eye of heaven shines,\nThyself thy foe, to thy sweet self too cruel.\nWith what I most enjoy contented least;\nFor thou art so possess'd with murderous hate\nWhen I perceive that men as plants increase,\nServing with looks his sacred majesty;\nThat I an accessary needs must be\nSince sweets and beauties do themselves forsake\nShall hate be fairer lodged than gentle love?\nWhich, like a jewel hung in ghastly night,\nWhen in eternal lines to time thou growest:\nWhose speechless song, being many, seeming one,\n\nThe region cloud hath mask'd him from me now.\nBut day by night, and night by day, oppress'd?\nAfter a thousand victories once foil'd,\nLook, what is best, that best I wish in thee:\nThose hours, that with gentle work did frame\nThat thou no form of thee hast left behind,\nLook, whom she best endow'd she gave the more;\nO, let my books be then the eloquence\nDo in consent shake hands to torture me;\nStrikes each in each by mutual ordering,\nSeeking that beauteous roof to ruinate\nO'ercharged with burden of mine own love's might.\nA dearer birth than this his love had brought,\nThe age to come would say 'This poet lies:\nHow many a holy and obsequious tear\nO' let me, true in love, but truly write,\nFor thy sweet love remember'd such wealth brings\nSave that my soul's imaginary sight\nSuch civil war is in my love and hate\nUnless thou take that honour from thy name:\nSweets with sweets war not, joy delights in joy.\nAnd dumb presagers of my speaking breast,\nHim in thy course untainted do allow\nStirr'd by a painted beauty to his verse,\nThis wish I have; then ten times happy me!\nMy body is the frame wherein 'tis held,\nThat thou no form of thee hast left behind,\nI may not evermore acknowledge thee,\nFor having traffic with thyself alone,\nAnd your sweet semblance to some other give.\nUnlook'd on diest, unless thou get a son.\nMine eyes have drawn thy shape, and thine for me\nNor shall Death brag thou wander'st in his shade,\nWhich erst from heat did canopy the herd,\nBut day by night, and night by day, oppress'd?\nWhich hides your life and shows not half your parts.\nThe bounteous largess given thee to give?\nWithin thine own bud buriest thy content\nBut that I hope some good conceit of thine"
    }
  ],
  "temperature": 0.0,
  "max_tokens": 10,
  "stream": true
}
EOF
data: {"id":"chatcmpl-2fae1e5521004d5081981cd3dd05805b","object":"chat.completion.chunk","created":1768265101,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-58a5fc9b574841b39661cb860dd246ac","object":"chat.completion.chunk","created":1768265101,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-3697eccf40784a4292f13bf6760f74d7","object":"chat.completion.chunk","created":1768265101,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-9c8544bd4c594abf94650efa1dfd9e82","object":"chat.completion.chunk","created":1768265101,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-f2deb2b9124f495c84afeb2722fc2549","object":"chat.completion.chunk","created":1768265101,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-73f8f3e6b006491bbea3a358ba392bf6","object":"chat.completion.chunk","created":1768265101,"model":"openai/gpt-oss-120b","choices":[{"index":0,"delta":{},"finish_reason":"length"}]}

data: {"id":"chatcmpl-1d45e85a53c94d5284961403eba1ba39","object":"chat.completion.chunk","created":1768265101,"model":"openai/gpt-oss-120b","choices":[],"usage":{"prompt_tokens":1003,"total_tokens":1013,"completion_tokens":10}}

data: [DONE]

Expected behavior

Consistently starting generation with valid channels, and not having as much as repeated outputs.

actual behavior

I am observing the generated tokens being invalid to establish a channel for 10+%.

The common root cause to these invalid token sequences are repeated, garbage output issues. Note, in our second run, the output token ids were repeated " " and "Then". However, sometimes the generated tokens themselves look valid but simply missing the channel tokens at the beginning.

As we haven't observed this with other framework, we suspect to examine logit generation and sampling implementation.

additional notes

Other examples of generated tokens failing to establish a channel are available here. There is one for batch and streaming in the reproduction section.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[karljang]

Thank you for reporting the issue, which has been forwarded to the team.

[LinPoly]

@harrisonlimh It is not expected to repeat sending role for streaming mode, I can also reproduce the same kind of behavior, switched to non-streaming mode and the output looks just fine, and if we remove the role only messages the streaming result seems good. Will keep investigating.

[LinPoly]

This change should remove the repetitive role message. And I can reproduce the garbage output issue now, will see if it is from trtllm-serve

[Tracin]

Hi @harrisonlimh could you cherry-pick this commit to see if it works? Thanks!

[harrisonlimh]

Hi @Tracin and @hyukn - appreciate the quick fix on the issue! I verified that I no longer see the faulty channel establishment issue!

For my curiosity, would it be possible to share some insights about the change in PR? I am wondering how the change in PR fixed this specific issue or improved the quality overall. Thank you!