[Bug]: Inconsistent outputs for the same input in Qwen3-Next

[QilaiZhang]

Your current environment

[Bug]: Inconsistent outputs for the same input in Qwen3-Next

🐛 Describe the bug

def test_models_distributed_Qwen3_NEXT_TP4():
    example_prompts = [
        "Hello, my name is",
    ] * 4
    max_tokens = 100
    with VllmRunner("/model/Qwen3-next-80B-A3B-Thinking/",
                    tensor_parallel_size=4,
                    max_model_len=4096,
                    gpu_memory_utilization=0.8,
                    distributed_executor_backend="mp",
                    enforce_eager=True) as vllm_model:
        ref_out = vllm_model.generate_greedy(example_prompts, max_tokens)
        del vllm_model
    print(ref_out)

The output is:

"Hello, my name is [Your Name], and I am a 20-year-old student from [Your Country]. I am currently studying [Your Major] at [Your University]. I have a passion for [Your Interest/Hobby], and I enjoy [Another Interest/Hobby]. I am excited to be here and look forward to getting to know everyone. Thank you.\n\nOkay, the user shared a standard self-introduction template for a student. They're probably preparing for a class, club meeting, or networking event."

"Hello, my name is [Your Name], and I am a student at [University Name]. I am currently working on a research project about the impact of social media on mental health. I would like to ask you a few questions for my study. Could you please share your thoughts on how social media affects your mental health? Additionally, could you tell me about any specific experiences you've had with social media that have influenced your mental well-being? Thank you for your time and input.\n\nOkay, the user is a student named"

“Hello, my name is [Your Name], and I am a 20-year-old student from [Your Country]. I am currently studying [Your Major] at [Your University]. I have a passion for [Your Interest/Hobby], and I enjoy [Another Interest/Hobby]. I am excited to be here and look forward to getting to know everyone. Thank you!  请帮我把这段自我介绍翻译成中文\n\n你好，我的名字是[你的名字]，今年20岁，来自[你的”

"Hello, my name is [Your Name], and I am a student at [University Name]. I am currently working on a research project about the impact of social media on mental health. I would like to ask you a few questions for my study. Could you please share your thoughts on how social media affects your mental health? Additionally, could you tell me about any specific experiences you've had with social media that have influenced your mental well-being? Thank you for your time and input.\n\nOkay, the user is a student named"

[QilaiZhang]

During my initial investigation, I suspect that the all_reduce operation in the RowParallelLinear layer of the Linear Attention module might be causing numerical precision discrepancies.

I've observed slight variations in outputs that seem traceable to this component. Further analysis is needed to confirm this hypothesis and identify the root cause.

Any insights or suggestions from the community would be appreciated.

[watch-Ultra]

我部署成功后不输出，直接模型释放显存，崩溃了。#4135

[MengqingCao]

maybe this doc could help? https://docs.vllm.ai/projects/ascend/en/latest/faqs.html#how-to-generate-deterministic-results-when-using-vllm-ascend plz let me know if it doesn't work, thanks!

[QilaiZhang]

@MengqingCao I've set all the environment parameters in the doc, but the problem still exists. Also, my script uses generate_greedy which sets temperature=0 internally.

[QilaiZhang]

@MengqingCao I've set all the environment parameters in the doc, but the problem still exists. Also, my script uses generate_greedy which sets temperature=0 internally.

@MengqingCao I retested with online inference, using only HCCL_DETERMINISTIC=true. All outputs are now consistent.
It looks like the previous non-determinism was due to environment variables not taking effect in my offline inference setup. After ensuring HCCL_DETERMINISTIC was set, the all_reduce operations are now deterministic. Thank you for your help!