Unified placement group breaks non-colocated code path

[youngeunkwon0405]

Describe the bug

Unified placement (vllm model parallel > 8) causes an error in non-colocated code path.
It fails at the pynccl group initialization.
I think this code path has been buggy, but it used to work without crashing. I think #1264 reveals the bug to the surface.

Steps/Code to reproduce bug

Any non-colocated run with TP=16.

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

qwen30B_GTP16_failure.log

Add any other context about the problem here.

[youngeunkwon0405]

Hi @terrykong and @yuki-97 do you have any idea of a potential reason for this issue? I was trying to fix this but the placement logic and mapping things are quite complex in the current codebase. So I thought it might be very helpful to get opinions from someone who already has expertise in Ray.

It looks like after this PR #1264 the unified-placement-group code path crashes at init_collective function.

[yuki-97]

I have seen transport/nvls.cc:598 NCCL WARN Cuda failure 1 'invalid argument' in your log. We saw the same error info in some DSV3 experiments which use vllm parallel > 8, and NCCL_NVLS_ENABLE=0 can help to fix it. Can you try it?

For the reason why we haven't this error before, not sure but I guess may because we somehow didn't use NVL at this part before.

[youngeunkwon0405]

Hi @yuki-97, NCCL_NVLS_ENABLE=0 fixes the error. Thank you so much!
Looks like you have already faced this error and provided a fix #898.
I guess the fix is somehow disabled?

[yuki-97]

Hi @yuki-97, NCCL_NVLS_ENABLE=0 fixes the error. Thank you so much! Looks like you have already faced this error and provided a fix #898. I guess the fix is somehow disabled?

#898 just support passing NCCL_NVLS_ENABLE to vLLM, but didn't set the env var in code. Since we only found it happen in DSV3 and didn't realize it's b/c cross node parallel before, so we manually set it when starting a run that time.

Your PR will fix it.