fix(ci): shard Nemotron Super vLLM deploy across 8 GPUs

[yuhezhang-ai]

What does this PR do?

Run the Nemotron Super V3 BF16 vLLM deployment checks across all eight GPUs and enable vLLM expert parallelism for the model's MoE layers.

The vLLM launcher defaults ci.vllm_deploy_gpus to one. Both Super V3 HellaSwag recipes enabled the native vLLM smoke path without overriding that default, so the smoke test derived tensor_parallel_size=1 and failed while allocating the 120B model. NVIDIA's deployment guidance uses tensor parallel size 8 and supports expert parallelism for this checkpoint.

This is the configuration fix for AM-668, a recurrence of AM-446.

Changelog

• Set ci.vllm_deploy_gpus: 8 for the Nemotron Super V3 HellaSwag SFT and PEFT recipes.
• Add an opt-in ci.vllm_enable_expert_parallel deploy-test setting and enable it for both 120B recipes.
• Preserve the existing behavior for all recipes that do not opt in.
• Add focused CPU unit coverage for enabled and default-disabled argument resolution.

Scoped CI

• Pipeline: https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/pipelines/57979613
• Combined validation SHA: 58a99704724275808f188e2afecf8212c9a33ec3 (fix(test): use SDPA for Nemotron-H HF reload #3060 + fix(ci): shard Nemotron Super vLLM deploy across 8 GPUs #3061)
• Scope: llm / release / eos / x86
• Checkpoint producer: Passed — nemotron_super_v3_hellaswag_peft (job 362958206)
• vLLM deploy: Blocked by deploy image — nemotron_super_v3_hellaswag_peft_vllm_deploy (job 362958210)

The deploy log confirms tensor_parallel_size=8 and enable_expert_parallel=True. It then reproduces the model-construction OOM under vLLM 0.11.1+...nv25.12. AutoModel's automation-owned deploy image still uses nvcr.io/nvidia/vllm:25.12-py3; NVIDIA's current Nemotron Super V3 guidance requires vLLM 0.18.1, and the NVIDIA 26.04 container explicitly adds Nemotron Super V3 support. The remaining blocker is therefore the shared deploy image, not this recipe's GPU or parallelism configuration.

Before your PR is "Ready for review"

Pre checks:

• Read and followed the contributor guidelines.
• Parsed both updated YAML recipes and verified they resolve GPUs 0,1,2,3,4,5,6,7.
• Added focused unit coverage: 2 passed.
• Ran ruff format --check on the changed Python files.
• Ran ruff check on the changed Python files.
• Ran the scoped checkpoint producer successfully.
• Confirmed TP8 and EP reach vLLM in the deploy job.
• Rerun the deploy job after @nvidia-nemo/automation upgrades the shared vLLM image.

Additional Information

• Linear: AM-668
• Exact earlier duplicate: AM-446
• Broader prior recurrence: AM-561

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yuhezhang-ai]

/ok to test b5c7a71

[yuhezhang-ai]

@nvidia-nemo/automation could you advise on or own an upgrade of the shared AutoModel deploy image?

The latest combined validation confirms the recipe-side configuration now reaches vLLM correctly:

• TP8 + expert parallel are present in vLLM's non-default args.
• The checkpoint producer passes: job 362958206.
• The deploy still OOMs during native Nemotron-H MoE weight construction: job 362958210.

The deploy image currently inherits nvcr.io/nvidia/vllm:25.12-py3, which provides vLLM 0.11.1+...nv25.12. NVIDIA's current Nemotron Super V3 model guidance requires vLLM 0.18.1, and NVIDIA vLLM container 26.04 explicitly adds Nemotron Super V3 support.

Would you prefer to handle the docker/Dockerfile.deploy base-image upgrade, or should we send a separate infrastructure PR upgrading it to at least 26.04 and run the deploy matrix?

[athitten]

@donghyukc to advise