Add data parallel support

[wuxun-zhang]

This is to add data parallel support for V1 gaudi plugin.

• add dp aware padding
• use all_gather and reduce_scatter
• add data parallel example

[adobrzyn]

Please resolve conflicts

[adobrzyn]

/run-gaudi-tests

[wuxun-zhang]

@adobrzyn Removed unused codes and rebased. Please review again.

[wuxun-zhang]

/run-gaudi-tests

[sys-hab-pt-service]

Only codeowners can request to run Gaudi tests. Contact list: kzawora-intel, xuechendi, mswiniarsk, adobrzyn

[wuxun-zhang]

@adobrzyn @xuechendi @mswiniarsk @kzawora-intel Please help review this. Thanks.

[adobrzyn]

/run-gaudi-tests

[wuxun-zhang]

Seems upstream vllm changes break gaudi plugin.

FAILED vllm-gaudi/tests/unit_tests/worker/test_hpu_model_runner.py::test_init_kv_cache_without_kv_sharing - AttributeError: 'ModelConfig' object has no attribute 'is_multimodal_raw_input_supported'. Did you mean: 'is_multimodal_raw_input_only_model'?

[adobrzyn]

/run-gaudi-tests

[Copilot]

Pull Request Overview

This PR adds data parallel support for the V1 Gaudi plugin by implementing DP-aware padding mechanisms and collective operations.

• Implements DP-aware padding for prefill and decode batches to ensure consistent tensor shapes across data parallel ranks
• Adds collective communication operations (all_gather, reduce_scatter) for expert parallelism support
• Includes a comprehensive data parallel example script with multi-node support

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
vllm_gaudi/v1/worker/hpu_worker.py	Updates distributed initialization to handle data parallel configuration and adds dummy batch execution
vllm_gaudi/v1/worker/hpu_model_runner.py	Implements DP-aware padding logic and dummy batch creation for consistent execution across ranks
vllm_gaudi/platform.py	Adds simple compile backend configuration
vllm_gaudi/distributed/device_communicators/hpu_communicator.py	Implements dispatch/combine methods for expert parallelism with collective operations
tests/full_tests/ci_tests.sh	Adds CI test for data parallel functionality
examples/data_parallel.py	Provides complete example demonstrating data parallel usage

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Fix typo: 'seuquence' should be 'sequence'.

Suggested change

	For testing different seuquence length in data parallel scenario
	For testing different sequence length in data parallel scenario

[adobrzyn]

Is this a copy-paste example file from upstream examples/offline_inference/data_parallel.py? Why are we coping it to our rep? We can use it from upstream

[wuxun-zhang]

It's intended for CI test, so I directly copy into plugin repo.
If copying not expected, any suggestion on how to test in CI?

[wuxun-zhang]

where False = <function isclose at 0x71d02ac55a70>(0.9, 0.84375, rtol=0.06)
2025-09-03T09:17:18Z tensorflow test_lm_eval_correctness.py:196: AssertionError
FAILED test_lm_eval_correctness.py::test_lm_eval_correctness - assert False

CI failed on above case, but I re-test in k8s and cannot reproduce due to measured_value is higher.