Improve performance on DAWN when running with COMPOSITE hierarchy (CSD3)

[rwood-97]

We are seeing ~2x worse performance when using ZE_DEVICE_HIERARCHY = COMPOSITE vs FLAT.

See suggestions for other environment variables here

TODO:

• Try I_MPI_OFFLOAD
• Try I_MPI_OFFLOAD_SYMMETRIC
• Dump xpu-smi stats when running jobs

[Iain-S]

• Intel mention "Explicit" scaling here. Does that mean that we can exercise more control over how the work is spread across multiple tiles?

[rwood-97]

No I_MPI_OFFLOAD

Results from #17 (comment)
1 node, 4 GPU.

[0] Average time per epoch (ignoring first): 17.708354949951172 seconds
[0] Effective time for an epoch: 4.427088737487793 seconds
[0] Equivalent training speed: 0.22588208317756653 epochs per seconds
[0] Total time for 32 epochs: 897.4592895507812
[0] Total time: 271.16698002815247

and

[0] Average time per epoch (ignoring first): 17.7179012298584 seconds
[0] Effective time for an epoch: 4.4294753074646 seconds
[0] Equivalent training speed: 0.22576037049293518 epochs per seconds
[0] Total time for 32 epochs: 993.3740234375
[0] Total time: 318.66189789772034

and

[0] Average time per epoch (ignoring first): 17.70682716369629 seconds
[0] Effective time for an epoch: 4.426706790924072 seconds
[0] Equivalent training speed: 0.22590157389640808 epochs per seconds
[0] Total time for 32 epochs: 984.6820678710938
[0] Total time: 321.9719829559326

I_MPI_OFFLOAD

Using 1 node, 4 GPUs, I_MPI_OFFLOAD=1

[0] Average time per epoch (ignoring first): 17.7200984954834 seconds
[0] Effective time for an epoch: 4.43002462387085 seconds
[0] Equivalent training speed: 0.22573238611221313 epochs per seconds
[0] Total time for 32 epochs: 1216.271484375
[0] Total time: 413.9336585998535

and

[0] Average time per epoch (ignoring first): 17.69921875 seconds
[0] Effective time for an epoch: 4.4248046875 seconds
[0] Equivalent training speed: 0.2259986698627472 epochs per seconds
[0] Total time for 32 epochs: 984.467529296875
[0] Total time: 321.9713673591614

I think from this we can conclude no difference.

I_MPI_OFFLOAD and I_MPI_OFFLOAD_SYMMETRIC

This seems to just hang after the following lines:

[0] My guessed rank = 0
[0] 2025:07:17-08:39:53:( 8408) |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
[0] 2025:07:17-08:39:53:( 8408) |CCL_WARN| value of CCL_ZE_IPC_EXCHANGE changed to be sockets (default:pidfd)

[rwood-97]

No I_MPI_OFFLOAD

Results from #17 (comment)
2 node, 8 GPU.

[0] Average time per epoch (ignoring first): 19.380022048950195 seconds
[0] Effective time for an epoch: 2.4225027561187744 seconds
[0] Equivalent training speed: 0.4127962291240692 epochs per seconds
[0] Total time for 32 epochs: 1380.4417724609375
[0] Total time: 211.3158929347992

and

[0] Average time per epoch (ignoring first): 19.383628845214844 seconds
[0] Effective time for an epoch: 2.4229536056518555 seconds
[0] Equivalent training speed: 0.4127194285392761 epochs per seconds
[0] Total time for 32 epochs: 1149.7550048828125
[0] Total time: 180.48243927955627

and

[0] Average time per epoch (ignoring first): 19.396133422851562 seconds
[0] Effective time for an epoch: 2.4245166778564453 seconds
[0] Equivalent training speed: 0.4124533534049988 epochs per seconds
[0] Total time for 32 epochs: 1171.873779296875
[0] Total time: 194.11750411987305

I_MPI_OFFLOAD

Using 2 nodes, 8 GPUs, I_MPI_OFFLOAD=1

[0] Average time per epoch (ignoring first): 19.60049819946289 seconds
[0] Effective time for an epoch: 2.4500622749328613 seconds
[0] Equivalent training speed: 0.40815287828445435 epochs per seconds
[0] Total time for 32 epochs: 1033.449462890625
[0] Total time: 143.5401611328125

I also tried to repeat this but got it hanging at the same point as above.

I_MPI_OFFLOAD and I_MPI_OFFLOAD_SYMMETRIC

Same as above, this hangs.