Why TP can't be used with pure DP?

[quic-meetkuma]

As per this, we can not be use TP along with pure DP (or DDP). We need to shard the model across further nodes by specifying dp_shard_size as well. Why this limitation exists? Is it just a software limitation?
Please share any documentation, code reference and justification for the same.

What to do inorder to do TP+DP?

[quic-akuruvil]

Also, what is the reason TP is only supported with FSDP currently?

[SunMarc]

DDP API is not super compatible with Dtensor for now. So this is why it is not compatible with FSDPv2 or TP. You can see in torchtitan that they have the same limitation: pytorch/pytorch#157445. When it gets, better, happy to add support for it

[quic-meetkuma]

Hi @SunMarc,
Looks like Pytorch guys are able to do plain DP+TP here: https://github.com/pytorch/pytorch/blob/7de041cb5a5817500b973eb32a70325187a83407/test/distributed/_composable/test_composability/test_2d_composability.py#L478