What are ones options for manually defining the parallelization? #9881
-
(Q1) Does PyTorch Lightning enable parallelization across multiple dimension or does it only allow data parallelism? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
Dear @roman955b, 1 ) Currently, Lightning automatically implement distributed data parallelism. However, we are currently working on making manual parallelization for users who want deeper control of the parallelisation schema. 2 ) Lightning supports only (S, P) with DeepSpeed, FSDP integrations. 3 ) Yes, we are currently working on this. Here is an issue to track the conversation #9375 Best, |
Beta Was this translation helpful? Give feedback.
Dear @roman955b,
1 ) Currently, Lightning automatically implement distributed data parallelism. However, we are currently working on making manual parallelization for users who want deeper control of the parallelisation schema.
2 ) Lightning supports only (S, P) with DeepSpeed, FSDP integrations.
3 ) Yes, we are currently working on this. Here is an issue to track the conversation #9375
Best,
T.C