Single GPU DDP Lightning implementation is 3x slower than Pytorch - Help needed

[sreeaurovindh]

I am trying to convert my Pytorch code into Lightning code with the goal of scaling up and reducing my training time. I am using DDP with single GPU and comparing it against a non lightning Pytorch implementation. The lightning implementation is 3x slower.The following is the simple profiler output.

If you have any recommendations on what I can look out for to debug , please let me know or any logs that may help, please let me know

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  275.12         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------
run_training_epoch                 	|  125.14         	|2              	|  250.28         	|  90.971         	|
run_training_batch                 	|  0.69695        	|200            	|  139.39         	|  50.665         	|
optimizer_step_and_closure_0       	|  0.63268        	|200            	|  126.54         	|  45.993         	|
training_step_and_backward         	|  0.63178        	|200            	|  126.36         	|  45.927         	|
model_forward                      	|  0.60904        	|200            	|  121.81         	|  44.274         	|
training_step                      	|  0.60867        	|200            	|  121.73         	|  44.247         	|
get_train_batch                    	|  0.26221        	|200            	|  52.441         	|  19.061         	|
on_validation_end                  	|  2.9615         	|3              	|  8.8846         	|  3.2293         	|
evaluation_step_and_end            	|  0.70029        	|12             	|  8.4035         	|  3.0545         	|
validation_step                    	|  0.69981        	|12             	|  8.3977         	|  3.0523         	|
backward                           	|  0.022139       	|200            	|  4.4278         	|  1.6094         	|
on_train_batch_end                 	|  0.00091574     	|200            	|  0.18315        	|  0.06657        	|
cache_result                       	|  1.7525e-05     	|870            	|  0.015246       	|  0.0055417      	|
on_validation_batch_end            	|  0.00082211     	|12             	|  0.0098653      	|  0.0035858      	|
on_after_backward                  	|  4.5223e-05     	|200            	|  0.0090446      	|  0.0032875      	|
on_batch_start                     	|  3.7663e-05     	|200            	|  0.0075325      	|  0.0027379      	|
on_before_zero_grad                	|  3.3201e-05     	|200            	|  0.0066403      	|  0.0024136      	|
on_batch_end                       	|  2.9852e-05     	|200            	|  0.0059704      	|  0.0021701      	|
on_train_batch_start               	|  2.7909e-05     	|200            	|  0.0055818      	|  0.0020289      	|
training_step_end                  	|  2.1387e-05     	|200            	|  0.0042773      	|  0.0015547      	|
on_train_start                     	|  0.0036514      	|1              	|  0.0036514      	|  0.0013272      	|
on_validation_start                	|  0.00064151     	|3              	|  0.0019245      	|  0.00069952     	|
on_train_epoch_start               	|  0.00048288     	|2              	|  0.00096576     	|  0.00035103     	|
on_train_epoch_end                 	|  0.00047305     	|2              	|  0.00094611     	|  0.00034389     	|
on_validation_batch_start          	|  6.6105e-05     	|12             	|  0.00079326     	|  0.00028833     	|
on_train_end                       	|  0.00055569     	|1              	|  0.00055569     	|  0.00020198     	|
validation_step_end                	|  3.2875e-05     	|12             	|  0.0003945      	|  0.00014339     	|
on_epoch_start                     	|  4.8603e-05     	|5              	|  0.00024301     	|  8.8329e-05     	|
on_epoch_end                       	|  4.5898e-05     	|5              	|  0.00022949     	|  8.3414e-05     	|
on_validation_epoch_end            	|  3.3901e-05     	|3              	|  0.0001017      	|  3.6967e-05     	|
on_validation_epoch_start          	|  2.1253e-05     	|3              	|  6.3758e-05     	|  2.3174e-05     	|
on_fit_start                       	|  2.7039e-05     	|1              	|  2.7039e-05     	|  9.828e-06      	|
on_train_dataloader                	|  1.7258e-05     	|1              	|  1.7258e-05     	|  6.2728e-06     	|
on_val_dataloader                  	|  1.5379e-05     	|1              	|  1.5379e-05     	|  5.5899e-06     	|
on_before_accelerator_backend_setup	|  1.0664e-05     	|1              	|  1.0664e-05     	|  3.8761e-06     	|

[ananthsub]

What effect would DDP have on a single GPU? Did you try training with multiple GPUs?

[sreeaurovindh]

Thanks for your response. I tried with multiple GPUs. Here are my statistics

1. 2 hrs 10 mins ( Single GPU, Non distributed, PyTorch)
2. 6 hrs 36 mins ( Single GPU, DDP)
3. 2 hrs 6 mins ( 4 GPU, DDP)

I am not sure what I am missing or why there was a three fold increase in training time

[awaelchli]

And what are the numbers for single GPU, no DDP in Lightning? Is it even possible to select DDP with 1 GPU in Lightning?