How many effective workers does my code use?

[miraodasilva]

If I select X workers in my dataloader, and I train my model using ddp with Y GPUs, is the effective amount of workers running in the machine X*Y or X?

[awaelchli]

Yes! The script will launch Y times in a separate process. Each process will then create their own dataloader with as many workers as given. So X*Y workers in total accessing your filesystem.

[miraodasilva]

Great, that's what I thought. Thanks a lot!

[rohitgr7]

just wondering will a particular worker handles the request calls from each dataloader one at a time or that will be parallelized too?

[awaelchli]

Each worker will process one sample at a time and once a batch is formed, it will apply collate_fn and return the batch back to the main process in a queue. While the batch waits in the main process to be consumed by the user, the worker will continue to load and process data. The worker always knows which samples to load because these are provided through a queue by the main process dataloader.

Here is the full picture how the processes are layed out with DDP 2GPUs + num_workers=2:

DDP process 0:
>--------------|---------------------------------------------------->
     worker 0  + --------------------------------> 
     worker 1  + -------------------------------->



DDP process 1:
>--------------|---------------------------------------------------->
     worker 2  + -------------------------------->
     worker 3  + -------------------------------->

In DDP process 0, dataloader will spawn 2 processes (where the + is) for data loading. Each worker will get a set of indices and load data in parallel. The result will be put in a queue that the main process reads and returns the data to the user (not visualized).
The exact same happens independently in parallel in DDP process 1.

References: https://pytorch.org/docs/stable/data.html

[rohitgr7]

I see. TIL. Thanks, @awaelchli for the great explanation. 😃