ddp replicates whole script

[rogertrullo]

Hi all, I am running a script that performs some calculations and then does a testing loop using multi GPU using a single node with DDP.
The code looks like this:

some prepocesing code
...
model = LitModel(path_weights)

dataloader = DataLoader(dataset, batch_size=512,
                        shuffle=False, num_workers=32)
trainer = pl.Trainer(accelerator='ddp', gpus=8)
trainer.test(model, dataloader)

The issue that I have is that it seems the the whole script is being computed several times, I can see that the preprocessing code is being called 8 times. It seems to be the same issue as posted in SO (https://stackoverflow.com/questions/66261729/pytorch-lightning-duplicates-main-script-in-ddp-mode)
Am I missing something in my code?

Thanks!

[rogertrullo]

after reading more carefully I realized that it is what it is supposed to do, I would like to know however, if is there a way to avoid replicating some heavy calculations several times, say I want to do the postprocessing only ones and share the results with all the other processes? i

[awaelchli]

You can do

if trainer.global_rank == 0:
    postprocess()

and it will run only in the first process. How you share your postprocessed data with the other processes depends on the data.

You can broadcast a tensor using trainer.accelerator.broadcast.
If you make predictions I suggest to use the Trainer.predict API.