How Multi-GPU works?

[dl-noob]

Hey,

I have a question related to multi-gpu mode.

I created some pl trainer object like:

...
trainer = pl.Trainer(
    max_epochs=1,
    devices=2,
    accelerator="gpu",
    strategy="ddp"
)

and run prediction mode:

dataloader = Dataloader(dataset, ....)
output = trainer.predict(
    pipeline.eval(),
    dataloader
)
print("done")
print(len(dataset))
print(len(output))

and get such output:

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
Predicting: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 328/328 [00:42<00:00,  7.69it/s]
done
2620
328
Predicting: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 328/328 [00:42<00:00,  7.68it/s]
done
2620
328

I expected that multiple processes should be spawned only for internal computations of the Trainer. But in practice, my external function that starts Trainer.predict(...), returns multiple values, and it looks like external function spawned to multiple processes. Is it right? How can we handle it for work with a single process and single return values? Thanks!

p.s. is there any default function for all_gather of the returned predictions?