I cannot assign the GPU index by ddp or dp backends..

[mhh0318]

It's weired. I have a single machine with 8 GPUSs and now 0~4 are full load.
I would like to use parallel training on 5 6 7 GPUs but cannot assign the task to them.

Some of my codes:

` parser = argparse.ArgumentParser(description='Solver')
parser.add_argument('--config', required=True, type=str)
fargs = parser.parse_args()
args = parse_config(fargs.config)

data = DDPMData(args.Data)
data.train_dataloader()
data.test_dataloader()

num_cls = data.n_classes
shape = data.data_shapes
args.Model.Unet.kwargs.num_classes = num_cls
args.Model.DiscreteDiffusion.kwargs.num_class = num_cls
args.Model.DiscreteDiffusion.kwargs.shape = shape
model = DDDPM(args)

wandb_logger = WandbLogger()

callbacks = []
callbacks.append(ModelCheckpoint(save_last=True,every_n_train_steps=600))
callbacks.append(LearningRateMonitor(logging_interval='step'))

trainer = pl.Trainer(callbacks=callbacks,max_steps=args.max_steps, 
                accelerator='dp', gpus=[5,6,7],
                logger = wandb_logger, check_val_every_n_epoch=120,
                num_sanity_val_steps=0)
trainer.fit(model, data)

`
And now nvidia-smi shows:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.42.01 Driver Version: 470.42.01 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:04:00.0 Off | N/A |
| 63% 72C P2 151W / 200W | 7997MiB / 8119MiB | 85% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce ... On | 00000000:06:00.0 Off | N/A |
| 71% 77C P2 134W / 200W | 8109MiB / 8119MiB | 98% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA GeForce ... On | 00000000:07:00.0 Off | N/A |
| 82% 83C P2 143W / 200W | 7405MiB / 8119MiB | 89% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA GeForce ... On | 00000000:08:00.0 Off | N/A |
| 80% 83C P2 158W / 200W | 7263MiB / 8119MiB | 59% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 NVIDIA GeForce ... On | 00000000:0C:00.0 Off | N/A |
| 78% 82C P2 148W / 200W | 5719MiB / 8119MiB | 100% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 5 NVIDIA GeForce ... On | 00000000:0D:00.0 Off | N/A |
| 0% 24C P8 7W / 200W | 4MiB / 8119MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 6 NVIDIA GeForce ... On | 00000000:0E:00.0 Off | N/A |
| 0% 34C P8 8W / 200W | 4MiB / 8119MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 7 NVIDIA GeForce ... On | 00000000:0F:00.0 Off | N/A |
| 0% 25C P8 7W / 200W | 4MiB / 8119MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 4082 C python 7993MiB |
| 1 N/A N/A 13619 C python 8105MiB |
| 2 N/A N/A 4082 C python 7401MiB |
| 3 N/A N/A 4082 C python 7259MiB |
| 4 N/A N/A 24206 C python 5715MiB |
+-----------------------------------------------------------------------------+

and when I run the codes, it will assign task to GPU:2 , I don't know why..
RuntimeError: CUDA out of memory. Tried to allocate 60.00 MiB (GPU 2; 7.93 GiB total capacity; 117.19 MiB already allocated; 45.50 MiB free; 120.00 MiB reserved in total by PyTorch)

Any idea..?

##############################
torch = 1.7.1
pl = 1.3.8

[mhh0318]

Problemed solved.
I forgot to map device when I load a ckpt file as data...