Error: found at least two devices with DataParallel

[avivko]

I'm trying to train the model on GPUs only inside a notebook. Why is the cpu available as a device and how do I get rid of this error:

from torch_geometric.nn import GCNConv, GATConv, SAGEConv, global_add_pool
from torch.nn.functional import mse_loss, nll_loss, relu, softmax, cross_entropy
from torch.nn import functional as F
from torchmetrics.functional import accuracy
#from pytorch_lightning.metrics.functional import accuracy
import pytorch_lightning as pl
import torch.nn as nn

class GraphNets(pl.LightningModule):
    def __init__(self):
        super().__init__()

        if model_name == 'GCN':
            self.layer1 = GCNConv(in_channels=3, out_channels=config.n_hid)
            self.layer2 = GCNConv(in_channels=config.n_hid, out_channels=config.n_out)

        elif model_name == 'GAT':
            self.layer1 = GATConv(3, config.num_att_dim, heads=config.num_heads, dropout=config.dropout)
            self.layer2 = GATConv(config.num_att_dim * config.num_heads, out_channels = config.n_out, heads=1, concat=False,
                                 dropout=config.dropout)

        elif model_name == 'GraphSAGE':
            self.layer1 = SAGEConv(3, config.n_hid)
            self.layer2 = SAGEConv(config.n_hid, config.n_out)

        self.decoder = nn.Linear(config.n_out, 7)

    def forward(self, g):
        x = g.coords
        x = F.dropout(x, p=config.dropout, training=self.training)
        x = F.elu(self.layer1(x, g.edge_index))
        x = F.dropout(x, p=config.dropout, training=self.training)
        x = self.layer2(x, g.edge_index)
        x = global_add_pool(x, batch=g.batch)
        x = self.decoder(x)
        return softmax(x)

    def training_step(self, batch, batch_idx):
        x = batch
        y = x.y
        y_hat = self(x)
        loss = cross_entropy(y_hat, y)
        acc = accuracy(y_hat, y)

        self.log("train_loss", loss)
        self.log("train_acc", acc)
        return loss

    def validation_step(self, batch, batch_idx):
        x = batch
        y = x.y
        y_hat = self(x)
        loss = cross_entropy(y_hat, y)
        acc = accuracy(y_hat, y)
        self.log("valid_loss", loss)
        self.log("valid_acc", acc)

    def test_step(self, batch, batch_idx):
        x = batch
        y = x.y
        y_hat = self(x)
        loss = cross_entropy(y_hat, y)
        acc = accuracy(y_hat, y)

        y_pred_softmax = torch.log_softmax(y_hat, dim = 1)
        y_pred_tags = torch.argmax(y_pred_softmax, dim = 1)
        f1 = f1_score(y.detach().gpu().numpy(), y_pred_tags.detach().gpu().numpy(), average = 'weighted')

        self.log("test_loss", loss)
        self.log("test_acc", acc)
        self.log("test_f1", f1)

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=config.lr)
        return optimizer

from pytorch_lightning.callbacks import ModelCheckpoint
import os

file_path = '/vol/tmp/kormanav/graphein_example_model'
if not os.path.exists(file_path):
    os.mkdir(file_path)

checkpoint_callback = ModelCheckpoint(
    monitor="valid_loss",
    dirpath=file_path,
    filename="model-{epoch:02d}-{val_loss:.2f}",
    save_top_k=1,
    mode="min",
)

# Train Model
model = GraphNets()
trainer = pl.Trainer(max_epochs=200, gpus=-1, callbacks=[checkpoint_callback], strategy="dp", accelerator="gpu")
trainer.fit(model, train_loader, valid_loader)

# evaluate on the model with the best validation set
best_model = GraphNets.load_from_checkpoint(checkpoint_callback.best_model_path)
out_best_test = trainer.test(best_model, test_loader)[0]

Results in:

INFO:pytorch_lightning.utilities.rank_zero:GPU available: True, used: True
INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores
INFO:pytorch_lightning.utilities.rank_zero:IPU available: False, using: 0 IPUs
INFO:pytorch_lightning.utilities.rank_zero:HPU available: False, using: 0 HPUs
WARNING:pytorch_lightning.loggers.tensorboard:Missing logger folder: /home/biologie/kormanav/lightning_logs
INFO:pytorch_lightning.accelerators.gpu:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2]
INFO:pytorch_lightning.callbacks.model_summary:
  | Name    | Type    | Params
------------------------------------
0 | layer1  | GCNConv | 32    
1 | layer2  | GCNConv | 72    
2 | decoder | Linear  | 63    
------------------------------------
167       Trainable params
0         Non-trainable params
167       Total params
0.001     Total estimated model params size (MB)
Sanity Checking DataLoader 0: 0%
0/1 [00:00<?, ?it/s]
/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/overrides/data_parallel.py:99: UserWarning:

Could not determine on which device the inputs are. When using DataParallel (strategy='dp'), be aware that in case you are using self.device in your code, it will reference only the root device.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [33], in <cell line: 4>()
      2 model = GraphNets()
      3 trainer = pl.Trainer(max_epochs=200, gpus=-1, callbacks=[checkpoint_callback], strategy="dp", accelerator="gpu")
----> 4 trainer.fit(model, train_loader, valid_loader)
      6 # evaluate on the model with the best validation set
      7 best_model = GraphNets.load_from_checkpoint(checkpoint_callback.best_model_path)

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:768, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    749 r"""
    750 Runs the full optimization routine.
    751 
   (...)
    765     datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`.
    766 """
    767 self.strategy.model = model
--> 768 self._call_and_handle_interrupt(
    769     self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    770 )

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:721, in Trainer._call_and_handle_interrupt(self, trainer_fn, *args, **kwargs)
    719         return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
    720     else:
--> 721         return trainer_fn(*args, **kwargs)
    722 # TODO: treat KeyboardInterrupt as BaseException (delete the code below) in v1.7
    723 except KeyboardInterrupt as exception:

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:809, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    805 ckpt_path = ckpt_path or self.resume_from_checkpoint
    806 self._ckpt_path = self.__set_ckpt_path(
    807     ckpt_path, model_provided=True, model_connected=self.lightning_module is not None
    808 )
--> 809 results = self._run(model, ckpt_path=self.ckpt_path)
    811 assert self.state.stopped
    812 self.training = False

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1234, in Trainer._run(self, model, ckpt_path)
   1230 self._checkpoint_connector.restore_training_state()
   1232 self._checkpoint_connector.resume_end()
-> 1234 results = self._run_stage()
   1236 log.detail(f"{self.__class__.__name__}: trainer tearing down")
   1237 self._teardown()

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1321, in Trainer._run_stage(self)
   1319 if self.predicting:
   1320     return self._run_predict()
-> 1321 return self._run_train()

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1343, in Trainer._run_train(self)
   1340 self._pre_training_routine()
   1342 with isolate_rng():
-> 1343     self._run_sanity_check()
   1345 # enable train mode
   1346 self.model.train()

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1411, in Trainer._run_sanity_check(self)
   1409 # run eval step
   1410 with torch.no_grad():
-> 1411     val_loop.run()
   1413 self._call_callback_hooks("on_sanity_check_end")
   1415 # reset logger connector

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/loops/base.py:204, in Loop.run(self, *args, **kwargs)
    202 try:
    203     self.on_advance_start(*args, **kwargs)
--> 204     self.advance(*args, **kwargs)
    205     self.on_advance_end()
    206     self._restarting = False

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py:154, in EvaluationLoop.advance(self, *args, **kwargs)
    152 if self.num_dataloaders > 1:
    153     kwargs["dataloader_idx"] = dataloader_idx
--> 154 dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
    156 # store batch level output per dataloader
    157 self._outputs.append(dl_outputs)

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/loops/base.py:204, in Loop.run(self, *args, **kwargs)
    202 try:
    203     self.on_advance_start(*args, **kwargs)
--> 204     self.advance(*args, **kwargs)
    205     self.on_advance_end()
    206     self._restarting = False

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py:127, in EvaluationEpochLoop.advance(self, data_fetcher, dl_max_batches, kwargs)
    124 self.batch_progress.increment_started()
    126 # lightning module methods
--> 127 output = self._evaluation_step(**kwargs)
    128 output = self._evaluation_step_end(output)
    130 self.batch_progress.increment_processed()

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py:222, in EvaluationEpochLoop._evaluation_step(self, **kwargs)
    220     output = self.trainer._call_strategy_hook("test_step", *kwargs.values())
    221 else:
--> 222     output = self.trainer._call_strategy_hook("validation_step", *kwargs.values())
    224 return output

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1763, in Trainer._call_strategy_hook(self, hook_name, *args, **kwargs)
   1760     return
   1762 with self.profiler.profile(f"[Strategy]{self.strategy.__class__.__name__}.{hook_name}"):
-> 1763     output = fn(*args, **kwargs)
   1765 # restore current_fx when nested context
   1766 pl_module._current_fx_name = prev_fx_name

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/strategies/dp.py:129, in DataParallelStrategy.validation_step(self, *args, **kwargs)
    127 def validation_step(self, *args, **kwargs) -> Optional[STEP_OUTPUT]:
    128     with self.precision_plugin.val_step_context():
--> 129         return self.model(*args, **kwargs)

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py:168, in DataParallel.forward(self, *inputs, **kwargs)
    166     return self.module(*inputs[0], **kwargs[0])
    167 replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
--> 168 outputs = self.parallel_apply(replicas, inputs, kwargs)
    169 return self.gather(outputs, self.output_device)

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py:178, in DataParallel.parallel_apply(self, replicas, inputs, kwargs)
    177 def parallel_apply(self, replicas, inputs, kwargs):
--> 178     return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py:86, in parallel_apply(modules, inputs, kwargs_tup, devices)
     84     output = results[i]
     85     if isinstance(output, ExceptionWrapper):
---> 86         output.reraise()
     87     outputs.append(output)
     88 return outputs

File /glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/_utils.py:457, in ExceptionWrapper.reraise(self)
    453 except TypeError:
    454     # If the exception takes multiple arguments, don't try to
    455     # instantiate since we don't know how to
    456     raise RuntimeError(msg) from None
--> 457 raise exception

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/overrides/data_parallel.py", line 64, in forward
    output = super().forward(*inputs, **kwargs)
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 93, in forward
    return self.module.validation_step(*inputs, **kwargs)
  File "/tmp/ipykernel_55211/3120268465.py", line 52, in validation_step
    y_hat = self(x)
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/tmp/ipykernel_55211/3120268465.py", line 31, in forward
    x = F.elu(self.layer1(x, g.edge_index))
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 191, in forward
    x = self.lin(x)
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/glusterfs/dfs-gfs-dist/kormanav/miniconda3/envs/graphein-gpu/lib/python3.8/site-packages/torch_geometric/nn/dense/linear.py", line 118, in forward
    return F.linear(x, self.weight, self.bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_mm)

I am new to using pytorch/pytorch_lightning, so I am sorry if the problem might be super trivial. I will really appreciate your help!

Thanks!

[avivko]

In the meanwhile I just decided to train the model by running this code as a python script (and not in a notebook) with strategy="ddp" and the training works now. I would still be happy to have a solution to the problem above though (with strategy="dp", for use in notebooks).
Thanks!

[akihironitta]

@avivko Do you see the error regardless of the model_name? I'm not a heavy user of DP at all 😓, but I could probably have a look at it this weekend if you could provide the full running script and your env details.

[avivko]

@akihironitta Thanks for responding!
Here are the relevant env details (tell me if you need more info):

torch version: 1.11.0
torch-geometric version: 2.0.4
pytorch-lightning version: 1.6.3

And here is a link to Google Drive with the running script (with ddp, as .py file) and the corresponding pickle file some randomly-labeled small test data. Once you change the strategy to dp, the error mentioned above should be thrown.
https://drive.google.com/drive/folders/1yJeWH1dUlzGFHNBOshTrHBgIrNFGJ31U?usp=sharing

Oh and if you find a solution, please make sure to test it out in a Jupyter notebook.

Thanks for helping out!

[akihironitta]

(@avivko I'm sorry I've been busy and haven't got a chance to test out your script yet...)

[akihironitta]

Hi @avivko, not directly related to your issue, but just wanted to let you know that the latest release (v1.7) added an experimental new strategy Trainer(strategy="ddp_notebook") that you can use in notebooks environments. See #13980 for the v1.7 release note :)