DataParallel + DataListLoader question

[DL-WallModel]

Hi everyone!

I'm trying to make a multi GPU implementation following the example found in /examples/multi_gpu/data_parallel.py

However, I have a problem with the forward function. In the example commented above, the list of data provided by the loader is fed straightforwardly to the forward function and then, the data is extracted as follows:

def forward(self, data):
    x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr

being data a data_list:

for data_list in loader:
    optimizer.zero_grad()
    output = model(data_list)

I'm trying to proceed the same way, but when I attempt to extract the data, an error is raised:
AttributeError: 'list' object has no attribute 'x'

Actually, this error makes sense to me, since the attribute x belongs to Data and not to a list. Please, could you tell me why in the example, the attributes x, edge_index, and edge_attr are extracted from a data_list? Sorry, I tried to run the example on my computer, but I don't know why it can't download the MNISTSuperpixels dataset and I can't check the variable types on my own.

Thank you in advance for your help! best regards
Joan

[rusty1s]

Did you forget to call DataParallel(model)? DataParallel will take care of distributing the list of data objects to different GPU creating mini-batches out of them.

Furthermore, I suggest you to look at our distributed training examples as well (https://github.com/rusty1s/pytorch_geometric/blob/master/examples/multi_gpu/distributed_batching.py) since this will allow you to have more control and is the recommended PyTorch way to train models on multiple GPUs as well.

[DL-WallModel]

Hi! Thank you for quick reply! No, I did not forget to call DataParallel. I'm calling it this way:

model = GCN()
if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)
model.to(device)

Then, I'm passing the data to the model this way:

  train_loader = DataListLoader(train_dataset, batch_size=train_batch_size, shuffle=True)

  for data_list in train_loader:

      optimizer.zero_grad()  #setting all the gradient to zero
      output = model(data_list) 

and in the forward function:

 def forward(self, data):
        print("cuda device=",torch.cuda.current_device()," data_list size=",len(data))

        x = data.x #here is where I get the error
        edge_index = data.edge_index

        x = F.relu(self.bn1(self.conv1(x, edge_index)))

In the present test, I'm using two GPU, and I checked whether the print was correct, and it was. So, I don't think this is the problem. Any other tips?

On the other hand, thanks for your proposal! However, at this moment I'm still quite new at using Pytorch geometric, and at this moment I would need something faster and easier. Once I master the more basic functionalities, I will try the distributed batching for sure! ;)

Thank you for your help!

[rusty1s]

Does our example script work for you? Are you using the DataParallel model from PyG or the one from PyTorch?

[DL-WallModel]

Hi! I was using the PyTorch method instead of the PyG one! Thank you for your help! I really appreciate it!