How to fix 'CUDA error: device-side assert triggered'?

[Kevin-shihello-world]

I want to train a model on a dataset made by myself but whenever I Run my training program it shows this warning:

python
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

I've asked this warning in the reply to other questions and I was first suggested to add '()'after x = x.cuda and then to remove the sentence of 'x = x.cuda()'. None of this works. So I searched on the internet and according to their words when this warning show after it successfully ran several epochs it means I may not use the right normalization, and while it shows it in the beginning it might be because I used the wrong label in my dataset like my y in my dataset's data are in the wrong type(my dataset is for a graph classification task). So I tried to let my read_myds_data function just return dataset[i] where the dataset is Proteins from tudataset but it still warns me the same.
I then load the data list with dataset[i] to the data loader to run the training program without a dataset and it turns out it would be the same when I let the dataset be my dataset while it runs successfully on the proteins dataset. But I've checked that each datas in my dataset are just like the ones in the proteins dataset, both have edge_index in shape [2, edge_num], x in shape [x.num, x.feature.num], and y in shape [1] and the tree are all tensors (I mean sometimes load dataset[i] with the dataset to be proteins also offered me this warning but it would run successfully after I rewrite it in a new document for thought some parameters about dimension may stick to the old one).
And after I rewrite the whole program with training and my dataset to a new document it would warn me like the following :

python
/tmp/ipykernel_189/1917562624.py in train(train_loader)
     90         data = data.to(device)
     91         optimizer.zero_grad()
---> 92         out, tot_loss = model(data.x, data.edge_index, data.batch)
     93         loss = F.nll_loss(out, data.y.view(-1)) + tot_loss
     94         loss.backward()

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/tmp/ipykernel_189/1917562624.py in forward(self, x, edge_index, batch)
     57 
     58     def forward(self, x, edge_index, batch):
---> 59         x = self.conv1(x, edge_index).relu()
     60 
     61         x, mask = to_dense_batch(x, batch)

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch_geometric/nn/conv/gcn_conv.py in forward(self, x, edge_index, edge_weight)
    192                     edge_index = cache
    193 
--> 194         x = self.lin(x)
    195 
    196         # propagate_type: (x: Tensor, edge_weight: OptTensor)

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch_geometric/nn/dense/linear.py in forward(self, x)
    116             x (Tensor): The features.
    117         """
--> 118         return F.linear(x, self.weight, self.bias)
    119 
    120     @torch.no_grad()

RuntimeError: mat1 and mat2 shapes cannot be multiplied (640x32 and 128x40)
​```
Lots of the helping pages on the internet say they've read the other's helping pages on this problem and they do not work so they found a new solution and write it to the next troubled people. So I think it must be a common problem and if the right solution is found it would help a lot of people so I would appreciate it if someone offer an answer to this question.

[rusty1s]

Can you ensure that the following runs through?

for data in dataset:
    assert data.x.size(1) == dataset[0].x.size(1)

[Kevin-shihello-world]

Wait,last time you say you round my code properly in the last question I asked . And uh, so can you tell me which cuda version did you use? It may help solving this problem a lot.
Thank you.

[Kevin-shihello-world]

I'm sorry I didn't find @afonso Sousa asked a question like this one. But his question hasn't got his answer anyway. And I found an answer to his question that seems to suit me.
I printed this

python 
data = data.to(args.device)
            print(data.validate())

And it says

python
ValueError: 'edge_index' contains larger indices than the number of nodes (896) in 'DataBatch' (found 928)

So it seems I got where I missed, and it shows true instead of this where I used the Proteins dataset, Bravo! But it shows the same warning when I use data from the Proteins dataset in my read_x_data function. So I think it may be the data stored in a dataset are different from the data to be stored in a dataset right?
And for my latest test code, I wrote this in my read_x_data function:

python
dataset_ = TUDataset(os.path.join('data', args.dataset), name=args.dataset, use_node_attr=True)
    
data=Data(x=dataset_[i].x,edge_index=dataset_[i].edge_index,y = dataset_[i].y)

and this in my process function:

python 
dataset_ = TUDataset(os.path.join('data', args.dataset), name=args.dataset, use_node_attr=True)
for i in range(1,len(dataset_)):
    data_list.append(read_myds_data('/netp/test_1/', 'netp/test_1_/', 'my_file.npy', 1))

Thank you for your help!
And I would try if it fixes the problem afterward.

[Kevin-shihello-world]

But where does DataBatch defined?It seems it came from a build-in function in PYG

[rusty1s]

Yeah, it looks like your edge_index points to non-existing nodes, so fixing this data issue should resolve your errors. Note that DataBatch is dynamically created by inheriting from both Data and Batch. You can see it as a regular torch_geometric.data.Data object :)

[Kevin-shihello-world]

Thanks,I got it. When I changed to using HGL-sl code I forget to change the node feature number. So now I've changed my dataset into one with the same node feature as the Proteins dataset it works well. Cover my face.jpg