CUDA error: device-side assert triggered and modelling a Dependency Parse

[afonso-sousa]

I am trying to use GNNs to learn a representation of a dependency parse. I have the following data batch: DataBatch(x=[652, 1], edge_index=[2, 652], batch=[652], ptr=[33]), where each node has a single feature (the corresponding word tokenized). The edge indices correspond to a Levy Graph (the relation types were transformed to nodes so the graph has no edge types, for simplicity). I have a simple network borrowed from a tutorial with a global pooling layer at the end to have a representation per batch.

class GCNEncoder(nn.Module):
    def __init__(self, num_features, hidden_size, dropout=0):
        super().__init__()
        self.conv1 = GCNConv(num_features, hidden_size)
        self.conv2 = GCNConv(hidden_size, hidden_size)
        self.relu = nn.ReLU(inplace=True)
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x = self.conv1(x, edge_index)
        x = self.relu(x)
        x = self.dropout(x)
        x = self.conv2(x, edge_index)
        x = self.relu(x)
        x = global_add_pool(x, data.batch, size=data.num_graphs)
        return x

I am getting RuntimeError: CUDA error: device-side assert triggered. I do not compute a loss so what do you think may be causing the problem? Thanks in advance for any help you can provide.

PS: Additionally, this problem would be better suited for a RGAT, but I did not figure out what was "edge type" supposed to be.

[EdisonLeeeee]

I think the problem might be x = global_add_pool(x, data.batch, size=data.num_graphs). Can you try this instead?

x = global_add_pool(x, data.batch)

[afonso-sousa]

It did not work. Thanks for the reply anyways.

[rusty1s]

Can you run your code via CUDA_LAUNCH_BLOCKING=1 python ...? In which module does your script crash?

[afonso-sousa]

Can the issue have something to do with the actual values?
In every batch sample, my graph is a dependency parse in which I tokenize the words, so my x has integer values, like: tensor([[2.2640e+03],
[2.7091e+04],
[7.4240e+03],
[1.0000e+02],
[1.6111e+04],
[5.6000e+02],
[3.7362e+04],
[4.7830e+03],...

My edge_index always starts with 0, like:
tensor([[ 1, 11, 4, ..., 644, 627, 645],
[ 11, 0, 12, ..., 628, 645, 629]])

Do you see anything wrong with that?

[rusty1s]

This is weird. Can you also try to run data.validate() and see if it reports any errors? Other than that, you should consider normalizing your input features. These large values are not really suitable for ML tasks:

x = (x - x.mean(dim=0, keepdim=True)) / x.std(dim=0, keepdim=True)

[afonso-sousa]

Trying to print data.validate() throws AttributeError: 'GlobalStorage' object has no attribute 'validate'. My batching is done with PyGDataLoader() and inside my torch.utils.data.Dataset I have a graph per sample created using torch_geometric.data.Data.
My batch prints as follows:
DataBatch(x=[652, 1], edge_index=[2, 652], batch=[652], ptr=[33])

[EdisonLeeeee]

It looks like you are using an old version of PyG. Can you upgrade your PyG to the latest version and try it again? Perhaps the bug has been fixed :) Otherwise you can run data.validate() again to see if it reports any errors.

[afonso-sousa]

Upgrading the version did the trick. It showed a more specific error message and I figured that somehow I had an edge index of -1. Thank you very much for the help.