integer multiplication overflow when adding self loops

[JosephGatto]

I am using GCNConv and get the following error randomly mid-way through training

 79     if add_self_loops:
---> 80         edge_index, tmp_edge_weight = add_remaining_self_loops(
     81             edge_index, edge_weight, fill_value, num_nodes)
     82         assert tmp_edge_weight is not None

/usr/local/lib/python3.8/dist-packages/torch_geometric/utils/loop.py in add_remaining_self_loops(edge_index, edge_attr, fill_value, num_nodes)
    324         loop_attr[edge_index[0][inv_mask]] = edge_attr[inv_mask]
    325 
--> 326         edge_attr = torch.cat([edge_attr[mask], loop_attr], dim=0)
    327 
    328     edge_index = torch.cat([edge_index[:, mask], loop_index], dim=1)

RuntimeError: numel: integer multiplication overflow

Any ideas why this might happen?

[rusty1s]

Never seen this before. Can you share the mini-batch for which this is happening?

[JosephGatto]

Sure, do you just want to see the output of printing the mini batch (which would just show shapes of everything) or the actual values in the samples?

I have printed this value before and the values in x get extremely huge and don't match the original input. E.g. if I do a try-except around my forward pass through the network, the value of x[0], which is supposed to be some 2-5 digit number, produces something crazy like 98743587594 in all the x elements when the model fails.

[rusty1s]

Interesting. Have you considered some normalization as part of your model to counteract that?

[JosephGatto]

I have! My current flow is:

Tokenize node embeddings using BERT -> Pass the BERT embeddings through Layer Norm -> Pass the normed embeddings through a GCNConv layer.

self.bert_embed = AutoModel.from_pretrained(MODEL_NAME)
self.emb_dim = self.bert_embed.embeddings.word_embeddings.weight.shape[1] # Get BERT Embedding shape 
self.max_length = 100 #Pre-defined max sentence length 

self.gc = GCNConv(self.emb_dim, self.emb_dim)
self.norm = LayerNorm(self.emb_dim)

def forward_single(self, data):
        
        node_embeddings, _, attention_mask = self.embed(data) # Run data through BERT 
        
        normed_input = self.norm(node_embeddings)
        graph_encoded_nodes  = self.gc(normed_input.view(-1,self.emb_dim), edge_index)
        pooled_nodes = self.pool(graph_encoded_nodes.reshape(node_embeddings.shape), attention_mask.view(-1, self.max_length))
        return pooled_nodes

I do this for multiple graphs and pass it through a variant of this loss.

• I have tried batching the data using the follow_batch method described in the documentation as well as just zipping multiple data loaders.

• If I comment out the forward pass through the GCN and just use the BERT embeddings, it runs fine.

• I have ensured all of the nodes in the edge_index are legit (not outside bounds)

• I have even since your last message tried passing identity matrices to the model and it still breaks, providing a less-useful error:

---> 16         outputs = model(g1, g2)
     17 
     18         loss = criterion(outputs)

10 frames

/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py in dropout(input, p, training, inplace)
   1250     if p < 0.0 or p > 1.0:
   1251         raise ValueError("dropout probability has to be between 0 and 1, " "but got {}".format(p))
-> 1252     return _VF.dropout_(input, p, training) if inplace else _VF.dropout(input, p, training)
   1253 
   1254 

RuntimeError: philox_cuda_state for an unexpected CUDA generator used during capture. In regions captured by CUDA graphs, you may only use the default CUDA RNG generator on the device that's current when capture begins. If you need a non-default (user-supplied) generator, or a generator on another device, please file an issue.

[rusty1s]

Thanks for digging deeper into this. This is still a bit mysterious to me though. Does it work when running on CPU? Still interested in a mini-batch sampler of the inputs of self.gc(normed_input.view(-1,self.emb_dim), edge_index) so that I can take a look as well.

[JosephGatto]

I finally debugged this. Long story short: The graph derived from the text was derived before truncating the text to fit into BERTs token limit. Thus, there were connections in edge_index that went outside the number of nodes that make it into X. Thank you for the help!