TransformerConv RuntimeError: Not compiled with CUDA support

[roshandhakal]

I can easily train my model with TransformerConv with my cpu. However, when gpu is enabled, it throws me this error. I don't know if its issue with my machine or TransformerConv itself. Can someone please help? Full traceback of the error:

`Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/envprep/scripts/gcn_train.py", line 164, in
main(args)
File "/usr/local/lib/python3.8/dist-packages/envprep/scripts/gcn_train.py", line 63, in main
out = model.forward(batch, device)
File "/usr/local/lib/python3.8/dist-packages/envprep/models/prepare_env_gcn.py", line 22, in forward
x = self.conv1(x, edge_index, edge_attr)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch_geometric/nn/conv/transformer_conv.py", line 176, in forward
out = self.propagate(edge_index, query=query, key=key, value=value,
File "/usr/local/lib/python3.8/dist-packages/torch_geometric/nn/conv/message_passing.py", line 317, in propagate
out = self.message(**msg_kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch_geometric/nn/conv/transformer_conv.py", line 216, in message
alpha = softmax(alpha, index, ptr, size_i)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/torch_scatter/scatter.py", line 160, in softmax
return scatter_min(src, index, dim, out, dim_size)[0]
elif reduce == 'max':
return scatter_max(src, index, dim, out, dim_size)[0]
~~~~~~~~~~~ <--- HERE
else:
raise ValueError
File "/usr/local/lib/python3.8/dist-packages/torch_scatter/scatter.py", line 72, in scatter_max
out: Optional[torch.Tensor] = None,
dim_size: Optional[int] = None) -> Tuple[torch.Tensor, torch.Tensor]:
return torch.ops.torch_scatter.scatter_max(src, index, dim, out, dim_size)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
RuntimeError: Not compiled with CUDA support

Training Device: cuda`

[rusty1s]

It looks like torch-scatter is not compiled with CUDA support. Can you share some more information about how you installed torch-scatter. Can you try to re-install it via pip install --verbose as well?

[roshandhakal]

I thing I noticed strange was; I trained the model on gpu and generated a network file after training. Now, I want to use the file, but I get the error on same thing. Now, the training has the error too. I am wondering if its a error caused by gpu?

[roshandhakal]

I am getting this error with explicit versions:

[rusty1s]

Is the previous issue resolved? For the second one, it would be good to have some minimal reproducible script.

If you are using GCNConv, ensure that any edge weights are positive, as otherwise the output of it may include NaNs.

[roshandhakal]

I am sorry for my delayed response. I was away due to spring break. The thing is everything worked fine earlier, both TransformerConv and GCNConv with gpu, no issues, not even error I showed in the discussion. Suddenly, it started showing all these errors with gpu. However, it works with cpu. Both TransformerConv and GCNConv. I honestly don't know how to go about this.

For GCNConv, there are no edge weights anyway. And, no previous issue with TransformerConv is not resolved.

[rusty1s]

You can do two things to spot the issue:

• Run your script with CUDA_LAUNCH_BLOCKING=1 (which disables asynchronous CUDA execution)
• Make use of torch.autograd.detect_anomaly to detect the causes of NaNs