heterogeneous graph attention network training error:torch.ops.torch_scatter.scatter_max

[Zhang-1218]

Hello！
I'm new to graph network design and creation and would appreciate some guidance if possible.
First, are there any specific model examples to study in the field of bipartite graph node classification?

I am struggling to find any example implementation. Especially those with edge features (edge_attr).

I created a preliminary HeteroData graph，I'm utilizing Heterogeneous Graph Attention Network for training on GPU，and the following error is generated during training
ERROR：

Traceback (most recent call last): 
  File "domain_train.py", line 78, in <module>
    train()
  File "domain_train.py", line 71, in train
    HeteGatModel(dataset, args)
  File "/alldata/heteGat.py", line 92, in HeteGatModel
    train_loss, train_acc, val_loss, val_acc = hetegat_train(hgtmodel, optimizer, data)
  File "/alldata/heteGat.py", line 49, in hetegat_train
    out = hgtmodel(data.x_dict, data.edge_index_dict)
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch/fx/graph_module.py", line 308, in wrapped_call
    return cls_call(self, *args, **kwargs)
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch/fx/graph_module.py", line 308, in wrapped_call
    return cls_call(self, *args, **kwargs)
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "<eval_with_key_1>", line 7, in forward
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch_geometric/nn/conv/gat_conv.py", line 218, in forward
    out = self.propagate(edge_index, x=x, alpha=alpha, edge_attr=edge_attr,
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch_geometric/nn/conv/message_passing.py", line 309, in propagate
    out = self.message(**msg_kwargs)
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch_geometric/nn/conv/gat_conv.py", line 258, in message
    alpha = softmax(alpha, index, ptr, size_i)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch_scatter/scatter.py", line 161, in softmax
        return scatter_min(src, index, dim, out, dim_size)[0]
    elif reduce == 'max':
        return scatter_max(src, index, dim, out, dim_size)[0]
               ~~~~~~~~~~~ <--- HERE
    else:
        raise ValueError
  File "/root/anaconda3/envs/hang/lib/python3.8/site-packages/torch_scatter/scatter.py", line 73, in scatter_max
        out: Optional[torch.Tensor] = None,
        dim_size: Optional[int] = None) -> Tuple[torch.Tensor, torch.Tensor]:
    return torch.ops.torch_scatter.scatter_max(src, index, dim, out, dim_size)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
RuntimeError: CUDA error: device-side assert triggered

Any guidance would be greatly appreciated.

Thanks:)!

[Zhang-1218]

It's really simple. The two types of nodes are cities and villages.

Edge features are multidimensional feature matrices

I have tried various GATConv implementations without success, but I know I may be missing the basics.

Below are the details of the data I use：

HeteroData(
  city={
    x=[7198, 32],
    y=[7198],
    num_nodes=7198,
    train_mask=[7198],
    val_mask=[7198],
    test_mask=[7198]
  },
  country={
    x=[4177, 32],
    y=[4177],
    num_nodes=4177
  },
  (city, have, country)={
    edge_index=[2, 39459],
    edge_attr=[39459, 3]
  },
  (country, rev_have, city)={
    edge_index=[2, 39459],
    edge_attr=[39459, 3]
  }
)

I used the model of the official website example， this is model:

class HeteGAT(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels, heads=3):
        super().__init__()

        self.hidden_channels = hidden_channels
        self.out_channels = out_channels
        self.heads = heads

        self.conv1 = GATConv((-1, -1), hidden_channels, heads=heads, add_self_loops=False)
        self.lin1 = Linear(-1, hidden_channels)
        self.conv2 = GATConv((-1, -1), out_channels, heads=heads, add_self_loops=False)
        self.lin2 = Linear(-1, out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index) + self.lin1(x)
        x = x.relu()
        x = self.conv2(x, edge_index) + self.lin2(x)
        return x

The following is a more detailed training module code:

hgmodel = HeteGAT(hidden_channels=args.hidden, out_channels=dataset.num_classes, heads=args.heads)
hgtmodel = to_hetero(hgmodel, data.metadata(), aggr='sum').to(device)
optimizer = torch.optim.Adam(hgtmodel.parameters(), lr=args.lr)
hgtmodel.train()
optimizer.zero_grad()
out = hgtmodel(data.x_dict, data.edge_index_dict)
trainmask = data['city'].train_mask
loss_train = F.cross_entropy(out['city'][trainmask], data['city'].y[trainmask])
acc_train = accuracy(out['city'][trainmask], data['city'].y[trainmask])
loss_train.backward()
optimizer.step()

Any guidance would be greatly appreciated.

Thanks!

[rusty1s]

The error suggests that there exists a problem with your torch-scatter installation. Can you give some information on how you installed torch-scatter, and (if accessible) post the installation log here?

[Zhang-1218]

Thank you very much for your reply！
I am running the program in the conda virtual environment, and I installed it through the offline version of conda downloaded from the pyg official website
Here is the version info I have installed:

pytorch                   1.8.0           py3.8_cuda10.1_cudnn7.6.3_0    pytorch
pytorch-cluster           1.5.9           py38_torch_1.8.0_cu101         pyg
pytorch-scatter           2.0.8           py38_torch_1.8.0_cu101         pyg
pytorch-sparse            0.6.12          py38_torch_1.8.0_cu101         pyg
pytorch-spline-conv       1.2.1           py38_torch_1.8.0_cu101         pyg
torch-geometric           2.0.2                    pypi_0                pypi
torchaudio                0.8.0                      py38                pytorch
torchvision               0.9.0                py38_cu101                pytorch

[Zhang-1218]

Additional relevant information：
python Version: 3.8.12
CUDA Version: 10.1

[rusty1s]

Thanks. Does the following code works for you?

import torch
from torch_scatter import scatter_max

src = torch.tensor([[2, 0, 1, 4, 3], [0, 2, 1, 3, 4]]).cuda()
index = torch.tensor([[4, 5, 4, 2, 3], [0, 0, 2, 2, 1]]).cuda()

out, argmax = scatter_max(src, index, dim=-1)

If yes, then this error might be related to some wrong/large indices contained in one of the edge_index tensors.

[Zhang-1218]

Whoops, this test case runs successfully, so it might be an issue with the edge indexing of the data I'm dealing with?

(hang) [root@webserve GCNProject]# cat scatest.py 
import torch
from torch_scatter import scatter_max

src = torch.tensor([[2, 0, 1, 4, 3], [0, 2, 1, 3, 4]]).cuda()
index = torch.tensor([[4, 5, 4, 2, 3], [0, 0, 2, 2, 1]]).cuda()

out, argmax = scatter_max(src, index, dim=-1)
print(out)
print(argmax)
(hang) [root@webserve GCNProject]# python3 scatest.py 
tensor([[0, 0, 4, 3, 2, 0],
        [2, 4, 3, 0, 0, 0]], device='cuda:0')
tensor([[5, 5, 3, 4, 0, 1],
        [1, 4, 3, 5, 5, 5]], device='cuda:0')
(hang) [root@webserve GCNProject]# 

[rusty1s]

Yeah, you should check that edge_index[0].max() < x_src.size(0) and edge_index[1].max() < x_dst.size(0) for every edge type in your HeteroData object.

[Zhang-1218]

Omg, there is really a problem with my data, the source node in the edge index is out of bounds

Thanks a lot for your answer. After verification, there is indeed a problem with my edge index.
After the correction, the program can run normally.

Finally, allow me to thank you again for your answer!