HeteroData for link prediction with directed and undirected edges

[lmy86263]

Dear communities and PyG team:

Thanks for the support and contributions first.

I want to use the heterogeneous graph to perform the link prediction task with PyG. Let me briefly describe the task I face first.
I have a heterogeneous graph, which is composed of two types of nodes: consumer and commodity.

The purposes:

1. Find the consumers who like this product most through the product?
2. Find another consumer with the same taste according to one consumer, even if there is no direct connection.

Heterogeneous graph:

• Node type: consumer, commodity
• Types of edges: (consumer, purchase, commodity) and (commodity, co-occur, commodity). The first one is directed and the other is undirected.

For the purpose of simplification, at present, each node does not have any attributes, but the edges (commodity, co-occur, commodity) have the attributes of co-occurrence times.

My questions

1. Do I have to use ToUndirected(data) to force my heterogeneous graph to be undirected, because I see the operation in hetero_link_pred.py (the example helps a lot 😁 ). But there is only a directed graph in its structure, and there is no undirected co-occurrence relation, so I have to use this operation?

2. For the processing of connections between nodes of the same type in ToUndirected() and their role in RandomLinkSplit, my code is as follows:

   transform = RandomLinkSplit(
           num_val=0.05,
           num_test=0.1,
           neg_sampling_ratio=0.0,
           edge_types=[('customer', 'purchase', 'commodity'), ('commodity', 'co-occur', 'commodity')],
           rev_edge_types=[('commodity', 'rev_purchase', 'customer'), ('commodity', 'co-occur', 'commodity')],
       )
   

   How to pass in edge_types and rev_edge_types？

3. There are no connections bwtween user nodes in MovieLens dataset in hetero_link_pred.py. it needs to be learned through embedding, but I am confused with the operations as follows:

   # Add user node features for message passing:
   data['user'].x = torch.eye(data['user'].num_nodes, device=device)
   del data['user'].num_nodes
   

No connections between users and there is no need for message passing. So what's the role of the identity matrix( just for self loop?) And why delete num_nodes, is this necessary for link prediction?

Thanks!

[rusty1s]

1. It's always good to ensure that reverse edges exist for message passing. In your case, this guarantees that a customer sends a message to a commodity as well. Note that this is a no-op for edges that are already undirected.
2. This is correct, although this will split both edge types. Dependent on the task, it might be better to train two individual models instead of a single one.
3. The del data['user'].num_nodes is not that important, and is simply removed since the number of nodes can now be inferred by the node feature matrix. Note that this matrix does not denote an adjacency matrix, it denotes a feature matrix. Each user receives a unique feature.

[CocoGzh]

Yes, let's continue our discussion here: #3682

Thx for your reply.
We will discuss all the probelm on the "metapath2vec" dataset in OGB_MAG.

1, I find the problem and it may be a bug or not.
If we do not use T.ToUndirected() to process the dataset as the official tutorial (https://pytorch-geometric.readthedocs.io/en/latest/notes/heterogeneous.html). Then we construct the heterogeneous model:
model = GAT(hidden_channels=2, out_channels=1)
model = to_hetero(model, data.metadata(), aggr='sum')
out = model(data.x_dict, data.edge_index_dict)

There will raise an error:
The traceback is:
Traceback (most recent call last):
File "../example_Heterogeneous_1.py", line 68, in
out = model(data.x_dict, data.edge_index_dict)
File "..\anaconda3\lib\site-packages\torch\fx\graph_module.py", line 513, in wrapped_call
raise e.with_traceback(None)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'Tensor'

Specifically, it can be said: if there is a directed heterogeneous graph without self loops, and some kinds of nodes (here is "author") receive no message from other kinds of nodes. The type of received message of these nodes (here is "author") will be "None" and it cannot be added to the original node features ("tensor") or other kinds of "tensor features".

2. I follow the the tutorials "Using Heterogenous Convolution Wrapper" in https://pytorch-geometric.readthedocs.io/en/latest/notes/heterogeneous.html. It always raise an error "AttributeError: 'tuple' object has no attribute 'size"
   Can you offer a complete example of "Using Heterogenous Convolution Wrapper"? Thx!

[rusty1s]

1. Yes, the node embeddings of node types that do not receive any update during propagation will be None. There's currently no way to "fix" that in to_hetero(model). It's unclear to me what should really happen in this case.

2. Here is a full example of HeteroWrapper. The problem may be that you are trying to use GCNConv, while GCNConv cannot handle bipartite message passing flow (it can only be used to send messages to the same node type).

[lmy86263]

1. The edge type ('commodity', 'rev_co-occur', 'commodity') will only be created if you set the merge flag in ToUndirected to False, see here. Otherwise it will merge reverse edges to the same edge type. Note that hetero_data.is_undirected will currently always return False for HeteroData objects. This is still a TODO, see here.
2. I'm not sure what you mean. In general, I don't think it makes much sense to train a single classifier for different edge types.
3. Note that torch.randn is equivalent to an Embedding layer that does not get trained. Hard to say which works better in your case :)

Thx,

• Question 1: The reply solved my question completely.

2. Now I try to explore to train a classifier for link prediction task. The data is as mentioned above(with labels modification). I think it may be better with codes.
   My model depends the examples here but is modifies a little as shown.

class GNNEncoder(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x


class EdgeDecoder(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        # self.lin1 = Linear(2 * hidden_channels, hidden_channels)
        # self.lin2 = Linear(hidden_channels, 1)

    def forward(self, z_in, edge_label_index):
        row, col = edge_label_index
        # Method 1
        z_out = (z_in[row] * z_in[col]).sum(dim=-1)

        # Method 2
        # z_out = torch.cat([z_in[row], z_in[col]], dim=-1)
        # z_out = self.lin1(z_out).relu()
        # z_out = self.lin2(z_out)
        # z_out = z_out.view(-1)
        return z_out


class HeteroModel(torch.nn.Module):
    def __init__(self, hidden_channels, data):
        super().__init__()
        self.encoder = GNNEncoder(hidden_channels, hidden_channels)
        self.encoder = to_hetero(self.encoder, data.metadata(), aggr='sum')

        self.decoder = EdgeDecoder(hidden_channels)
        self.decoder = to_hetero(self.decoder, data.metadata(), aggr='sum')

    def forward(self, x_dict, edge_index_dict, edge_label_index):
        z_dict = self.encoder(x_dict, edge_index_dict)
        return self.decoder(z_dict, edge_label_index)

the big difference is that the decoder is wrapped with to_hetero(). But the result shows that:

• the encoder can work: it generates tensor for nodes and links with different types.
• the decoder doesn't work: The error is shown as follows:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/torch/fx/graph_module.py", line 608, in wrapped_call
return super(type(self), self).call(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in call_impl
return forward_call(*input, **kwargs)
File "<eval_with_key>.3", line 12, in forward
getitem__concept__rev_has_ consumer = edge_label_index__concept__rev_has__ consumer[0]
TypeError: 'NoneType' object is not subscriptable

Call using an FX-traced Module, line 12 of the traced Module's generated forward function:
getitem__concept__coc__concept = edge_label_index__concept__coc__concept[0]
getitem__concept__rev_has__ consumer = edge_label_index__concept__rev_has__ consumer[0]

    getitem_1__consumer__has__concept = edge_label_index__consumer__has__concept[1];  edge_label_index__consumer__has__concept = None

    getitem_1__concept__coc__concept = edge_label_index__concept__coc__concept[1];  edge_label_index__concept__coc__concept = None

Traceback (most recent call last):
  File "/Users/lmy86263/Google Drive/Colab Notebooks/QueryExpand/Graph/gnn/main.py", line 108, in <module>
    loss = train(train_data, weight)
  File "/Users/lmy86263/Google Drive/Colab Notebooks/QueryExpand/Graph/gnn/main.py", line 23, in train
    pred = model(train_data.x_dict, train_data.edge_index_dict,
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/lmy86263/Google Drive/Colab Notebooks/QueryExpand/Graph/gnn/HeteroConverter.py", line 53, in forward
    return self.decoder(z_dict, edge_label_index)
  File "/usr/local/lib/python3.8/site-packages/torch/fx/graph_module.py", line 616, in wrapped_call
    raise e.with_traceback(None)
TypeError: 'NoneType' object is not subscriptable

It seems to be that the to_hetero() can't duplicates to work on each edge type individually when using symbolic tracer. So I want to know there are any problems with my application or not.
And, my candidate solutions are in this scenario:

• we should handle each link type in the decoder individually instead of using to_hetero()?
• or change to another way, break this graph into two subgraphs?
  • HeteroModel1 handles such as consumer-concept connections ;
  • Another model handles concept-concept connections which can be a homogenous GNN model. However, this way may block some message passing.

3. To summarize, there are many ways for feature parameters initialization for nodes without any actual features, such Embeddings in the model, torch.randn(), torch.eyes with one-hot matrix (which may be unfriendly to million of nodes). Although these ways are equivalent, it is hard to say which works better for specific models. Am I right?

[CocoGzh]

1, This is really a special scene. I'll consider adding self loops.
2, It is useful. Thx.

[rusty1s]

@lmy86263 The reason the decoder does not work is that it now expects dictionaries as input (e.g., edge_label_index for each edge type), while you only input a single tensor into it. I don't think there is a reason to use to_hetero in the decoder part. What about something like:

class EdgeDecoder(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()

    def forward(self, z_src, z_dst, edge_label_index):
        row, col = edge_label_index
        # Method 1
        z_out = (z_src[row] * z_dst[col]).sum(dim=-1)
        return z_out

class HeteroModel(torch.nn.Module):
    def __init__(self, hidden_channels, data):
        super().__init__()
        self.encoder = GNNEncoder(hidden_channels, hidden_channels)
        self.encoder = to_hetero(self.encoder, data.metadata(), aggr='sum')

        self.decoder = EdgeDecoder(hidden_channels)

    def forward(self, x_dict, edge_index_dict, edge_label_index):
        z_dict = self.encoder(x_dict, edge_index_dict)
        return self.decoder(z_dict[src_node_type], z_dict[dst_node_type], edge_label_index)