GAT that also incorporates edge_weight

[adhikarirsr]

Is there a GATConv like GNN that uses edge_weight to learn better representation for nodes?
Does RGAT (Relational GAT ) take edge_weight or edge_attr into account? Is there an implementation of RGAT in PyG? Can we use edge_weight as edge_attr?

RGAT is a generalization GAT, so I think it a simple extension of GAT should help us have RGAT in PyG, right? If so then how can I go about doing so? If there is a tutorial to do so, then we can implement bunch of new GNNs in PYG.

[rusty1s]

In GATConv , the edge_weight is determined via attention. We follow the official formulation here. If you want to make use of additional edge_weight argument, you need to extend the GATConv to incorporate this into the attention coefficient calculation.

There doesn't exist a tutorial yet on how to convert GNN operators into relational ones, but this reply should get you going. At one point, I will write a wrapper to make this more convenient.

[haoming-codes]

Thank you @rusty1s ! If I am interested in using both edge_weight and edge_attr, between the two options:

1. Multiplying edge_attr by edge_weight at the start of Conv.forward()
2. Multiplying message by edge_weight at the end of Conv.message()

Which one is preferred or more universally applicable for different Conv? I am worried that doing this may not make sense for certain convolutions (although I am not sure which would be an example). Or is it that the role of edge_attr and edge_weight in different Conv are designed to be consistent, so that either of these options will always be correct and equivalent? Thank you!

[rusty1s]

I don't think either of this is necessary. If you add the edge weights to the edge_attr, then the neural network can learn to interpret this as an edge weight on its own.

[haoming-codes]

Thank you @rusty1s . Correct me if I'm wrong, but I believe in my case, the edge_weight is actually the adjacency matrix, edge_attr is the edge features , and edge_index is the edges in a complete graph. I need edge_weight to function like an adjacency matrix, e.g. to determine the edges along which the messages are passed or the attentions are paid (or, in the case that the adjacency matrix are edge probabilities, the weight of messages or attention). Similar to the role of edge_weight in GCNConv.

Shouldn't the edge_weight multiply something, as opposed to being concatenated with edge_attr?

[haoming-codes]

class WeightedGATConv(GATConv):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def forward(..., edge_weight = None):
        ...
        out = self.propagate(..., edge_weight=edge_weight)
        ...
    def message(..., edge_weight = None):
        ...
        alpha = edge_weight.view(-1, 1) * alpha
        for i in range(10): # normalize attention
            alpha[index == i] = alpha[index == i] / alpha[index == i].sum()
        return x_j * alpha.unsqueeze(-1)

nnode = 10
out_channels = 4
heads = 1
nfeat = 6
efeat = 7

G = nx.random_partition_graph([nnode//2, nnode//2], .9, .1)
CG = nx.complete_graph(nnode)
nattrs = {v: np.random.randn(nfeat) for v in CG.nodes}
eattrs = {(u, v): np.random.randn(efeat) for u, v in CG.edges}
nx.set_node_attributes(G, nattrs, "block_one_hot")
nx.set_node_attributes(CG, nattrs, "block_one_hot")
nx.set_edge_attributes(G, eattrs, "block_edge_one_hot")
nx.set_edge_attributes(CG, eattrs, "block_edge_one_hot")
data = from_networkx(G, group_node_attrs=["block_one_hot"], group_edge_attrs=["block_edge_one_hot"])
dataC = from_networkx(CG, group_node_attrs=["block_one_hot"], group_edge_attrs=["block_edge_one_hot"])
adj_sparse = dense_to_sparse(to_dense_adj(data.edge_index)+(1-torch.eye(nnode)))
dataC.edge_weight = adj_sparse[1]-1
data.x = data.x.float()
dataC.x = dataC.x.float()
data.edge_attr = data.edge_attr.float()
dataC.edge_attr = dataC.edge_attr.float()
dataC.edge_weight = dataC.edge_weight.float()

layer = GCNConv(in_channels=nfeat, out_channels=out_channels, add_self_loops=False)
out1 = layer(x=data.x, edge_index=data.edge_index)
out2 = layer(x=dataC.x, edge_index=dataC.edge_index, edge_weight=dataC.edge_weight)

layer = WeightedGATConv(in_channels=nfeat, out_channels=out_channels, heads=heads, edge_dim=efeat, add_self_loops=False)
out3 = layer(x=data.x, edge_index=data.edge_index, edge_attr=data.edge_attr)
out4 = layer(x=dataC.x, edge_index=dataC.edge_index, edge_attr=dataC.edge_attr, edge_weight=dataC.edge_weight)

Essentially, dataC is a complete graph, whose edge weights are the adj matrix of data.
Here, we'd expect out1 == out2 and out3 == out4. Does this look like a correct implementation?

[rusty1s]

Edge weights do not necessarily need to be used directly to weight the messages. In case of GATConv, using edge weights as edge features will actually drive the computation of attention weights, i.e., the model will take in the original edge weight to determine the attention score. I think this is exactly what you want in this case.

[dhall1995]

I've implemented an edge weighted GAT conv here:

https://github.com/dhall1995/GrapHiC/blob/master/GrapHiC/layers/GATEConv.py

for a project of mine. Example usage in a net being found here (although the train.py script unfortunately can't be run at the minute the layers do propagate information correctly in testing):

https://github.com/dhall1995/GrapHiC-ML/blob/master/train.py

This implements the following convolution:

with the top equations being the original GAT convolution and the bottom layers being my edge weighted version. The linearly transformed edge features are also output to the next layer.

[ZRH0308]

Hello, Is there any progress on the paper mentioned here?

[rusty1s]

GATConv now has edge_weight/edge_attr support in PyG :)

[ZRH0308]

Oh! Hey! @rusty1s Thank you for your reply! Yes, I have implemented GAT with edge_attr in my task! :)

And... by the way, I also need to train GAT in an unsupervised way like GraphSAGE https://github.com/pyg-team/pytorch_geometric/blob/b57489fd759273a4a7ba4a9613fa87831daa937d/examples/graph_sage_unsup.py, is there any problem if i do this?

[dhall1995]

Hello! Apologies for the late reply - that publication I was referring to ended up being mothballed while my funders wanted me to focus on something else. Aiming to one day get back to it...

However as pointed out - the edge weighted GAT conv got put in PyG after this post!

[ZRH0308]

Hello! Apologies for the late reply - that publication I was referring to ended up being mothballed while my funders wanted me to focus on something else. Aiming to one day get back to it...

However as pointed out - the edge weighted GAT conv got put in PyG after this post!

Still thanks for inspiring me with your ideas!