More detailed information guide when creating my own GNN?

[zcaicaros]

Dear communities and PyG creators,

I would like to create my own message-passing GNN with PyG, but have some trouble when doing so. I am appreciated any advice or information from the community. Many thanks in advance.

The problem I have:
I find it a bit unclear when creating my GNN. Specifically, the information provided in this tutorial is not enough for me to carry on the implementation. According to the tutorial, the only functions we need to implement are the message() (corresponding to phi function) and update() (corresponding to gamma function). Then, by defining which aggregation sum, mean, or max, the PyG will automatically compute the message for each node using message() and aggregate and update() the message accordingly.

My question is mainly about the message() and update() function. Actually the message() and update() can take some magic "internal" arguments which are created during the message passing flow, like the tutorial saying "In addition, tensors passed to propagate() can be mapped to the respective nodes and by appending _i or _j to the variable name, .e.g. x_i and x_j.". Great, for example in the GCN layer:

def message(self, x_j, norm):
        # x_j has shape [E, out_channels]

        # Step 4: Normalize node features.
        return norm.view(-1, 1) * x_j

x_j is passed to message() which is automatically computed during the message passing process (somewhere inside the propagate() function). But in the tutorial, it is not clear enough for such automatically-computed arguments, i.e., except for x_i and x_j, what are else such arguments? When are those arguments computed and where? From the example in GATv2_conv, I find another such argument size_i:

def message(self, x_j: Tensor, x_i: Tensor, edge_attr: OptTensor,
                index: Tensor, ptr: OptTensor,
                size_i: Optional[int]) -> Tensor:
        x = x_i + x_j

        if edge_attr is not None:
            if edge_attr.dim() == 1:
                edge_attr = edge_attr.view(-1, 1)
            assert self.lin_edge is not None
            edge_attr = self.lin_edge(edge_attr)
            edge_attr = edge_attr.view(-1, self.heads, self.out_channels)
            x += edge_attr

        x = F.leaky_relu(x, self.negative_slope)
        alpha = (x * self.att).sum(dim=-1)
        alpha = softmax(alpha, index, ptr, size_i)
        self._alpha = alpha
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        return x_j * alpha.unsqueeze(-1)

From this finding, I realize there are a lot of details going on inside the MassagePassing base class. If I am not clear with every detail about it, I am concerning the GNN model I write. This is important because for example if I already know x_j or x_i will be computed before calling message(), I will not have to compute them myself.

So is there any more detailed tutorial/docs for explaining exactly how the MassagePassing base class works?

[rusty1s]

Thanks for the issue. You are right that there are some hidden arguments that can be accessed which are never explained in the doc. I'm sorry for that.

In general, each node-level tensor can be mapped to source and destination edge-level tensors via *_j, and *_j, respectively.

Additional arguments that are accessible during message are (see here):

• edge_index. The original edge indices
• edge_index_i. The edge indices of destination nodes, i.e., edge_index[:, 1]
• edge_index_j. The edge indices of source nodes, i.e., edge_index[:, 0]
• size: A tuple (size_j, size_i) that describes the shape of the adjacency matrix. This will get automatically inferred if not explicitly passed to propagate.
• index: Corresponds to edge_index_i
• dim_size: Corresponds to size_i
• ptr and adj_t are accessible in case you pass a SparseTensor to propagate (see here), which stores edge indices in CSR format. Here, ptr denotes the compressed format of storing destination node indices.

[nastaranmarzban]

when i omit return x and alpha in def forward, and call a.forward(edge_index, x) in order to receive out, as an output , i see this error 'tuple' object has no attribute 'dim'

[rusty1s]

Hey @nastaranmarzban, do you have a minimal example to describe your issue? I'm happy to help, but it's currently unclear to me what you are trying to do. I'm sorry.

[nastaranmarzban]

no, not really
I just wanna understand what happen inside the code, so I use the tiny graph that I mentioned before......
I can get, out and alpha as outputs....
at first I've written the same code which is in source code and then written

a = GATConv(in_channels= 2, out_channels= 3, heads = 2)
print(a)
print("lin_src.weight:",a.lin_src.weight)
print("*"15)
print("lin_dst.weight:",a.lin_dst.weight)
print(""*15)
print("att_src:",a.att_src, "\n","att_dst:", a.att_dst)

x= torch.tensor([[1,2],[3,4],[5,6],[7,8],[9,10]],dtype=torch.float32)
edge_index = torch.tensor([[0,0,1,1,1,3],
[1,2,0,2,3,4]],dtype=torch.long)

print(a.forward(x, edge_index, return_attention_weights=True))
here I have out and alpha

but how can I see x_i, x_j alpha_i, alpha_j and the output of a.message()?this is my main problem I know these are computed automatically by self.propagate but I wanna see them in output. Is it possible?

[rusty1s]

Ah, got it. Those are intermediate results, and they are not part of the output. One thing you can do is look into our MessagePassing hooks, e.g. register_propagate_forward_hook. Hope this helps :)

[nastaranmarzban]

thank you .... I hope so, I'll test it

[nastaranmarzban]

Hi, sorry I have some questions in source code about GATConv:

1. Is the two below lines true?
   a) alpha_i = alpha[edge_index[1]] = x[col], in which alpha = alpha_dst---->is computed in def forward before propagation.
   and
   b) alpha_j = alpha[edge_index[0]] = x[row], in which alpha = alpha_src

2. what would happen if we produced att_src and att_dst equally? that is,
   self.att_src = self.att_dst = Parameter(torch.Tensor(1, heads, out_channel))

3)I got these things from the answer that you have written for the first time in this page, are they correct?
a) size = (N,M) in which N is num_nodes and M is num_messages.
b) index = edge _index_i = edge_index[1,:] or edge_index[:,1]--->I cannot understand why you have written like this!!

4. Imagine, in_channels = 2, out_channels = 3, heads = 2, num_nodes = 5 and edge_index = torch.tensor([[0,0,1,1,1,3],
   [1,2,0,2,3,4]],dtype=torch.long)
   what is the shape of x_j and alpha before unsqueegeeing in def message?what is the shape of each separately?

5. why do we use dropout?

I would be thankful if you answered me...I know there are many!!! so please answer me just if you have time. I don't want to bother you
thanks again

[rusty1s]

1. Yes, this is correct edge_index[0] refers to the source nodes (that send a message to destination nodes) and *_j.
2. The weights will be shared, i.e., there is no distinction of source and target node features and how attention coefficients are computed based on them
3. size refers to the size of the adjacency matrix (it can be usually omitted since we automatically try to infer it). In your case, it is (num_nodes, num_nodes). index = edge_index_i = edge_index[1, :] is correct (we aggregate along the index of target nodes)
4. x_i and x_j will have shape [num_edges, heads, out_channels]. alpha_i and alpha_j will have shape [num_edges, heads].
5. It helps to avoid overfitting and helps to explore more diverse paths computed from attention :)

[nastaranmarzban]

Thanks for your replying, everything is clear...thanks a lot
But yet I have a problem about number two,you mean we can write:
self.att_src = self.att_dst = Parameter(torch.Tensor(1, heads, out_channel))????
Am I right?
In the source code they are produced separately
self.att_src = Parameter(torch.Tensor(1, heads, out_channels))
self.att_dst = Parameter(torch.Tensor(1, heads, out_channels))

[rusty1s]

Yes, I just meant that it is possible to share those parameters (which will ignore the direction of edges), while the original GAT formulation (and our implementation) uses distinct parameters to compute the attention score.

[nastaranmarzban]

Thank you🙏🙏
got it🙂