Question about message function on the GAT conv layer

[cjissmart]

1. I am confused about the index and the ptr parameter in the message function of the GATConv layer. Following Questions about propagate function and message function on the GAT conv layer #1949, the index means the edge_index[1] and the ptr is about the CSR format. If I softmax alpha ( attention value) according to edge_index[0] and the type of edge_index is SparseTensor, how do I modify the message function?

    def message(self, x_j: Tensor, alpha_j: Tensor, alpha_i: OptTensor, index: Tensor, ptr: OptTensor, size_i: Optional[int]) -> Tensor:
        ......
        alpha = softmax(alpha, index, ptr, size_i)

2. I am confused about the code of the softmax function.

        out = src - scatter(src, index, dim=0, dim_size=N, reduce='max')[index]
        out = out.exp()
        out_sum = scatter(out, index, dim=0, dim_size=N, reduce='sum')[index]
        return out / (out_sum + 1e-16)

Why does src first need to subtract a specific value? And the value would be about off by out / (out_sum + 1e-16).

[rusty1s]

1. There is no need to modify message when using SparseTensor: index and ptr will be automatically filled in this case.
2. This increases numerical stability of the softmax function (softmax is translation invariant). For example, inputs may be really large, and exponential increasing such values leads to instabilities. The subtraction via out.max() is a common trick to prevent this.

[cjissmart]

Thanks for your reply very much! Because index and ptr will be automatically filled in this case, the index is the edge_index[0]. Now I want to execute this code:

alpha = softmax(alpha, edge_index[0], ptr, size_i)

How can I get edge_index[0] when the type of edge_index is SparseTensor. I try to add a suffix to edge_index edge_index_j. But I get an error.

[rusty1s]

def message(self, edge_index_j, ...) should work. Can you show me the exact error?

[cjissmart]

Thanks very much! def message(self, edge_index_j, ...) works. I ignore something.