Doubt about attention coefficient of gat

[dhorka]

Hi,

After checking the current implementation of GAT I have not been able to find the concatenation of [ Whi || Whj] described in the paper The original formulation says that:

What I saw is instead of doing a concatenation, the code is doing a summation in the message function alpha = alpha_j if alpha_i is None else alpha_j + alpha_i. Is there something that I am missing? I am asking that because I would like to change this concatenation for a dot product, but I was not able to find this concatenation step.

Thanks,

[rusty1s]

We concatenate implicitly, leading to a smaller memory footprint. In particular, we hold two versions of the a parameter vector, one for W@h_i (named att_r) and one for W@h_j (named att_l). We then multiply the source and destination node features with these parameters and sum the resulting parts together. This is equivalent to first concatenating source and destination node features, and multiply with a single attention parameter vector afterwards. Hope this is understandable :)

[dhorka]

Hi,

after checking the code that you have pointed I have not been able to find where the dot product is implemented. What I am trying to do is modify the function a(concat(Wx_i, Wx_j)) by dot(Wx_i, Wx_j). This modification is in the code that you have sent me? Do you think that you can give me some hint to how to implement this efficiently with the current GAT implementation?

Thanks!

[rusty1s]

Yes, this is in the code, i.e., line 265-268, in which attention coefficients are implemented via (x_i * x_j).sum(dim=-1). You can do the same in GATConv, where you simply replace the alpha computation in message via alpha = (x_i * x_j).sum(dim=-1) / math.sqrt(self.out_channels).

[dhorka]

I didn't realize that was the dot product. Sorry, but there is something that I do not understand why are you dividing the product for the sqrt of the number of channels?

[rusty1s]

This actually refers to the scaled dot-product attention, as introduced in the "Attention is All You Need" paper.

[dhorka]

Thanks! All my doubts have been clarified!