The bias of official implementation in GATConv and GATv2Conv

[hatute]

After taking a deep look at the source code of GATConv and GATv2Conv, although the difference between them is limited, I have a common concern to the implementation of these two conv layers.

Original ideas

Paper 1

From the original paper of GATConv, "GRAPH ATTENTION NETWORKS" by Velickovic et al. We can see that double vertical lines denote for "Concatenation".

Paper 2

From the original paper of GATv2Conv, "HOW ATTENTIVE ARE GRAPH ATTENTION NETWORKS?" by Brody et al. We can see that they means "Concatenation" as well.

Implementation

Paper 1

In the code of GATConv,
The first line inside method edge_update() is

# Given edge-level attention coefficients for source and target nodes,
# we simply need to sum them up to "emulate" concatenation:
alpha = alpha_j if alpha_i is None else alpha_j + alpha_i

Paper 2

In the code of GATv2Conv,
The first line inside method message() is

 x = x_i + x_j

Questions

So in essence, both of the methods replace the concatenation with the addition. Mathematically, these two operations are not equivalent. Could you guys give me more information about the correctness and precision behind this?
If I miss something, please let me know ;) Thank you.

[wsad1]

a.[k||h] = a.k + a.h. The implementation is using this equivalence to convert concatentation and then dot product TO dot product and addition.
So the formula from the paper and the code in pyg achieve the same thing.

[rusty1s]

Indeed. We first apply the transformation and then add the results together, which is equivalent to first concatenating the embeddings and then applying the transformation. The former is more efficient since it avoids creating the concatenated intermediate embedding.

[hatute]

Thank you guys for the explanation. It makes perfect sense.