Formula of Graph Attention Networks with Edge Features

[stefanschutz]

Hi, I'm using Graph Attention Networks (GAT), and I saw the implementation here.

The calculation of attention coefficients is done by summing up parts of the node representations instead of concatenation. I read that this helps to avoid the memory consumption issue..

My question is: how do you write the math formula that involves the concatenation of edge features? From the implementation code, it seems that the weight matrix and attention vector for the edge feature has different shape compared to the two used for node representations.

Original:

$$\alpha_{ij} = \text{Softmax}(\pmb{a}^\top [\pmb{W}h_i || \pmb{W}h_j])$$

Should I write:

$$\alpha_{ij} = \text{Softmax}(\pmb{a}^\top [\pmb{W}h_i || \pmb{W}h_j ] || \pmb{a}_p^\top \pmb{W}_p \pmb{e}_{ij} )$$

Thank you.

[rusty1s]

It would be

a_ij = softmax(a^T [ W h_i || W h_j || W_2 e_ij ])