Bias in GAT layers

[dhall1995]

Taking a look into the code for a GATconv it looks like when we apply a bias to node features in a GAT layer the bias is applied after working out attention coefficients i.e. we first propagate messages and then add bias to node features post-propagation.

Would it be legitimate to apply the bias before working out attention coefficients? Surely this would make for a more expressive attention mechanism? Or am I missing something...

[rusty1s]

No reason to delete. This is a great question :)

We follow the official implementation here, but you are absolutely right that adding a bias might lead to a more expressive attention mechanism after the LeakyReLU call.