Conceptual differences between GATConv and TransformerConv

[alzaia]

Hello, can anyone give me insights into the differences between the 2 layers? I get that the computation of the attention is different between them, but other than that, why would someone use TransformerConv instead of GATConv for a specific task? Thanks a lot!

[wsad1]

TransformerConv and GATv2Conv are more expressive than GATConv as explained in this paper. Using them might give you better model performance.

[stefanschutz]

Hi, I get that GATv2 is more expressive with the dynamic attention mechanism, which makes it a universal approximator and performs better than GAT. But what about TransformerConv vs. GATv2? The attention mechanism in TransformerConv seems to suffer from the "collapsing" problem described in GATv2 paper. But I am not quite sure about how to compare these 2.

[rusty1s]

Can you point me to the section where "collapsing" of dot product attention is described?

[stefanschutz]

In section 3.3 BUILDING DYNAMIC GRAPH ATTENTION NETWORK: "...the learned layers... can be collapsed into a single linear layer."

[rusty1s]

Yes, but that only holds for GAT-based attention, right? TransformerConv uses dot-product attention.

[stefanschutz]

My apology, it's my mistake. I did try to implement both in my project, using different number of layers for ablation study, and it seems there is no clear winner at least for my problem. So it would be very interesting to compare their expressiveness of the attention mechanisms.