GAT has the same cost a self attention even for sparse adjency matrix

[MehdiZouitine]

Hello,

I am wondering why graph attention (GAT) is as expensive in memory as self attention for a very sparse adjacency matrix. I understand that both GAT and self attention involve a lot of matrix multiplications and dot products, but I would expect GAT to be more memory efficient since it is dealing with sparse graphs.

Can anyone explain the reason for this or offer any suggestions for how to reduce the memory requirements for GAT on sparse graphs?

Thank you for your help.

[wsad1]

Which implementations are you comparing here? I want to understand how you arrived at this conclusion.

Also this doc talks about how GNNs use up memory, and how this can be improved for certain GNNs. Currently GATConv doesn't have support for this, but there is an open PR that tries to add support.