GAT different edge size and attention size in each batch

[mertkiray]

Hello, I am fairly new to graph neural networks and I am training a model using GAT and I have a few questions.

My model input is such that each node correspond to an object coming from the data.

I am calling the function by
result, (edge_index, att_weights) = self.att_matrix(input, edge_index, return_attention_weights=True)

The question is my node count and edge_index size differs in each batch such that in the first batch I have:

input shape: torch.Size([74, 512])   ----> 74 node
edge_index shape: torch.Size([2, 5476]) ---> 74 x 74 edge such that each node is connected
att_weights shape: torch.Size([5476, 1])  ---> as expected a weight matrix for each edge

In the second batch I have:

input shape: torch.Size([50, 512])  ----> 50 node
edge_index shape: torch.Size([2, 2500]) ---> 50 x 50 edge such that each node is connected
att_weights shape: torch.Size([2500, 1]) ---> as expected a weight matrix for each edge

My question is can pytorch_geometric GAT implementation can handle this and having different edge_index and node size for each batch cause a problem for the model?

Should I try to pad the graph in order to have same amount of attention_weights?

If this is expected and not problematic how should pytorch_geometric handle different size of nodes and edge_index? Please guide me any help is greatly appreciated :)

[rusty1s]

PyG automatically handles different sizes of nodes and edges. You don't have to worry about that. Each GNN layer performs node-wise aggregation from its direct neighbors, which is independent of the actual node dimension. In particular, the node dimension can be seen as the "batch" dimension.

[mertkiray]

Okay thanks for the reply :)
I have another question, training attention weights should be used for validation and testing batches too right? What if the validation and test batches have different node and edge size from the training batches?

[rusty1s]

That is also not a problem. Keep in mind that attention weights are not "trained" (i.e., they do not act as trainable embeddings), but are instead directly inferred by the data.

[mertkiray]

Wonderful thank you so much for your help.
Are there any downside making the graph fully connected? At the end GAT will learn to not take into account the useless information for each node right?

[rusty1s]

Yes, at least this is our intuition :)

The only downside of making your graph fully-connected is that PyG is optimized for sparse graph structures. As a result, your pipeline may not be as fast as it could :)

[mertkiray]

So should I increase my epoch because of the learning speed of GAT on fully connected graphs?

[rusty1s]

I was actually talking about runtimes per layer, e.g., the time it takes to process the fully-connected graph. Since we are utilizing sparse operations, those are not as efficient when operating on fully-connected graphs.