GCN-based Autoencoder not training as intended

[sivaramkrishnan101]

Hi everyone,

Dataset - I have a dataset of 10000 graphs. Each of the graph consists 3 nodes, and the nodes have a node features corresponding to their euclidean(x, y) coordinates. The graphs are fully connected.

Problem statement - I want to train a simple graph autoencoder (GAE) and backpropagate based on loss(node_features, predicted node_features). The proposed encoder consists of 2 GCN layers, each followed by a Relu activation, very similar to the one in the examples.

Issue - But I have been seeing a problem, when looking at the output of the encoder part, i.e. the node embeddings, all the node embeddings have the same vector. For ex., see the image below, all I have done is generate the embedding using the GCN layer for a single graph.

Conclusion - I am not sure what is causing this problem, I have implemented the same GAE architecture that I have come up with on the Citeseer dataset and the results seem to be okay. But in the case of using my dataset, the training is not working fine, and it is better when I just include the FCN layers and remove the GCN layers completely.

I would love to hear your feedback on this and reasons to why this might be happening.

Thank you!

[rusty1s]

GCN cannot operate on a fully connected graph, it will produce identical node embeddings (that's exactly what you observe). Try replacing with SAGEConv :)

[sivaramkrishnan101]

Wow! Thank you, Matthias, for your prompt feedback. I will go ahead and try using the SAGEConv layer as you have suggested.

P.S. Now that you mention it, I was able to figure out as to why I am getting same node embeddings for each node.

[sivaramkrishnan101]

Hi Matthias, I have tried the SAGEConv layer as compared to the GCNConv.

I am still running into the same issues of similar node embeddings for each node (My hypothesis is that this issue arises because of the graph's small size, causing SAGEConv to generate such embeddings, making it ill to use SAGEConv for low-scale graphs especially when it is fully connected).

I have kept the architecture of the encoder to be relatively simple just to generate some 'meaningful' embeddings before proceeding to add more layers in the architecture.

I plan to experiment with other GNN layers available in the torch.nn module mentioned here: https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html.

Kindly inform me if you have any suggestions.

[rusty1s]

I think SAGEConv will only produce identical embeddings if all your input features are identical as well. Is that the case?

[sivaramkrishnan101]

Hello Matthias,

I appreciate your response.

I made mistakes in my previous attempts, but I have corrected them now. In short, I am now capable of creating distinct node embeddings for every node.

Thank you very much for your help.

This is what it looks like after using just a single layer of SAGEConv (I have not used any activations yet).