Stability of the embeddings for different random initializations

[dpaysan]

Hi all,

While training a Node2Vec model for different random initialization with the cudnn backend set to deterministic leading to highly differently structured node embeddings spaces. That is when comparing the co-clustering of two node embeddings I find little to no significant agreement as measured by the adjusted mutual information and adjusted RandIndex. While I would not expect perfect agreement for the same hyperparameter set I would expect certain structure in the latent space to be present independent of the random initialization. I see this desired characteristic when using the implementation for Node2Vec from eliorc/node2vec.

Does anybody has an explanation for why the Node2Vec implementation in pytorch_geometric does depend so much on the initialization?

Thanks a lot for the help in advance!

[rusty1s]

Interesting, I never did that study. Can you show me how you evaluate the difference in embeddings in detail?

[dpaysan]

I ran a little bit more extended analyses with various graphs of varying sparsity and different choices of hyperparameters for the Node2Vec embeddings. So I guess it is the geometry of the solution space of the model induced by the chosen hyperparameters of the model that is so highly convex that for different initializations one might obtain highly different embedding spaces even for the same choice of hyperparameters and when operating in a relatively low-dimensional setting.

Nonetheless, in case you are interested I have created a jupyter notebook outlining my analysis. I am also happy for any comments on its design!

[rusty1s]

Wow, great analysis :) Do you know about the differences in implementation of eliorc/node2vec and our Node2Vec model?

[dpaysan]

eliorc/node2vec essentially just implements the random walks and trains a Word2Vec model with those "sentences" using the implementation of the gensim library. I haven't looked into the details tho.

[rusty1s]

I see. AFAIK, gensim trains both input and output embeddings (and discards the outputs after training), while we (and the official formulation in the paper) re-use the inputs for comparing. This might explain the differences in training dynamics, but needs to be investigated. One thing I could think of is to allow gensim-stily training in our Node2Vec directly.