Update initial node embeddings during each iteration?

[jasperhyp]

I became a little bit confused when thinking about whether to update initial node embeddings or not in a RecSys task. So in shallow embedding-based models, e.g. matrix factorization, the shallow embeddings have to be updated. This is the same for LightGCN, where the only learnable weights are the initial embeddings. But in most scenarios, node features are not updated.

Concretely, my questions are:

(1) In what cases would we not want to set the initial node features back to the original in every iteration? I am sure generative models would usually require those x's to be updated, but unsure about others.

(2) In those cases where initial features are updated in each iteration (say, LightGCN), if I have some manually curated features for each node but not many (say, a couple of properties), I assume it is better to randomly initialize and then concatenate the small feature vector, forming a large "initial vector". In this case, is it common practice that I set the small feature vector to no_grad (but leave the randomly initialized part updatable)? Or perhaps, there could be some sort of embedding layer (probably not linear layers, because input dim is too small) similar to NLP tasks, where words are first embedded to a vector, and then only the projections are updated...?

[rusty1s]

1. I think you definitely do not want to do that in inductive learning scenarios in which you apply your model to unseen graphs/unseen nodes. Updating node features will also result in a huge increase of parameters which may come at the cost of overfitting.
2. Interesting. I think this is non-trivial to combine. One idea is to impute missing features first, see GNNs with Missing Node Features 🚀 #4176.

[jasperhyp]

Thanks Matthias! You are right that updating initial node features would be too costly. I haven't seen such updates in non-generative modeling scenarios before getting into RecSys tasks either lol. Perhaps LightGCN is special in that it's only parameterized by node features and thus needs to update them, and other RecSys papers using GNNs are usually not updating initial features.

I might've skimmed thru this paper a while ago but it's always very interesting to look at. I think my issue is a bit different though. In their problem, they are dealing with a subset of nodes that have certain missing features, while in my scenario I only have a few features (no missingness because I've already preprocessed) and nothing else. I now tend to think CONCAT(Embedding, Real_features) might be the way to go...

[rusty1s]

Yes, if you have some features for every node, there is no reason to throw them away (sorry for misreading your question). torch.cat(embedding, real_features) is the way to go here.

[jasperhyp]

Thanks so much! Missing feature is actually another concern I didn't mention, currently we just threw them all away. Would be interesting to see how the algo would perform if we impute them with that paper you mentioned!