Heterogeneous data; nodes without features.

[michalisfrangos]

Hi all,

Just a discussion point..

In the example for Heterogenous graphs here, each node comes with a feature.

data['paper'].x = ... # [num_papers, num_features_paper]
data['author'].x = ... # [num_authors, num_features_author]
data['institution'].x = ... # [num_institutions, num_features_institution]
data['field_of_study'].x = ... # [num_field, num_features_field]

However it could be a case where a node, e.g., 'field of study' does not have any features, and the node is fully defined by it's index. However in the graph framework it is not possible to have nodes with no features (no information to send to neighboring nodes) Isn't this a limitation though? if a paper belongs to a field of study, then this could be all the information we have.

How would one approach this in terms of implementation in heterogenous graphs? ( using as feature an "id" for the field of study isn't the best, as this number will be increasing with increasing number of topics.) Creating an embedding for each topic based on the field id, could be another option, but how do we capture the information in our graph that some papers are linked together due to the fact that they belong to a particular field of study, but we do not have any other information/features for this fields of study?

Cheers,
M

[rusty1s]

That is correct. In OGB_MAG, we use metapath2vec for nodes with missing features, which works well in practice. Alternatively, one can initialize node features randomly or make them learnable via torch.nn.Embedding.

[michalisfrangos]

Hi @rusty1s ,

Thank you for the response to the discussion above. It took me some time to respond but finally here I am . I read a couple of papers to follow and also had a look to a colab tutorial.

I see that generating embeddings using metapath2vec (or learning torch.nn.Embdeding) suggestion would work as transductive approaches. However, given new unseen nodes, perhaps one could repeat the metapath2vec process to learn all embeddings(maybe expensive, unless we could learn just the embeddings of the new nodes from their neighborhood somehow). But, if I am not mistaken we cannot generate embeddings on new nodes if we have learned a torch.nnEmbeddings on a different structure/nodes etc...

[rusty1s]

Yes, metapath2vec or learned embeddings via torch.nn.Embedding are transductive by nature and cannot be applied to other graphs. For this, you may want to utilize some synthetic features, e.g., node degree, or the average of features from direct neighbors.

[michalisfrangos]

Hi @rusty1s ,thanks for the response. Regarding the use of torch.nn.Embedding, in the heterogeneous framework... could you please point me to the right approach/example of implementation please that you might be aware please? for example in the case where we have a node of type X (in a heterogeneous framework) and the node has a 1-dim feature taking categorical values [1,2,3], how would we integrate a torch.nnEmbedding in the framework for the features of node X (with trainable or fixed embedding parameters) ?
Thank you ..

[rusty1s]

For fixed but random embeddings, you can simply set data[node_type].x = torch.randn(num_nodes, num_features). In case you want to make use of torch.nn.Embedding, you will need to instantiate such an embedding for every node type that is missing features:

self.emb_dict = torch.nn.ModuleDict()
for node_type in node_types_with_missing_features:
    self.emb_dict[node_type] = torch.nn.Embedding(num_nodes_of_node_type, num_features)

and apply it right at the start of forward:

for node_type, emb in self.emb_dict.items() 
    x_dict[node_type] = emb.weight

[michalisfrangos]

Thanks you very much for the quick response ! I will give it a go just now.