Node classification with TransformerConv

[payload56]

Hi to everyone!
Is there anyone that has reproduced the experiment on ogbn-proteins of the paper "Masked Label Prediction: Unified Message Passing model for semi-supervised classification" with the Pyg operator TransformerConv ? And so which layers to use and how to embed the labels? The PGL implementation is not so clear. Thanks!

[rusty1s]

I haven't personally tried to reproduce the OGB experiments, but you should be able to construct a GNN model via TransformerConv as described in the paper. The only missing piece is then to use masked labels during forward propagation as well, e.g.:

y_one_hot = torch.nn.functional.one_hot(data.y)
train_mask = data.train_mask.clone()

rnd_mask = torch.rand(train_mask.size(0)) < 0.5

y_one_hot[(~train_mask) | (~rnd_mask)] = 0.   # Do not leak information
x = torch.cat([data.x, y_one_hot], dim=-1)

train_mask = train_mask & rnd_mask  # New mask to train against

[payload56]

Two questions about:

1. Have I to scatter/sum the one-hot labels? (since we add another dimension)
2. Will the train mask be less during training than "standard" batch training idxs? Right?

[rusty1s]

1. Not sure I understand. x will be of shape [num_nodes, num_features] and y_one_hot will be of shape [num_nodes, num_classes]. As a result, you can simply concatenate them in the last dimension.
2. Yes, the train_mask will only denote a fraction of original training nodes, as the remaining set of training nodes will be used to inject ground-truth label information into the forward pass.

[payload56]

1. Yes @rusty1s, when I use one_hot(data.y) I found a tensor with dim [num_nodes, num_classes, 2]. Is it correct ?
2. Ok perfect, got it!

[rusty1s]

I think you might need to run one_hot(data.y.view(-1).