Validation and Test Nodes Evaluation Procedure

[yogusmalikombi]

Hi all,

In the Node Classification Colab, all nodes were used at prediction step. After that, test nodes are filtered for evaluation.

def test():
      model.eval()
      out = model(data.x, data.edge_index)
      pred = out.argmax(dim=1) 
      test_correct = pred[data.test_mask] == data.y[data.test_mask] 
      test_acc = int(test_correct.sum()) / int(data.test_mask.sum())  
      return test_acc

May this approach cause some leakage from train nodes? Is there a chance to get information from train nodes, if there are edges exist between train and test nodes?

In my project, I'm trying to do inductive learning. I used NeighborLoader for preparing train batches by giving input_nodes=data.train_mask, and pass these batches into model. I also use train mask on loss calculation. By doing this, I hope that no validation or test nodes are used in training process.

Should I create subgraphs that only consist of validation or test nodes for the performance evaluation? Or, is it safe to continue with the above approach.

Similar question also asked on StackerOverFlow. As I understand from this discussion, Colab approach is used for transductive learning.

[rusty1s]

Yes, the colab notebook refers to a transductive learning scenario. If you want to strictly prevent that validation/test nodes are not seen during training, you should create the induced subgraph from training nodes and work from this moving forward:

train_data = data.subgraph(data.train_mask)