Computing accuracy with a RandomNodeLoader

[sandeepmukh]

Hi! I have a question about how RandomNodeLoader works, because I am seeing significantly higher validation accuracy using this dataloader than running inference on the entire graph.

My understanding is that we sample nodes and compute their induced subgraph with a certain partition, but how are these partitions computed? Is it just we compute the subgraph until we hit a target number of nodes? Doesn't this mean that nodes near the edges of the subgraph will have fewer neighbors than those originally sampled? Is this why the sampled strategy differs substantially from performing inference on the entire dataset?

Is the code in test() a valid way to compute validation accuracy?

Thanks in advance for your time :)

My code:

model = GAT(train_data.num_features, 512, 2, 100, dropout=.2)
criterion = torch.nn.CrossEntropyLoss()  # Define loss criterion.
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=5e-4)  # Define optimizer.
def train(loader):
      model.train()
      loss = 0
      for batch in loader:
            optimizer.zero_grad()  # Clear gradients.
            out = model(batch.x, batch.edge_index)  # Perform a single forward pass.
            loss = criterion(out, batch.y)  # Compute the loss solely based on the training nodes.
            loss.backward()  # Derive gradients.
            optimizer.step()  # Update parameters based on gradients.
      return loss

def test(loader):
      model.eval()
      test_correct = 0
      total = 0
      for batch in loader:
            out = model(batch.x, batch.edge_index)
            pred = out.argmax(dim=1).cpu()  # Use the class with highest probability.
            test_correct += sum(pred == batch.y)  # Check against ground-truth labels.
            total += len(batch.y)
      return test_correct / total  # Derive ratio of correct predictions.

train_loader = RandomNodeLoader(train_data, num_parts=100, shuffle=True, num_workers=4)
val_loader = RandomNodeLoader(val_data, num_parts=50, shuffle=False, num_workers=4)
for epoch in range(1, 10):
    loss = train(train_loader)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}')

print(test()) # Around 57%
# Inference on the whole graph
preds = model(val_data.x, val_data.edge_index).argmax(dim=1).cpu().numpy()
print((preds == val_data.y.cpu().numpy()).mean()) # Around 26%

[sandeepmukh]

Ah it turns out I was misinterpreting what an induced subgraph was and because I wasn't shuffling my data I was getting a subgraph with pretty much all the same labels. Shuffling fixed the issue.