Beginner Question : GCNConv with Reddit

[jiyoungAn]

Hi I am very new in this field.
I am doing a project that has to process Reddit data only with GCNConv.
If I don't use sparse representation, it gets killed.
if I use sparse representation, I can run the program, but each epoch takes about 37s in my GPU.
Also the accuracy is very bad (40%).
I'd like to ask your opinion and idea to improve this model. Thank you for your help in advance.

• Also, is it possible to use DataLoader(batch) for GCNConv ?

`device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
path = osp.join(osp.dirname(osp.realpath(__file__)), 'Reddit')
dataset = Reddit(path, transform=T.ToSparseTensor())

class GCN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, out_channels)


    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index)
        x = F.relu(x) 
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        return F.log_softmax(x, dim=1)


model = GCN(dataset.num_features, 16, dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
data = dataset[0].to(device)
def train():
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.adj_t)
    loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
    loss.backward()
    optimizer.step()

@torch.no_grad()
def test():
    model.eval()
    log_probs, accs = model(data.x, data.adj_t), []
    for _, mask in data('train_mask', 'test_mask'):
        pred = log_probs[mask].max(1)[1]
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


for epoch in range(1, 21):
    train()
    train_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Train: {train_acc:.4f}, Test: {test_acc:.4f}')

[LukeLIN-web]

Normal GCN is training at whole graph.
If you want to use DataLoader(batch) . You may need Neighborloader, see pytorch_geometric/examples/reddit.py
Also you can refer pytorch_geometric/examples/gcn.py

[rusty1s]

+1. You should be able to achieve much faster training when using mini-batch training.

[jiyoungAn]

Thank you for your help!
After applying mini-batch training, the accuracy is now 90%.
I used neighborloader with -1 (including all neighbors).
But I wonder why this makes difference,
because I believe the result of full-batch training should be the same with mini-batch training with all neighbors.
(full batch training = full batch training with sparse representation = mini batch training with all neighbors included)

If we use mini-batch training with all neighbors, does this work as a inductive training?

[LukeLIN-web]

Thank you for your help! After applying mini-batch training, the accuracy is now 90%. I used neighborloader with -1 (including all neighbors). But I wonder why this makes difference, because I believe the result of full-batch training should be the same with mini-batch training with all neighbors. (full batch training = full batch training with sparse representation = mini batch training with all neighbors included)

If we use mini-batch training with all neighbors, does this work as a inductive training?

use neighborloader with -1 (including all neighbors) is only train with their first hop neighbors, refer https://snap.stanford.edu/graphsage/ Inductive Representation Learning on Large Graphs - arXiv
origin full batch, each node is trained with whole graph. it is time consuming.

[jiyoungAn]

However, even with full batch, each node should be updated using the first hop neighbors(adjacency matrix x node feature x weight). Each mini-batch will have sub adjacency matrix that includes nodes and adjacent nodes with first hop neighbors.
So, if we iterate all batch set, the final result of the mini-batch training with all first hop neighbors, regardless of time, should be the similar with the full batch. Even if losses vary from each epoch, shouldn't they be converging similarly?..
That is what I thought.. Please correct me if I am wrong... :)

[LukeLIN-web]

However, even with full batch, each node should be updated using the first hop neighbors(adjacency matrix x node feature x weight). Each mini-batch will have sub adjacency matrix that includes nodes and adjacent nodes with first hop neighbors. So, if we iterate all batch set, the final result of the mini-batch training with all first hop neighbors, regardless of time, should be the similar with the full batch. Even if losses vary from each epoch, shouldn't they be converging similarly?.. That is what I thought.. Please correct me if I am wrong... :)

Please read Inductive Representation Learning on Large Graphs - arXiv
a full batch will aggregate further node(such as 2 hop) features to 1 hop feature.