My model is not learning

[eddyakiki]

I am using the ogb molhiv dataset for graph classification, i imported the data and created the DataLoader following the ogb documentation. The data is composed of 41127 graphs and there are 2 classes. Below is the code to implement the neural network in pytorch, and my train and test functions. I have tried implementing other neural networks, tried not to use Dataloaders , changed the loss function as well as the optimizer but the model does not seem to be learning the evaluator provided by ogb keeps giving me an accuracy of 50%. Anyone can help me identify the issue.

I am using the ogb molhiv dataset for graph classification, i imported the data and created the DataLoader following the ogb documentation. The data is composed of 41127 graphs and there are 2 classes. Below is the code to implement the neural network in pytorch, and my train and test functions. I have tried implementing other neural networks, tried not to use Dataloaders , changed the loss function as well as the optimizer but the model does not seem to be learning the evaluator provided by ogb keeps giving me an accuracy of 50%. Anyone can help me identify the issue.

class GCN(torch.nn.Module):
    def __init__(self, hidden_channels):
        super(GCN, self).__init__()
        torch.manual_seed(12345)
        self.conv1 = GCNConv(100, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, hidden_channels)
        self.conv3 = GCNConv(hidden_channels, hidden_channels)
        self.lin = Linear(hidden_channels, dataset.num_classes)

    def forward(self, x, edge_index, batch,edge_attr):
        # 1. Obtain node embeddings 
        x = self.conv1(x, edge_index)
        x = x.relu()
        x = self.conv2(x, edge_index)

        # 2. Readout layer
        x = pyg_nn.global_mean_pool(x, batch)  # [batch_size, hidden_channels]

        # 3. Apply a final classifier
        
        return x

model = GCN(hidden_channels=64)
model = model.to(device)

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
evaluator = Evaluator(name='ogbg-molhiv')
def train():
    model.train()

    for data in train_loader:  # Iterate in batches over the training dataset.
         data.x = atom_encoder(data.x)
         data.edge_attr = bond_encoder(data.edge_attr)
         #data.batch = data.batch.to(device)
         out = model(data.x, data.edge_index, data.batch,data.edge_attr)  # Perform a single forward pass.
         loss = F.nll_loss(out, data.y.squeeze(1))  # Compute the loss.
         loss.backward()  # Derive gradients.
         optimizer.step()  # Update parameters based on gradients.
         optimizer.zero_grad()  # Clear gradients.

def test(loader):
     model.eval()
     y_true = []
     y_pred = []
     correct = 0
     for data in loader:  # Iterate in batches over the training/test dataset.
         data.x = atom_encoder(data.x)
         data.edge_attr = bond_encoder(data.edge_attr)
         out = model(data.x, data.edge_index, data.batch,data.edge_attr)  
         y_pred_batch = out.argmax(dim=1)  # Use the class with highest probability.
         y_true.append(data.y.view(y_pred_batch.shape).detach().cpu())     
         y_pred.append(y_pred_batch.detach().cpu())
     y_true = torch.cat(y_true, dim = 0).numpy()
     y_pred = torch.cat(y_pred, dim = 0).numpy()
     acc = evaluator.eval({'y_true': y_true.reshape(len(y_true),1),
                                'y_pred': y_pred.reshape(len(y_true),1),
                               })
     return acc['rocauc']  # Derive ratio of correct predictions.


for epoch in range(1, 171):
    train()
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Train Acc: {train_acc:.4f}, Test Acc: {test_acc:.4f}')

[EdisonLeeeee]

It seems that the atom_encoder and bond_encoder are not updated during training. Can you confirm?

[eddyakiki]

this did not work i am still getting a constant ROC-AUC of 50%

[EdisonLeeeee]

You are missing a softmax layer before the loss. Can you try on F.cross_entropy rather than F.nll_loss?

[eddyakiki]

I changed it started getting results like the following
before it was strictly 0.5000 for both train and test ROC-AUC

Epoch: 001, Train Acc: 0.5057, Test Acc: 0.4999
Epoch: 002, Train Acc: 0.5068, Test Acc: 0.5028
Epoch: 003, Train Acc: 0.5000, Test Acc: 0.5000
Epoch: 004, Train Acc: 0.5000, Test Acc: 0.5000
Epoch: 005, Train Acc: 0.5004, Test Acc: 0.5000
Epoch: 006, Train Acc: 0.5000, Test Acc: 0.5000
Epoch: 007, Train Acc: 0.5000, Test Acc: 0.5000
Epoch: 008, Train Acc: 0.4999, Test Acc: 0.5000
.
.
.

[EdisonLeeeee]

Not really sure about it. Could you show me your updated code? Or you can kindly provide a script for me to reproduce the experiments and figure it out.

[eddyakiki]

Here is the code below

import os
import torch
os.environ['TORCH'] = torch.__version__
print(torch.__version__)

!pip install -q torch-scatter -f https://data.pyg.org/whl/torch-${TORCH}.html
!pip install -q torch-sparse -f https://data.pyg.org/whl/torch-${TORCH}.html
!pip install -q git+https://github.com/pyg-team/pytorch_geometric.git

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch_geometric

!pip install ogb

# define the device
device = 'cuda' if torch.cuda.is_available() else 'cpu'
device = torch.device(device)

from ogb.graphproppred import PygGraphPropPredDataset, Evaluator
from torch_geometric.data import DataLoader
import torch_geometric.transforms as T
from torch.utils.data.dataloader import default_collate

dataset = PygGraphPropPredDataset(name = 'ogbg-molhiv')#,transform=T.ToSparseTensor()) 
dataset.data.to(device)
split_idx = dataset.get_idx_split() 
train_loader = DataLoader(dataset[split_idx["train"]], batch_size=32 ,shuffle=True)
valid_loader = DataLoader(dataset[split_idx["valid"]], batch_size=32, shuffle=False)
test_loader = DataLoader(dataset[split_idx["test"]], batch_size=32, shuffle=False)

from ogb.graphproppred.mol_encoder import AtomEncoder, BondEncoder
atom_encoder = AtomEncoder(emb_dim = 100)
bond_encoder = BondEncoder(emb_dim = 100)
atom_encoder = atom_encoder.to(device)
bond_encoder = bond_encoder.to(device)

import torch_geometric.nn as pyg_nn
import torch_geometric.utils as pyg_utils

from torch.nn import Linear
import torch.nn.functional as F
from torch_geometric.nn import GCNConv,GraphConv
from torch_geometric.nn import global_mean_pool


class GCN(torch.nn.Module):
    def __init__(self, hidden_channels):
        super(GCN, self).__init__()
        #torch.manual_seed(12345)
        self.conv1 = GraphConv(100, hidden_channels)
        self.conv2 = GraphConv(hidden_channels, hidden_channels)
        self.conv3 = GraphConv(hidden_channels, hidden_channels)
        self.lin = Linear(hidden_channels, dataset.num_classes)

    def forward(self, x, edge_index, batch,edge_attr):
        # 1. Obtain node embeddings 
        x = self.conv1(x, edge_index)
        x = x.relu()
        x = self.conv2(x, edge_index)

        # 2. Readout layer
        x = pyg_nn.global_mean_pool(x, batch)  # [batch_size, hidden_channels]

        # 3. Apply a final classifier
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.lin(x)
        return x

model = GCN(hidden_channels=64)
model = model.to(device)

optimizer = torch.optim.Adam(list(model.parameters())+list(atom_encoder.parameters())+list(bond_encoder.parameters()), 
                          lr=0.01)
evaluator = Evaluator(name='ogbg-molhiv')
criterion = torch.nn.CrossEntropyLoss()
def train():
    model.train()
    for data in train_loader:  # Iterate in batches over the training dataset.
         data.x = atom_encoder(data.x)
         data.edge_attr = bond_encoder(data.edge_attr)
         out = model(data.x, data.edge_index, data.batch,data.edge_attr)  # Perform a single forward pass.
         loss = F.cross_entropy(out, data.y.squeeze(1))  # Compute the loss.
         loss.backward()  # Derive gradients.
         optimizer.step()  # Update parameters based on gradients.
         optimizer.zero_grad()  # Clear gradients.

def test(loader):
     model.eval()
     y_true = []
     y_pred = []
     correct = 0
     for data in loader:  # Iterate in batches over the training/test dataset.
         data.x = atom_encoder(data.x)
         data.edge_attr = bond_encoder(data.edge_attr)
         out = model(data.x, data.edge_index, data.batch,data.edge_attr)  
         y_pred_batch = out.argmax(dim=1)  # Use the class with highest probability.
         y_true.append(data.y.view(y_pred_batch.shape).detach().cpu())     
         y_pred.append(y_pred_batch.detach().cpu())
     y_true = torch.cat(y_true, dim = 0).numpy()
     y_pred = torch.cat(y_pred, dim = 0).numpy()
     acc = evaluator.eval({'y_true': y_true.reshape(len(y_true),1),
                                'y_pred': y_pred.reshape(len(y_true),1),
                               })
     return acc['rocauc']  # Derive ratio of correct predictions.


for epoch in range(1, 171):
    train()
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Train Acc: {train_acc:.4f}, Test Acc: {test_acc:.4f}')

[EdisonLeeeee]

Thanks for your script. The problem is that the computation of AUC score takes the class probability instead of class labels as the input. So you don't have to compute the out.argmax(dim=1). So the code in test() function should be:

y_pred_batch = F.softmax(out, dim=-1)[:, 1] # And don't forget the softmax operation to obtain the class probability

BTW, you need to use a large batch size (e.g., 512) since the dataset is imbalanced.

[eddyakiki]

Thank you so much for your help, it finally worked.