Model loss starting off too low / Eval metric not moving

[johntcomo]

Hello, I am working with this code : https://github.com/pyg-team/pytorch_geometric/blob/master/examples/hetero/bipartite_sage_unsup.py , but only a subset of users (50k), products (1 mill) and edges (5 mill) - I sorted base df by user_id so that all edges for each user in graph are present. After running the first epoch I saw the loss was .00002. I figured this can be due to the many batches and gradient updates so printed the loss after every 10 batches in the first epoch and saw for the first batch was also .00002 - scarily too low for being so early on in training I believe. However, the evaluation metric - AUROC increased over this epoch from .51 to .83 as expected.

I am also working with the same graph but using Metapath2Vec instead of a GNN. I am experiencing the inverse problem of a steadily decreasing loss, but AUROC evaluation being constant over many epochs.

Does anyone know why this may be happening? Could it be I took too small subset of original dataset? My GNN code is exactly same as the linked code, my Metapath2Vec evaluation code is as follows - I added a linear head to provide similar evaluation as in the sample code :

linear_model = torch.nn.Sequential(
    torch.nn.Linear(2 * 128, 128), 
    torch.nn.ReLU(), 
    torch.nn.Linear(128, 1)
)

@torch.no_grad()
def test(loader, linear_model): 
    model.eval()

    preds, targets = [], []

    for batch in tqdm.tqdm(loader): 
        batch = batch.to(device)

        users = model.forward(node_type='user')
        items = model.forward(node_type='item')

        edge_index = batch['user', 'item'].edge_label_index.cpu()

        user_feature = users[edge_index[0]]
        item_feature = items[edge_index[1]]

        z = torch.cat([user_feature, item_feature], dim = -1)
        z = linear_model(z)
        pred = z.squeeze().sigmoid().cpu()

        target = batch['user', 'item'].edge_label.long().cpu()

        preds.append(pred)
        targets.append(target)

    pred = torch.cat(preds, dim = 0).numpy()
    target = torch.cat(targets, dim = 0).numpy()


    return roc_auc_score(target, pred)

[rusty1s]

I don't have a good explanation yet. Are you using just binary cross entropy as in the example? What happens if you increase the number of negatives?

[johntcomo]

Hello @rusty1s , thanks for taking the time to field this question. For both I increased the negative sampling ratio to be 4:1, and yes I am using BCE.

• For the GNN model increasing the negative sampling ratio has still kept the loss at .0002 very early in first epoch, the AUROC has improved slightly and accuracy has decreased slightly - both of which make sense.
• For metapath2vec, the loss is decreasing in a similar fashion as before but the AUROC is fixed.

For the metapath problem perhaps is it a poor problem formulation? I am trying to compare graph design spaces by using a link prediction evaluation across all designs. I train the full graph of 5 million edges using metapath to learn the embeddings of each node and then create test edges to evaluate on the link prediction task and pass into a LinkLoader.

train_loader = model.loader(batch_size = 512, shuffle = True, num_workers=1) 
....
train_data, test_data, val_data = T.RandomLinkSplit(...)

test_loader = LinkNeighborLoader(
    data=test_data,
    num_neighbors=[8, 4],
    edge_label_index=(
        ('user', 'to', 'item'),
        test_data[('user', 'to', 'item')].edge_label_index,
    ),
    edge_label=test_data[('user', 'to', 'item')].edge_label,
    batch_size=2048,
    shuffle=False,
    num_workers=2,
)

Then I iterate through edges in test loader to evaluate embeddings for link prediction task for comparison.

[rusty1s]

Sorry for my late reply. For the metapath formulation, I think you need to make sure to remove the test edges before training metapath2vec. Otherwise, you would "leak" information.

[johntcomo]

No worries at all, any response is very much appreciated. Thank you for this insight. I see I was training on all nodes and edges and indeed leaking - the correct problem is training on all nodes and train test split edges as that is the task. This leaking however would contribute to high accuracy and AUROC eval which is not the case however as both are stuck ~50%. I will rerun some experiments and provide an update. Thanks again!

[johntcomo]

This change has not affected any results.