Low AP and precision scores for training data in link prediction task

[USTC-xlab]

Hi, i'm working with the link prediction task and refer to your script Tutorial6.ipynb in the Colab to build GAE model. The model performance is OK for the test dataset (AUC: 0.7724, AP: 0.7693), but i got low AP and precision scores (AUC: 0.8661, AP: 0.0249) when i try to calculate for training dataset. It means that there are lots of false positive sample, and the probability of some negative edges is even higher than that of positive edges according to model output. Did something go wrong?

Code

import os
import torch
os.environ['TORCH'] = torch.__version__
from torch_geometric.datasets import Planetoid
import torch_geometric.transforms as T
from torch_geometric.nn import GCNConv
from torch_geometric.nn import GAE
from torch_geometric.utils import train_test_split_edges

dataset = Planetoid("\..", "CiteSeer", transform=T.NormalizeFeatures())
data = dataset[0]
data.train_mask = data.val_mask = data.test_mask = None
data = train_test_split_edges(data)

class GCNEncoder(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super(GCNEncoder, self).__init__()
        self.conv1 = GCNConv(in_channels, 2 * out_channels, cached=True) # cached only for transductive learning
        self.conv2 = GCNConv(2 * out_channels, out_channels, cached=True) # cached only for transductive learning

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)

# parameters
out_channels = 2
num_features = dataset.num_features
epochs = 100

# model
model = GAE(GCNEncoder(num_features, out_channels))

# move to GPU (if available)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
x = data.x.to(device)
train_pos_edge_index = data.train_pos_edge_index.to(device)
train_neg_edge_index = data.train_neg_adj_mask.nonzero().t().to(device)

# inizialize the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

def train():
    model.train()
    optimizer.zero_grad()
    z = model.encode(x, train_pos_edge_index)
    loss = model.recon_loss(z, train_pos_edge_index)
    loss.backward()
    optimizer.step()
    return float(loss)

def test(pos_edge_index, neg_edge_index):
    model.eval()
    with torch.no_grad():
        z = model.encode(x, train_pos_edge_index)
    return model.test(z, pos_edge_index, neg_edge_index)

for epoch in range(1, epochs + 1):
    loss = train()

    auc, ap = test(data.test_pos_edge_index, data.test_neg_edge_index)
    print('Epoch: {:03d}, AUC: {:.4f}, AP: {:.4f}'.format(epoch, auc, ap))

model output

Epoch: 095, AUC: 0.7732, AP: 0.7702
Epoch: 096, AUC: 0.7723, AP: 0.7685
Epoch: 097, AUC: 0.7716, AP: 0.7677
Epoch: 098, AUC: 0.7723, AP: 0.7689
Epoch: 099, AUC: 0.7727, AP: 0.7696
Epoch: 100, AUC: 0.7724, AP: 0.7693

I used the code as follow to calculate auc and ap scores for training dataset.

model.eval()
with torch.no_grad():
    z = model.encode(x, train_pos_edge_index)
model.test(z, train_pos_edge_index, train_neg_edge_index)

(0.8661695763788778, 0.024931692256452965)

[rusty1s]

What's the training AUC and AP during model training/after each epoch?

[USTC-xlab]

Thanks for your reply. I am a newbie for the link prediction task and really appreciate your tutorials for helping me. Following your tutorial, i chose the same number of negative edges as positive ones by negative sampling in training, validation and testing. This works well! The model can get high AUC and AP in validation and testing.

After that, although the task is finished, i wanna compare the whole reconstructed graph (from decoder) with the input graph. Thus, i use all negative and positive edges from the input graph against reconstructed adjacent matrix to calculate AUC and AP for testing. The number of negative edges is 600 times more than that of positive edges. I got results with high AUC and low AP. Is this due to imbalance sample? And Is this a reasonable result?

[rusty1s]

Yeah, I think this is due to the imbalance sample, the small dataset size and the rather non-expressive GNN variant.

[USTC-xlab]

Okay got it. Just one last question, how do you suggest I improve the AP score? Or that low AP scores don't matter?

[rusty1s]

I would suggest to use a more expressive decoder, e.g., an MLP

h = model.encode(x, edge_index)
h_src, h_dst = h[edge_index[0]], h[edge_index[1]]

return self.MLP(torch.cat([h_src, h_dst], dim=-1))

[USTC-xlab]

Thanks for your suggestion. I'll do a try.