Heterogeneous GNN for Link prediction should infer the original links of the training dataset?

[jameswpm]

I'm working with Heterogeneous GNN for Link prediction and using the PyG example from here. I'm using the movie lens as an example and adapting it for other problems.

It took me some time to understand what was happening under the hoods and how to make the final edge prediction of the missing links in my original graph. Actually, I've posted a previous question here on the discussions page exactly to be sure I'm going in the right direction. Now I know that I'm doing it, thanks to the answer to that question.

Now I have a new question for this problem, which is more a problem of understanding the example.

After executing the code from the blog post, I have the following ROC_AUC.

It looks like a very nice result, and I guess, I should have reasonable inferences using this model.

Using the code from the aforementioned question, I am getting new recommendations. Iterating over all users, creating an edge for every movie, and asking the model if the edge is a good recommendation. It's the same code, but I'll post it here for completeness.

from tqdm.auto import tqdm

model = model.cpu() 
model.eval() 

total_users = len(unique_user_id) 
total_movies = len(movies_df) 
threshold = 0.5
predictions = {} 

for user_id in tqdm(range(0, total_users)): 
    user_row = torch.tensor([user_id] * total_movies) 
    all_movie_ids = torch.arange(total_movies) 
    edge_label_index = torch.stack([user_row, all_movie_ids], dim=0) 
    data["user", "rates", "movie"].edge_label_index = edge_label_index
    
    with torch.no_grad(): 
        pred = model(data) 
    probabilities = torch.sigmoid(pred) 
    pred_labels = (probabilities > threshold).long() 
    
    recommended_movies = all_movie_ids[pred_labels == 1].tolist() 
    predictions[user_id] = recommended_movies
    print("The user {} will enjoy the following movies:\n{}".format(user_id, recommended_movies), end="\n\n")

My intuition here is if I get a random user (let's say user ID 8) and look for the recommended movies list, I should get back the same movies he previously rated. It means the edge that was used for the model training.

However, I got a bunch of examples, and most of the time, my recommendations are not inferring the original edges used for the training. Is it correct? If so, what should I change if I want more precise predictions?

Let me know if I'm mixing up some concepts here.

[rusty1s]

I think what you can do is not do any thresholding, but instead just sort the predictions based on probability and compare the top-k outcome. You can then measure recall and precision @ k. There is a high chance your threshold may not be optimally chosen, and it will give you a better understanding about how well the recall actually is on a global level.

[jameswpm]

Thank you for your answer. I'll give it a try to this approach.

However, I probably misunderstood some concepts learning about GNNs (and maybe the classification task as a whole 😛) because, in my previous understanding, threshold selection intends to choose the best option to get the most precise predictions, precisely to avoid sorting the probabilities.

In the example code, I manually put the threshold as 0.5 for simplicity, but in my actual code, I'm using the approach described in the answers this post to select the best one. Anyway, I'll check the precision @ K metric to be sure about my approach.