LinkNeighborLoader doesn't work well on movielens dataset.

[kazuma-jp]

Summary

Hello.
Thank you for developing pytorch geometric. I am enjoying it very much!!!

I have a question.
I implemented GNN on movielens dataset and tried to predict links !
I used the pytorch geometric tutorial as a reference for the implementation.
However, LinkNeighborLoader returns wrong data.
Please tell me the proper implementation of LinkNeighborLoader.

Desciption

Code and Stdout

Code

import numpy as np
from torch_geometric.data import (
    HeteroData,
    InMemoryDataset,
    download_url,
    extract_zip,
)
import torch_geometric.transforms as T
import os
import pandas as pd
import torch
from torch.nn import Linear, Embedding
from torch_geometric.loader import LinkNeighborLoader
from torch_geometric.nn import SAGEConv, to_hetero

class AverageMeter():
    """Computes and stores the average and current value"""
    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

movies = pd.read_csv('./data/movies.csv')[['movieId', 'genres']]
ratings = pd.read_csv('./data/ratings.csv')

movie_feat_df = movies.set_index('movieId')['genres'].str.get_dummies('|')
movie_feat = torch.from_numpy(movie_feat_df.values).to(torch.float)

# user_id_df
unique_user_ids = ratings['userId'].unique()
unique_user_id_df = pd.DataFrame({
    'userId':unique_user_ids,
    'mappedId': pd.RangeIndex(len(unique_user_ids)),
})

# movie_id df
unique_movie_ids = ratings['movieId'].unique()
unique_movie_id_df = pd.DataFrame({
    'movieId': ratings['movieId'].unique(),
    'mappedId': pd.RangeIndex(len(unique_movie_ids)),
})

ratings_user_id = pd.merge(ratings['userId'], unique_user_id_df, on='userId', how='left')
ratings_user_id = torch.from_numpy(ratings_user_id['mappedId'].values)

ratings_movie_id = pd.merge(ratings['movieId'], unique_movie_id_df, on='movieId', how='left')
ratings_movie_id = torch.from_numpy(ratings_movie_id['mappedId'].values)

edge_index_user_to_movie = torch.stack([ratings_user_id, ratings_movie_id], dim=0)

data = HeteroData()

data['user'].node_id = torch.arange(len(unique_user_ids))
data['movie'].node_id = torch.arange(len(unique_movie_ids))


data["movie"].x = movie_feat
data["user", "rates", "movie"].edge_index = edge_index_user_to_movie

data = T.ToUndirected()(data)

transform = T.RandomLinkSplit(
    num_val=0.1, 
    num_test=0.1, 
    edge_types=("user", "rates", "movie"),
    rev_edge_types=("movie", "rev_rates", "user"),
    disjoint_train_ratio=0.3,
    add_negative_train_samples=False, 
    neg_sampling_ratio = 2
)

train_data, val_data, test_data = transform(data)

def get_loader(dat, is_train=True):
    edge_label_index = dat["user", "rates", "movie"].edge_label_index
    edge_label = dat["user", "rates", "movie"].edge_label
    print(edge_label.unique())

    loader = LinkNeighborLoader(
        data = dat,
        num_neighbors=[20, 10],
        neg_sampling_ratio=2,
        edge_label_index=(("user", 'rates', "movie"), edge_label_index),
        edge_label=edge_label,
        batch_size=128,
        shuffle=is_train,
    )
    return loader

next(iter(get_loader(train_data)))

Return
movie x dimension is 9724. But, appropriate dimension is 2714.

HeteroData(
  user={ node_id=[608] },
  movie={
    node_id=[2714],
    **x=[9742, 20]**
  },
  (user, rates, movie)={
    edge_index=[2, 17341],
    edge_label=[384],
    edge_label_index=[2, 384]
  },
  (movie, rev_rates, user)={ edge_index=[2, 7902] }
)

Want Return

HeteroData(
  user={ node_id=[608] },
  movie={
    node_id=[2714],
    **x=[2714, 20]**
  },
  (user, rates, movie)={
    edge_index=[2, 17341],
    edge_label=[384],
    edge_label_index=[2, 384],
    **input_id=[128]**
  },
  (movie, rev_rates, user)={ edge_index=[2, 7902] }
)

environment

PYTORCH="1.12.0"
CUDA="11.3"
CUDNN="8"

[rusty1s]

Can you print(data) your data object here before inputting it into the LinkNeighborLoader?

[rusty1s]

Thanks for confirming. Do you know in which operator the error is thrown? If this is not shown, you can try to run your script via

CUDA_LAUNCH_BLOCKING=1 python ...

[kazuma-jp]

Thank you very much for your advice!!
I appreciate all the advice you gave me.
I found that my model was incorrect and fixed it.
Now I no longer get the error.

However, my model is overfitting train_data.
I looked into the cause and found that there are two cases: (1) overfitting and (2) not overfitting.
(1) Overfitting case: Negative_sampling is dynamically performed on the LinkNeighborLoader side.
(2) Non-overfitting case: Negative_sampling is performed on the RandomLinkSplit side.

Due to the dynamic negative sample switching, the training does not seem to work.
In this case, is it correct to train as in (2)? Also, what is the best training method?

(1)

class CFG:
    is_dynamic_negative_sampling = True

### OUTPUT ###
Epoch: 0 Train loss: 0.41060061445487817 Valid loss: 0.8810698983610088 
Epoch: 1 Train loss: 0.3360097217001229 Valid loss: 0.8683305130976835 
Epoch: 2 Train loss: 0.3104639100700138 Valid loss: 0.9402838466262446 
Epoch: 3 Train loss: 0.29362667407468745 Valid loss: 1.0781569697657176 
Epoch: 4 Train loss: 0.2779366972882276 Valid loss: 1.0548860040116796 
Epoch: 5 Train loss: 0.2696830945153418 Valid loss: 1.1620737207612146 
Epoch: 6 Train loss: 0.25780814357431636 Valid loss: 1.2799582669366794 
Epoch: 7 Train loss: 0.2508755982153662 Valid loss: 1.2127061353917516 
FInish train
##############

(2)

class CFG:
    is_dynamic_negative_sampling = False

### OUTPUT ###
Epoch: 0 Train loss: 0.384119969431862 Valid loss: 0.3471822407904639 
Epoch: 1 Train loss: 0.3068970925061809 Valid loss: 0.3276535371998455 
Epoch: 2 Train loss: 0.272300852594081 Valid loss: 0.3277329506489922 
Epoch: 3 Train loss: 0.2456925002083743 Valid loss: 0.35532713899762497 
Epoch: 4 Train loss: 0.2216535508727071 Valid loss: 0.3521181536889706 
Epoch: 5 Train loss: 0.1972199542532561 Valid loss: 0.39974333825058267 
Epoch: 6 Train loss: 0.17368467847005414 Valid loss: 0.4422553781034643 
Epoch: 7 Train loss: 0.15450498990579808 Valid loss: 0.47487952607699263 
FInish train
##############

commom code

transform = T.RandomLinkSplit(
    num_val=0.1, 
    num_test=0.1, 
    edge_types=("user", "rates", "movie"),
    rev_edge_types=("movie", "rev_rates", "user"),
    disjoint_train_ratio=0.3,
    add_negative_train_samples=not(CFG.is_dynamic_negative_sampling),
    neg_sampling_ratio = 2
)

def get_loader(dat, is_train=True):
    edge_label_index = dat["user", "rates", "movie"].edge_label_index
    edge_label = dat["user", "rates", "movie"].edge_label

    if CFG.is_dynamic_negative_sampling:
        neg_sampling_ratio=2
    else:
        neg_sampling_ratio=0

    loader = LinkNeighborLoader(
        data = dat,
        num_neighbors=[20, 10],
        neg_sampling_ratio=neg_sampling_ratio,
        edge_label_index=(("user", 'rates', "movie"), edge_label_index),
        edge_label=edge_label,
        batch_size=128,
        shuffle=is_train,
    )
    return loader

[rusty1s]

This is interesting. I always assumed that dynamic negative sampling will always out-perform static negative sampling in a pre-processing step.

[kazuma-jp]

Thank you for your comment!!
I would like to reference a sample code using LinkNeighborLoader and RandomLinkSplit. Is there somewhere I can find it?
I want to compare it with my code to see if there are any mistakes.

I would also like to know if there is any code comparing dynamic negative sampling and static negative sampling.

[kazuma-jp]

Anyway, my main problem was solved.
Thank you!!