Transductive Train, Validation and Test Split

[jamil2388]

Hello Pyg Team!

I am an amateur in GNN and ML. I am currently experimenting with a toy heterogeneous graph very similar to the structure of Movielens dataset (Small version). My task is for link prediction in a single graph and finally getting the node embeddings. The train-valid and test split should be Transductive. I am trying to understand a few issues and concepts here. I am putting them step by step after attaching the code below :

My Graph :

Node types : user, movie
Edge types : (user, rates, movie), 
                    (movie, rev_rates, user) (from the ToUndirected transform)
num_user = 4
num_movie = 7
num_edges = 12

My relevant portion of code :

def create_custom_data():

    data = HeteroData()

    num_user = 4
    num_movie = 7

    # create
    data["user"].node_id = torch.arange(start = 1, end = num_user + 1, step = 1)
    data["movie"].node_id = torch.arange(start = 1, end = num_movie + 1, step = 1)

    # define edges
    # urm = user rates movie
    # urrm = user rev_rates movie
    urm_edge_index = torch.tensor([[1, 1], [1, 3], [1, 5], \
                      [2, 3], [2, 5], [2, 6], \
                      [3, 3], [3, 4],  \
                      [4, 1], [4, 2], [4, 5], [4, 7]], dtype = torch.long)
    data["user", "rates", "movie"].edge_index = urm_edge_index.t().contiguous()

    # make the graph undirected
    data = T.ToUndirected()(data)

    # review the created data
    print(f'data = {data}')

    return data

def define_splits(data):
    transform = T.RandomLinkSplit(
        num_val=0.1,
        num_test=0.1,
        disjoint_train_ratio=0.3,
        neg_sampling_ratio=2.0,
        add_negative_train_samples=False,
        edge_types=("user", "rates", "movie"),
        rev_edge_types=("movie", "rev_rates", "user"),
    )
    train_data, val_data, test_data = transform(data)

Issues and Questions :

1. As far as I understand (I might be wrong), after Transductive splitting of the edges, the train_data.edge_index and test_data.edge_index should not overlap with any edge. Am I correct?
   Currently after splitting, I am getting weird results on the split edge_indices of the train, valid and test data. Like below :
   train_data[].edge_index = tensor([[3, 3, 4, 4, 1, 4, 2],
   [4, 3, 5, 7, 1, 1, 6]])
   val_data["user", "rates", "movie"].edge_index = tensor([[1, 4, 2, 3, 3, 4, 4, 1, 4, 2],
   [3, 2, 3, 4, 3, 5, 7, 1, 1, 6]])
   test_data["user", "rates", "movie"].edge_index = tensor([[1, 4, 2, 3, 3, 4, 4, 1, 4, 2, 2],
   [3, 2, 3, 4, 3, 5, 7, 1, 1, 6, 5]])

2. Here, if I mention the num_test = 0.1, then from a total of suppose 10 edges in the graph, 9 edges should go to the train_data and the remaining 1 edge should go to the test_data. Ofcourse, train_data and test_data will have all the nodes in them. Should the split happen in this way?

3. I want to perform a very simple split here so I need to know how the parameters of the RandomLinkSplit should be manipulated to get some desired result (just like the normal data splits apart from graph data). I want to understand how the splitting should work here. I want to be able to decode that the given split is correct or incorrect according to my expected output.

I apologize that I could not decode from the instructions given in the documentation and previous discussions or maybe I am misinterpreting a lot of things. So came here to take a direct help from the respective contributors. Thanks a lot in advance! ^_^

[rusty1s]

1. They will overlap. The training split use a subset of training edges for message passing, and the validation split will use all training edges for message passing (both message passing + supervision edges).
2. Yes, this is the intended behavior.
3. Given your configuration, edge_index is split into 80%/10%/10% training/validation/test edges. From the 80% of training edges, 70% are used for message passing, and 30% are used for supervision.

[jamil2388]

Hello @rusty1s, thank you so much for your response. I have been experimenting following your directions. But as you can see from my example output, the number of train / val / test split edges are not aligned with the ratio that I mentioned. Even when I am generating the output mentioning different ratio, the numbers do not align, or I can not judge the validity of the split edges. Also, you can see that I set 0.1 as the test split in the example, but the number of edges in the test split is greater than that of train and val splits. I would really appreciate if you could give me an example of an edge index and then show me a split that follows the definitions, or even something that makes me understand the concept in case of both homogeneous and heterogeneous graphs. Thanks again

[rusty1s]

For me, the output is

data = HeteroData(
  user={ node_id=[4] },
  movie={ node_id=[7] },
  (user, rates, movie)={ edge_index=[2, 12] },
  (movie, rev_rates, user)={ edge_index=[2, 12] }
)
train_data = HeteroData(
  user={ node_id=[4] },
  movie={ node_id=[7] },
  (user, rates, movie)={
    edge_index=[2, 7],
    edge_label=[3],
    edge_label_index=[2, 3],
  },
  (movie, rev_rates, user)={ edge_index=[2, 7] }
)
val_data = HeteroData(
  user={ node_id=[4] },
  movie={ node_id=[7] },
  (user, rates, movie)={
    edge_index=[2, 10],
    edge_label=[3],
    edge_label_index=[2, 3],
  },
  (movie, rev_rates, user)={ edge_index=[2, 10] }
)
test_data = HeteroData(
  user={ node_id=[4] },
  movie={ node_id=[7] },
  (user, rates, movie)={
    edge_index=[2, 11],
    edge_label=[3],
    edge_label_index=[2, 3],
  },
  (movie, rev_rates, user)={ edge_index=[2, 11] }
)

which looks absolutely correct:

• Train data holds 0.8 * 0.7 * 12 = 7 edges for message passing (edge_index), and 0.8 * 0.3 * 12 = 3 edges for supervision (edge_label_index
• Validation data holds all training edges (10 edges) for message passing, and 0.1 * 12 = 1 edge plus 2 negative edges for supervision (3 edges)
• Test data holds all training and validation edges (11 edges) for message passing, and and 0.1 * 12 = 1 edge plus 2 negative edges for supervision (3 edges)

[jamil2388]

Hello @rusty1s, that is a very nice elaboration. It was much needed for me. Thank you so much!