Using GraphSage to do unsupervised node embeddings

[scigeek72]

Hi,
There are few similar questions on the forum, this question seems to be closest to what I am trying to do. But the answers provided there didn't really cleared up my doubts.

Here is my data:

• No node feature
• multi-dimensional edge features
• No labels (hence using unsupervised )
• undirected graph
• test nodes and edges extend the training graph

In order to use GraphSage template, I have defined my own layer (extending the message-passing class) with a forward method that takes in (x, edge_index, edge_attr). Like others, I tried to mimic graph_sage_unsup.py but like others got stuck at replacing NeighborSampler by NeighborLoader.

Now, I have couple of questions.

1. NeightborSampler returns a computational graph for each node in the mini-batch, while NeighborLoader returns the actual subgraph. Here is an example of a mini-batch (of size 148) from running NeighborLoader: Data(x=[1361, 1433], edge_index=[2, 2630], y=[1361], train_mask=[1361], val_mask=[1361], test_mask=[1361], batch_size=148) . From what I understand, data.edge_index[0,:148] contains the actual mini-batch nodes and the rest are sampled neighbors. Is that correct? If that is the case, I can get the pos nodesandneg nodesfor this mini-batch but in order to get their corresponding embeddings (to calculate the loss function), I need the subgraphs corresponding to thesepos nodesandneg nodes`. At this point, I am not sure how to get these. Any way there?
2. Since NeighborSampler is being deprecated, we are changing to NeighborLoader. But in the source code of NeighborLoader, i noticed that it is still using neighbor_sampler in collate_fn=self.neighbor_sampler (here.

I know it is a rather long question, but I have spent considerable time (~2 month since I started reading on GNNs) on it and would like to use the GraphSage template for my data because it seems to fit our need the most.

But if there is another graph neural net that I could use in my scenario, please let me know.
Any help would be highly appreciated.

[rusty1s]

Sorry about not having updated the GraphSAGE unsupervised example yet. The current recommended approach is to use the LinkNeighborLoader for this - the link neighbor loader automatically takes care of sampling negative edges, and as a result, we do not need to longer take care of this by overriding the sampling function in the first place. The general procedure to get there is

• Define a LinkNeighborLoader: The mini-batches will produce additional edge_label_index and edge_label (containing zeros for negatives and ones for positives) attributes that can be used for supervision
• You input the mini-batch just as usual to your GNN
• After obtaining node embeddings, we leverage edge_label_index and edge_label to derive at your loss computation

If you are interested, we can work together on upgrading the current graph_sage_unsup.py to make use of this new interface. WDYT?

[scigeek72]

@rusty1s Thank you very much for your answer. I would be happy to work with you to update graph_sage_unsup.py. But first, let me see if I can incorporate your suggestions into my work and let you know what comes out of it.
Regards

[scigeek72]

@rusty1s, I was playing around with LinkNeighborLoader (BTW, I had to get the nightly build of pyg to get LinkNeighborLoader, otherwise it was throwing an ImportError). I need couple of clarifications. Below is a run of this loader on Planetoid data.

loader = LinkNeighborLoader(data, batch_size=1024, shuffle=True, num_neighbors=[10,10],neg_sampling_ratio=1.0)

sampled_data = next(iter(loader))
print(sampled_data)

output:
Data(x=[2664, 1433], edge_index=[2, 9411], y=[2664], train_mask=[2664], val_mask=[2664], test_mask=[2664], n_id=[2664], edge_attr=[9411, 10], edge_label_index=[2, 2048], edge_label=[2048])

From what I understood from the documentation and your suggestions above, If the following is true, then I can calculate the loss function.

h = model(sampled_data.x, sampled_data.edge_index, sampled_data.edge_attr)

out = h[sampled_data.edge_label_index[0][sampled_data.edge_label == 1]]
pos_out = h[sampled_data.edge_label_index[1][sampled_data.edge_label == 1]]
neg_out = h[sampled_data.edge_label_index[1][sampled_data.edge_label == 0]]

Where out, pos_out and neg_out are the corresponding quantities in graph_sage_unsup.py example.

Is it correct or am I missing something?

Your help is highly appreciated. Thanks.

[rusty1s]

Thanks!

[scigeek72]

@rusty1s, have another question on LinkNeighborLoader. Is it going to land on the stable build soon?

BTW, It seems that LinkNeighborLoader is not working with custom data. I will try some more before posting it in another discussion thread instead of mentioning it here.

[rusty1s]

LinkNeighborLoader was just recently released within PyG 2.1.0. Let me know the issues you encounter with using it on custom data.

[scigeek72]

@rusty1s Got it. I was able to get LinkNeighborLoader from the latest release. Thanks!

BTW, No problem with custom data. I was able to pinpoint my mistake. It is working fine. :)
But I would like to ask you one thing about the iteration (specifically iter()) once the loader is created. To explain it, I need to write a bit (so please bear with me). I used a csv file to generate the graph.

• Loaded the file using pandas
• converted the source col and target col (for edge) into integer ids
• Split it in train_df and test_df
• the proceeded to create train_data and test_data objects (torch_geometric.data.data.Data)
• created a loader on train_data (using LinkNeighborLoader)
• tried to iterate through it (next(iter(loader))) to see the batches
• This is where it failed and threw an error: Index out of bound

What I figured out, is that converting the node ids to integer in the second step above, is causing the error as iter() likes node ids to be in the range(num_nodes). But the way I did it, some of the node ids were not present in train_data (and belonged to test_data). So to fix that, I first split the df and then converted the nodes to integer ID. It is now fixed.

But my question is, shouldn't that be a bit problematic that node ids should be in the range(num_nodes)?

Once again, thanks a lot for taking time to answer my question. Your help is highly appreciated.

[rusty1s]

Really sorry for the late reply. Yes node indices need to be in range [0, num_nodes) for efficient indexing between edge_index and the node feature matrix. With PyG 2.1, we added a data.validate() method which you can use to confirm that this is indeed the case.

[LukeLIN-web]

I also wonder how can I get the actual mini-batch nodes. I use set(batch.edge_index[1]) to get it. I don't think it is suitable.
My batchsize = 6, batch.edge_index[1] = tensor([0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5]), batch.edge_index[0] include subgraph nodes. In my understanding, batch.edge_index[1] don't have sampled neighbors.

[rusty1s]

The easiest approach is just do to

data.n_id = torch.arange(data.num_nodes)

before inputting the data into the mini-batch loader.

[LukeLIN-web]

Thank you for your reply. But I print(batch.n_id) get the subgraph nid (the actual mini-batch nodes + sampled neighbors ) instead of the actual mini-batch nodes. How could I get the actual mini-batch nodes without sampled neighbors?

[rusty1s]

Actual mini-batch nodes are always placed first:

batch.n_id[:batch.batch_size]

[fcossio]

Hi, is there a reason why there is a hadamard product instead of the inner product? It looks to me like it should be inner product.

pytorch_geometric/examples/graph_sage_unsup.py

Line 48 in a218de3

pred = (h_src * h_dst).sum(dim=-1)

I just realized that for single vectors the definition of the inner product is the sum of the hadamard product