Making Inferences with a Trained Model

[ozlander]

I've noticed that most of the examples include a test phase to evaluate the trained model. For example, in examples/tgn.py this is achieved through knowing pos_dst and generating a random neg_dst. If we assume src is known in a link prediction task, how does one make an inference on completely unknown connections to dst? I.e., we cannot assume to know anything about pos_dst, but we do know the nodes of both src and dst. So in a real-world case, there is no test data since we're trying to predict that.

One thought I had was to loop predict every possible edge combination and infer the strongest cases for edges that way. For example:

out1 = link_pred(z[assoc[src]], z[assoc[1,1,1,1,1]])
out2 = link_pred(z[assoc[src]], z[assoc[2,2,2,2,2]])

My reasoning being that these probabilities can then be compared somehow to arrive at the "strongest" probabilities associated with specific edges. However, I fear that this may not be correct or appropriately comparable in this way. Is there a recommended way to make a completely unseen inference using a trained model? Thank you (forgive my lack of a mathematical foundation; slowly and amateurishly learning about all of this)!

[rusty1s]

There is sadly no good answer to this. If you have no further knowledge and simply want to predict the next state of the graph, you need to compute the probability of every potential edge, which obviously does not scale well and is kind of a limitation of link prediction models. However, often, you may be only interested in a set of candidates or the existence of a single edge.

[rusty1s]

The outputs are only dependent based on the previous graph state. You can safely query and derive predictions of any node pair from z.

[ozlander]

I'm trying to reconstruct the full predicted graph on an edge-by-edge basis now to see if I understand the overall process correctly. Is it correct to say that the highest probability given by sigmoid().cpu() will reveal the predicted edge? So for a particular node A, whichever destination node B gives it the highest value relative to other destination nodes will always be the predicted edge. It doesn't matter if node C has a higher probability for node B than A, it's A's relationship with its potential destination nodes that's used to calculate the prediction?

[rusty1s]

Not necessarily since there can be multiple interactions involving the same node, i.e. node A might interact with node B and C for a given timestamp. IMO, a simple threshold, i.e., sigmoid().cpu() > 0.85 should be sufficient to reveal likely interactions.

[ozlander]

Using a threshold makes sense, but what I can't seem to understand is how the authors implemented inductive versus transductive link prediction. With my data, I achieve performance of approximately 95% using the code in examples/tgn.py, but once I loop over edge permutations to calculate sigmoid.cpu(), I get terrible performance when choosing the edge with the highest probability.

I'm trying to replicate this with the Wikipedia dataset, but I can't understand how the method above should produce substantially different performance than the code below:

pos_out = link_pred(z[assoc[src]], z[assoc[pos_dst]])
neg_out = link_pred(z[assoc[src]], z[assoc[neg_dst]])

y_pred = torch.cat([pos_out, neg_out], dim=0).sigmoid().cpu()
y_true = torch.cat([torch.ones(pos_out.size(0)), torch.zeros(neg_out.size(0))], dim=0)

aps.append(average_precision_score(y_true, y_pred))
aucs.append(roc_auc_score(y_true, y_pred))

This is especially perplexing given that the authors of the paper show similar performance for both inductive and transductive approaches. I feel like I'm missing something?

[rusty1s]

What do you mean with getting terrible performance? Note that TGN evaluates the performance of the model only by comparing predictions to a single randomly sampled negative example. You should get a more realistic feel of how well TGN performs by increasing the number of sampled negative edges.