Stacking Graph Convolutions in the case of Bipartite Graph

[sasan73]

Hello and thank you for your great efforts.

I have been trying to implement graph convolutions to build a recommendation system. I have a bipartite graph of users and articles. I'm using torch geometric's SAGEConv. My code looks like this:

class SpectralCF(nn.Module):
  def __init__(self, in_channel1, in_channel2):
    super(SpectralCF, self).__init__()
    self.conv1_user = SAGEConv((in_channel1, in_channel2), 8)
    self.conv1_item = SAGEConv((in_channel2, in_channel1), 8)
    
    self.conv2_user = SAGEConv((8 ,8), 16)
    self.conv2_item = SAGEConv((8, 8), 16)

    self.conv3_user = SAGEConv((16, 16), 32)
    self.conv3_item = SAGEConv((16, 16), 32)
    
  def forward(self, data):

    edge_index, x_user, x_item = data.edge_index, data.x_u, data.x_i

    x_user_1 = self.conv1_user((x_item, x_user), edge_index[torch.tensor([1,0])])
    x_user_1 = F.sigmoid(x_user_1)
    x_item_1 = self.conv1_item((x_user, x_item), edge_index)
    x_item_1 = F.sigmoid(x_item_1)

    x_user_2 = self.conv2_user((x_item_1, x_user_1), edge_index[torch.tensor([1,0])])
    x_user_2 = F.sigmoid(x_user_2)
    x_item_2 = self.conv2_item((x_user_1, x_item_1), edge_index)
    x_item_2 = F.sigmoid(x_item_2)
    
    x_user_3 = self.conv3_user((x_item_2, x_user_2), edge_index[torch.tensor([1,0])])
    x_user_3 = F.sigmoid(x_user_3)
    x_item_3 = self.conv3_item((x_user_2, x_item_2), edge_index)
    x_item_3 = F.sigmoid(x_item_3)

    x_user_4 = torch.cat((x_user_1, x_user_2, x_user_3), dim=1)
    x_item_4 = torch.cat((x_item_1, x_item_2, x_item_3), dim=1)

    return (x_user_4, x_item_4)

However, I noticed that after the second layer of convolution the layer outputs become very similar. This is how the output of the last layer looks like:

tensor([[0.4445, 0.4241, 0.4541, ..., 0.4134, 0.3659, 0.3035],
[0.4438, 0.3707, 0.4923, ..., 0.4131, 0.3656, 0.3036],
[0.4118, 0.4384, 0.4705, ..., 0.4133, 0.3673, 0.3042],
...,
[0.3790, 0.4457, 0.3726, ..., 0.4139, 0.3660, 0.3037],
[0.3949, 0.4109, 0.3705, ..., 0.4135, 0.3641, 0.3038],
[0.3477, 0.3963, 0.3873, ..., 0.4134, 0.3644, 0.3054]],
grad_fn=)

This persists after an arbitrary number of epochs as well. Is there something wrong with how I am implementing the graph convolutional layers?

I have also tried replacing my feature matrix(data.x_i, data.x_u) with random values, but did not see any difference this happens no matter what the initial values of x_i and x_u are.

[rusty1s]

Thanks for your interest. It's impossible to say how features should look like :) Does your model learn at all or is it producing random results?

Since all of your features are close to 0.5, I think this might be due to the use of sigmoid as an activation function. Have you tried to use something like relu?

[sasan73]

Thank you for the quick reply.
The model seems to be learning, but not to correctly predict the links between each user and item. The only thing I have noticed is that after a few epochs the output layer of the user features x_u and the items x_i start to become very similar. for example in the output tensor above the feature embedding of the first user is:

[0.4445, 0.4241, 0.4541, ..., 0.4134, 0.3659, 0.3035]
and for the second user we have:

[0.4438, 0.3707, 0.4923, ..., 0.4131, 0.3656, 0.3036],
this similarity becomes stronger after each epoch.

However, when I use relu the model doesn't learn and I get an output feature layer that is mainly comprised of zero columns.

tensor([[0.0000, 0.4563, 0.0000, ..., 0.0000, 0.4588, 0.0000],
[0.0000, 0.3401, 0.0000, ..., 0.0000, 0.4563, 0.0000],
[0.0000, 0.0738, 0.0000, ..., 0.0000, 0.4547, 0.0000],
...,
[0.0000, 0.3382, 0.0000, ..., 0.0000, 0.4582, 0.0000],
[0.0000, 0.4284, 0.0000, ..., 0.0000, 0.4590, 0.0000],
[0.0000, 0.1384, 0.0000, ..., 0.0000, 0.4596, 0.0000]],
grad_fn=)

[rusty1s]

How do you use those outputs to make predictions? Is there an additional MLP/linear transformation on top of it?