Question about the min-cut-pooling example

[zhan-xu]

I got some questions about the usage of dense_mincut_pool.

In examples/proteins_mincut_pool.py, I intentionally ignore the nll loss for graph classification. I would like to see how mincut pooling layers works for clustering. I think by minimizing mincut_loss + ortho_loss, this layer can still help learn to cluster. However, with these two losses only, the total loss decreases slowly and at some point got stuck. Am I missing something here?

As a side question, I would like to ask if it is possible to feed in a "soft" adjacent matrix into dense_mincut_pool where each element indicates the confidence of the affinity?

Thanks for your help.

[rusty1s]

Hi and thanks for your interest!

yes, you can use soft adjacencies as input to dense_mincut_pool. Note that the output of dense_mincut_pool is already a soft adjaccency matrix.

Regarding your first question, @danielegrattarola might be able to help.

[zhan-xu]

Thanks @rusty1s and @danielegrattarola for your kind help!

I tried to run the tensorflow version of the code and seems the clustering losses decreased more reasonably on the Cora dataset (and got the same plots as Fig.5). However with the pytorch version, based on examples/proteins_mincut_pool.py, I removed the nll loss from the total optimized loss and modified the architecture to contain only one time mincut_pooling. The loss seems get stuck.

For your better understanding I pasted a few importance code snippets below. Is it because of the difference of two datasets? I will perhaps try to use the tensorflow version on this dataset. Thanks again for your time!

class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels, hidden_channels=32):
        super(Net, self).__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        num_nodes = ceil(0.5 * average_nodes)
        self.pool1 = Linear(hidden_channels, num_nodes)

    def forward(self, x, edge_index, batch):
        x = F.relu(self.conv1(x, edge_index))
        x, mask = to_dense_batch(x, batch)
        adj = to_dense_adj(edge_index, batch)
        s = self.pool1(x)
        x, adj, mc1, o1 = dense_mincut_pool(x, adj, s, mask)
        return None, mc1, o1

for data in train_loader:
    data = data.to(device)
    optimizer.zero_grad()
    out, mc_loss, o_loss = model(data.x, data.edge_index, data.batch)
    loss = mc_loss + o_loss
    loss.backward()
    loss_all += data.y.size(0) * loss.item()
    optimizer.step()

Outputs:

Epoch: 001, Train Loss: 0.24621914
Epoch: 002, Train Loss: 0.24614548
Epoch: 003, Train Loss: 0.24612426
Epoch: 004, Train Loss: 0.24611660
Epoch: 005, Train Loss: 0.24611321
Epoch: 006, Train Loss: 0.24611142
Epoch: 007, Train Loss: 0.24611033
Epoch: 008, Train Loss: 0.24610971
Epoch: 009, Train Loss: 0.24610925
Epoch: 010, Train Loss: 0.24610903
Epoch: 011, Train Loss: 0.24610881
Epoch: 012, Train Loss: 0.24610870
Epoch: 013, Train Loss: 0.24610867
Epoch: 014, Train Loss: 0.24610868
Epoch: 015, Train Loss: 0.24610862
Epoch: 016, Train Loss: 0.24610857
Epoch: 017, Train Loss: 0.24610854
Epoch: 018, Train Loss: 0.24610853
Epoch: 019, Train Loss: 0.24610853
Epoch: 020, Train Loss: 0.24610853
Epoch: 021, Train Loss: 0.24610853
Epoch: 022, Train Loss: 0.24610853
Epoch: 023, Train Loss: 0.24610853
Epoch: 024, Train Loss: 0.24610853
Epoch: 025, Train Loss: 0.24610853
Epoch: 026, Train Loss: 0.24610853
...

[danielegrattarola]

It's difficult to say a priori what is happening here. Have a look at the two loss terms separately and print the loss value after every batch in the first epoch, it may be that the layer is quickly converging to a solution that is good on average and then it simply stays there because it cannot find any better clustering strategy.
The nll loss is important to introduce a tension between the clustering objective and the task objective, so that might be an explanation for what you're seeing.

And yes, you can expect the layer to work better on some datasets and worse on others. The node features also play an important role in the overall clustering performance, there are many elements at play.
If you want to try the version in Spektral, let me know if you find differences in performance. In principle, the results should be identical.

Cheers
Daniele

[zhan-xu]

Thanks @danielegrattarola for your suggestion. As a quick feedback, I printed out two losses for each batch. Seems cut loss quickly (within several iterations) goes close to -1, while ortho loss stuck at some strange number without changes. As a result I added 1000 as a weight to loss_ortho (loss = mc_loss + 1000*o_loss), and then the ortho loss began to decrease, while the cut loss increased from -1 up slowly. Seems to me these two losses cannot go down together but rather like a trade-off, is that true?

Will let you know when I get more information. Thanks!

[zhan-xu]

As a follow-up, I tried Cora dataset for the clustering task with pytorch-geometric implementation. The performance is consistent with the tensorflow implementation. In both of them, cut loss increases while ortho loss decreases (total loss decreases). I suppose Protein dataset might not have clear clustering pattern which make it hard to train with these two losses only.

Thank you and good luck for your research!

[danielegrattarola]

Yes, the ortho loss is there to contrast some unwanted behaviour of the cut loss, namely it's there to prevent a degenerate minimum of the cut loss at -1 given by a homogeneous cluster assignment (when 1/K-th of each node is assigned to each cluster). This is discussed a bit in the paper and in my blog post about mincut.

The behaviour you're observing is exactly what you would expect (which is what happens on Cora), it's good to know that you can get a better overall performance by weighting the two terms. We did not try Proteins in the original batch of experiments, but if you manage to find new insights on how to make mincut work better I'm happy to update the paper and/or write about it somewhere.

Feel free to get in touch via email ([email protected]).

Cheers