Does not improve loss values during training

[hkim716]

Hi Matt,

I'm trying to make a simple classification GNN, which is similar to mnist_graclus.py example.
I transformed original mnist superpixel data into Delaunay data form, and stored cluster vectors for pooling, which is cluster0, cluster1, cluster2. From that clusters, I calculated and stored edge_attr (0,1,2).
However, the loss values are not decreasing at all during the training.

Epoch: 01, Test: 0.1013
Epoch: 02, Test: 0.0962
Epoch: 03, Test: 0.1008
Epoch: 04, Test: 0.0962
Epoch: 05, Test: 0.1008
Epoch: 06, Test: 0.1075
Epoch: 07, Test: 0.1075
Epoch: 08, Test: 0.1133
Epoch: 09, Test: 0.0905
Epoch: 10, Test: 0.1013
Epoch: 11, Test: 0.0955
Epoch: 12, Test: 0.1013
Epoch: 13, Test: 0.1133
Epoch: 14, Test: 0.1013
Epoch: 15, Test: 0.0987
Epoch: 16, Test: 0.1133
Epoch: 17, Test: 0.1133
Epoch: 18, Test: 0.1133
Epoch: 19, Test: 0.1133
Epoch: 20, Test: 0.1133
Epoch: 21, Test: 0.1133
Epoch: 22, Test: 0.1133
Epoch: 23, Test: 0.1133
Epoch: 24, Test: 0.1133
Epoch: 25, Test: 0.1013
Epoch: 26, Test: 0.1133
Epoch: 27, Test: 0.1133
Epoch: 28, Test: 0.1133
Epoch: 29, Test: 0.1133
Epoch: 30, Test: 0.1133
Epoch: 31, Test: 0.1133
Epoch: 32, Test: 0.1133
Epoch: 33, Test: 0.1133
Epoch: 34, Test: 0.1133
Epoch: 35, Test: 0.1133

My GNN looks like this :

class Encodr(nn.Module):
    def __init__(self):
        super(Encodr,self).__init__()
        self.conv1_out = 32
        self.conv2_out = 64
        self.conv3_out = 64
        self.fc1_hidden = 128
        self.x_lat = 10
        self.kernel_size = 5
        self.pool_reduce = 'max'
        
        self.conv1 = SplineConv(1, self.conv1_out, dim=2, kernel_size=self.kernel_size)
        self.conv2 = SplineConv(self.conv1_out, self.conv2_out, dim=2, kernel_size=self.kernel_size)
        self.conv3 = SplineConv(self.conv2_out, self.conv3_out, dim=2, kernel_size=self.kernel_size)
        self.fc1 = nn.Linear(self.conv3_out, self.fc1_hidden)
        self.fc2 = nn.Linear(self.fc1_hidden, self.x_lat)
    

    def forward(self,d):
        x_conv1 = F.elu(self.conv1(d.x.reshape(-1,1), d.edge_index0, d.edge_attr0))

        x_pool1 = scatter(x_conv1, d.cluster0,dim=0, reduce=self.pool_reduce)
        
        x_conv2 = F.elu(self.conv2(x_pool1, d.edge_index1, d.edge_attr1))

        x_pool2 = scatter(x_conv2, d.cluster1, dim=0, reduce=self.pool_reduce)
        
        x_conv3 = F.elu(self.conv3(x_pool2, d.edge_index2, d.edge_attr2))

        x_pool3 = scatter(x_conv3, d.cluster2, dim=0, reduce=self.pool_reduce)

        x_fc1 = F.elu(self.fc1(x_pool3))
        
        x_fc2 = self.fc2(x_fc1)

        softmax = F.log_softmax(x_fc2, dim=1)

        return softmax


def train(epoch):
    model.train()

    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()

        F.nll_loss(model(data), data.y).backward()

        optimizer.step()

I though it should work but I don't know what was the problematic one (it's so frustrating). Is there any wrong point here?

[rusty1s]

Thanks for reporting. Can you share the pre-processing of the dataset as well? I can try to reproduce it.

[rusty1s]

Have you visualized the pyvista clustering and confirmed its correctness? For graclus, you should be able to directly make use of our mnist_graclus.py example (where you simply replace the edge_index with the edge_index coming from the Delaunay triangulation.

[hkim716]

Thanks Matt.
What is the difference between new_x = scatter(data.x, cluster, dim=0, reduce='max') and new_data = max_pool(cluster, data, transform)?
I guess they both are doing max_pooling, but the output is different.
new_x is just new node feature values, and new_data has new node feature x, new edge_index, and new edge_attr.
Is that correct? In which case should we use max_pooling vise versa?

[rusty1s]

That is correct. max_pool will also modify node positions and graph structure based on cluster, while new_x just denotes the pooled node features.

[hkim716]

Understood.
I investigated the problem that does not reduce loss values with Delaunay transformed triangular mesh data.
The difference between the original MNIST superpixel data and my mesh data is :
my mesh data does not have that many edges compared to the original superpixel dataset. See this picture.

Moreover, most of node feature x are zero, except for about 20 node points, which might significantly reduce the performance during pooling by using pre-scheduled clusters.
Without that many edges compared to original superpixel dataset, is it reasonable to assume that pooling performance can be reduced significantly?

[rusty1s]

I think most pooling approaches will maintain most edges present in the original graph (new edges are computed by merging all edges within the same cluster). I definitely agree though that pre-computed pooling might fail here. To counteract, you may want to leverage more GNN layers between two intermediate pooling steps. This way, information of individual nodes can get better distributed.