eval mode giving worse loss than training mode.

[dwidemann]

Hi:

I have a GNN model that in eval mode is giving ~10 larger loss than in training. The training set and the test are the same. The model does not do dropout, batchnorm or any other layer that could affect eval/train differences. At least not that I can think of. Has anyone experienced something like this w/ pytorch-geometric?

Thanks

p.s. model details, I'm taking a graph at time t and predicting all of the node features at time t+1.

[rusty1s]

That shouldn't be the case. Can you show me an example of your model?

[dwidemann]

Thank you for getting back to me. Yes.

class MLP(nn.Module):
    #MLP with LayerNorm
    def __init__(self, 
            in_dim, 
            out_dim=128, 
            hidden_dim=128,
            hidden_layers=2,
            normalize_output=True):

        super(MLP, self).__init__()

        layers = [nn.Linear(in_dim, hidden_dim), nn.ReLU()]
        for _ in range(hidden_layers - 1):
            layers += [nn.Linear(hidden_dim, hidden_dim), nn.ReLU()]
        layers.append(nn.Linear(hidden_dim, out_dim))

        if(normalize_output):
            layers.append(nn.LayerNorm(out_dim))

        self.model = nn.Sequential(*layers)

    def forward(self, x):
        return self.model(x)

class NodeProcessor(nn.Module):
    def __init__(self, 
            in_dim_node=128, in_dim_edge=128,
            hidden_dim=128, 
            hidden_layers=2):

        super(NodeProcessor, self).__init__()
        self.node_mlp = MLP(in_dim_node + in_dim_edge,  
            in_dim_node,
            hidden_dim,
            hidden_layers)

    def forward(self, x, edge_index, edge_attr, u=None, batch=None):
        row, col = edge_index
        out = scatter_sum(edge_attr, col, dim=0) #aggregate edge message by target (previously by source, but graphs are now directed)
        out = cat([x, out], dim=-1)
        out = self.node_mlp(out)
        out += x #residual connection

        return out

[dwidemann]

Arg... I just found out that it's a tensorboard issue. Not in the model. Sorry to bother you. Let me close this issue.

[dwidemann]

This issue is closed. The tensorboard losses were different than the model losses.