Constant validation loss

[cwoolfo1]

Hello,
I am training a GRU. The metrics are constant.

I originally believed this was overfitting but I have tried various hyperparameters, regularization, and optimizers.

I believe this is an issue with my code.

class NeuralNetwork(LightningModule):
    def __init__(self, input_size, hidden_size, output_size, num_layers):
        super(NeuralNetwork, self).__init__()
        self.loss = torch.nn.L1Loss()
        self.lr = .0001
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.num_layers = num_layers
        # Above initializes all the variables.
        self.hiddenlayers = nn.GRU(input_size, hidden_size, num_layers=num_layers, batch_first=True)  #, dropout=0.5)
        self.outputlayer = nn.Linear(input_size, output_size)
        # This code will create a stacked GRU with n number of layers.
   
    def forward(self, x):
        x = x.float()
        output, hidden = self.hiddenlayers(x)
        # Pushes input through the stacked GRU.
        # Returns the last GRU layers last hidden state.
        x = F.relu(hidden[-1])
        # Pushes hidden layer from GRU through activation function.
        x = F.softmax(self.outputlayer(x), dim=1)
        # Pushes that hidden state through a linear softmax layer.
        return x
        # Returns the predicted protein coding expression values.
    def training_step(self, batch, batch_idx):
        x, y = batch
        logits = self.forward(x)
        # Run through neural network
        loss = self.loss(logits, y.float())
        self.log("train_loss", loss, prog_bar=True, logger=True)
        # Log the loss for each training step.
        Acc = Accuracy()   #subset_accuracy=True)
        self.log("train_accuracy", Acc(torch.round(logits), y), logger=True)
        # Log the accuracy for each training step.
        f1score = F1Score()   #mdmc_average="samplewise")
        self.log("train_f1score", f1score(torch.round(logits), y), logger=True)
        # Log the f1score for each training step.
        precision = Precision()    #mdmc_average="samplewise")
        self.log("train_precision", precision(torch.round(logits), y), logger=True)
        # Log the precision for each training step.
        recall = Recall()   #mdmc_average="samplewise")
        self.log("train_recall", recall(torch.round(logits), y), logger=True)
        # Log the recall for each training step
        cosine_similarity = CosineSimilarity(reduction = 'mean')
        self.log("train_cosineSimilarity", cosine_similarity(logits.int(), y), logger=True)
        # Computes the cosine similarity
        #r2score = R2Score()
        #self.log("train_R2Score",r2score(logits.int(), y), prog_bar=True, logger=True)
        # Computes the r2 score
        variance = ExplainedVariance(multioutput='uniform_average')
        self.log("train_variance", variance(logits.int(), y), logger=True)
        return loss
    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self.forward(x)
        loss = self.loss(logits, y.float())
        self.log("val_loss", loss, prog_bar=True, logger=True)
        # Logs the loss for each validation step.
        Acc = Accuracy()   #subset_accuracy=True)
        self.log("validation_accuracy", Acc(torch.round(logits), y), logger=True)
        # Log the accuracy for each validation step.
        f1score = F1Score()   #mdmc_average="samplewise")
        self.log("validation_f1score", f1score(torch.round(logits), y), logger=True)
        # Log the f1score for each validation step.
        precision = Precision() #mdmc_average="samplewise")
        self.log("validation_precision", precision(torch.round(logits), y), logger=True)
        # Log the precision for each validation step.
        recall = Recall()   #mdmc_average="samplewise")
        self.log("validation_recall", recall(torch.round(logits), y), logger=True)
        # Log the recall for each validation step
        cosine_similarity = CosineSimilarity(reduction = 'mean')
        self.log("val_cosineSimilarity", cosine_similarity(logits.int(), y), logger=True)
        #r2score = R2Score()
        #self.log("val_R2Score",r2score(logits.int(), y), prog_bar=True, logger=True)
        variance = ExplainedVariance(multioutput='uniform_average')
        self.log("val_variance", variance(logits.int(), y), logger=True)
        return loss
    def test_step(self, batch, batch_idx):
        # Here we just reuse the validation_step for testing
        return self.validation_step(batch, batch_idx)
    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.lr, weight_decay=1e-5)
        return optimizer
# Creates the neural network class
class CustomDataset(IterableDataset):
    def __init__(self, data):
        super(CustomDataset, self).__init__()
        self.X, self.y = data
    def __iter__(self):
        for i in range(len(self.X[:,1,1])):
            yield self.X[i,:,:], self.y[i,:]
    def __len__(self):
        return len(self.X[:,1,1])
# Creates a custom dataset class to train the neural network.
class DataModule(p1.LightningDataModule):
    def __init__(self):
        super().__init__()
        self.input = np.load('4thin.npy')
        self.output = np.load('newOutput.npy')
        # Loads in npy files
        self.output = self.output.transpose(1,0)
        self.input = self.input[:,:,:20]
        self.output = self.output[:416,:]
        self.batch_size = 10
        self.num_workers = 1
        # Set number of samples per batch
    def setup(self, stage):
        if stage == 'fit':
            split = len(self.input[:,1,1]) / 2
            split = int(split)
            #print(split)
            self.trainX, self.trainy = self.input[:split, :, :], self.output[:split, :]
            self.valX, self.valy = self.input[split:, :, :], self.output[split:, :]
            # Splits the dataset into two smaller datasets for training.
        if stage == 'test' or stage is None:
            self.testX, self.testy = self.input, self.output
            # Returns the full dataset for testing.
    def train_dataloader(self):
        data = self.trainX, self.trainy
        return DataLoader(CustomDataset(data), batch_size = self.batch_size, num_workers=self.num_workers)
    # Loads data for training.
    def val_dataloader(self):
        data = self.valX, self.valy
        return DataLoader(CustomDataset(data), batch_size = self.batch_size, num_workers=self.num_workers)
    # Loads data for validation.
    def test_dataloader(self):
        data = self.testX, self.testy
        return DataLoader(CustomDataset(data), batch_size = self.batch_size, num_workers=self.num_workers)
    # Loads data for testing.
parser = ArgumentParser()
parser.add_argument("--gpus", default=0)
parser.add_argument("--num_nodes", default=1)
args = parser.parse_args()
# Pass the num of nodes and gpus as arguments to program
dm = DataModule()
dm.setup('fit')
train_dataloader = dm.train_dataloader()
val_dataloader = dm.val_dataloader()
model = NeuralNetwork(20, 20, 1, 3,)
tb_logger = p1_loggers.TensorBoardLogger("logs/")
# Defines the logger as a tensor board logger for training.
n = 1
trainer = Trainer(num_nodes = args.num_nodes, gpus = args.gpus, logger=tb_logger, accumulate_grad_batches=10, reload_dataloaders_every_n_epochs=n, stochastic_weight_avg=True) #auto_scale_batch_size="binsearch",
# Uses tensorboard to log metrics and gradient accumulation to speed up processes.
trainer.fit(model, train_dataloader, val_dataloader)

Can yall see if there are any issues with my code that is preventing my neural network from training

[akihironitta]

Hi @cwoolfo1, not sure if below fixes your issue, but if you're using metrics from torchmetrics, I believe the right way to use the classes, e.g. Accuracy, F1Score, etc., is instantiate them only once in your LightningModule's __init__.

class NeuralNetwork(LightningModule):
    def __init__(self, input_size, hidden_size, output_size, num_layers):
        ...
+       self.train_accuracy = Accuracy()
+       self.validation_accuracy = Accuracy()

    def training_step(self, batch, batch_idx):
        ...
-       Acc = Accuracy()
-       self.log("train_accuracy", Acc(...), ...)
+       self.train_accuracy(...)
+       self.log("train_accuracy", self.train_accuracy, ...)
        ...

    def validation_step(self, batch, batch_idx):
        ...
-       Acc = Accuracy()
-       self.log("validation_accuracy", Acc(...), ...)
+       self.validation_accuracy(...)
+       self.log("validation_accuracy", self.val_accuracy, ...)
        ...

Would you mind trying this? Also, I would also try to make sure that it's working in a CPU environment first.

For more examples of torchmetrics, see the documentation here: https://torchmetrics.readthedocs.io/en/latest/pages/lightning.html

[cwoolfo1]

I gave this a try and it does not appear to fix the issue.

[cwoolfo1]

This code is being run in a CPU environment

[akihironitta]

The metrics are constant.

Can I ask which metrics are constant?

This code is being run in a CPU environment

By "being run", you mean it works as expected, or you mean the issue persists on CPU?

If you could provide your full script, I can have a look this weekend :)

[akihironitta]

I'd try to debug it by pruning extra Trainer flags to identify the cause of the issue.

[cwoolfo1]

The validation metrics are all constant. I have visualized learning curves on tensorboard and all of the metrics are straight lines. Not a single change in value.

I am not using GPUs yet

I will send you a copy of the script shortly.

[cwoolfo1]

Here is the code I have been running. For some reason the validation metrics never change!

[qsong1012]

Hi @cwoolfo1 did you fix this? I'm running into the same issue.
I even used the training dataloader as the eval loader, the training loss is changing but the val_loss on the training data does not change at all...

[akihironitta]

the training loss is changing but the val_loss on the training data does not change at all...

@qsong1012 I'd suggest checking if the weights change instead of the training loss since the training loss changes because of any randomness in your script, e.g., data augmentation.

[qsong1012]

@akihironitta Good point! I will check it. I'm training transformer models using text and I didn't augment the text data from time to time, to make sure this I took the exact datapoint and predict on the same datapoint after different epochs and it output different things, so I believe my model is training.

I'm suspecting that it's not the actual model issue but more like a MLflow autolog and pl log conflict., now it's logging different validation losses but same accuracy after each epoch... will keep debugging!

[akihironitta]

@qsong1012 If it's reproducible with a simple model please do let us know by creating a GitHub issue!