Cant see validation accuracy & loss in validation_step?

[SamPusegaonkar]

Hi everyone,

I am trying to log validation loss and accuracy on my progress bar.

class MyModel(pl.LightningModule):

    def __init__(self, experiment_spec, export=False):
        super().__init__()
      
        self.train_accuracy = torchmetrics.Accuracy()
        self.val_accuracy = torchmetrics.Accuracy()

    def train_dataloader(self):
        """Build the dataloader for training."""
        train_loader = \
            build_dataloader(
            experiment_spec=self.model_config,
            is_train = True,
            batch_size=self.dataset_config["batch_size"],
            workers=self.dataset_config["workers"],
            collate_fn=train_collate_fn)
        return train_loader

    def val_dataloader(self):
        """Build the dataloader for validation."""
        val_loader = build_dataloader(experiment_spec=self.model_config,
                                      is_train = False,
                                      batch_size=self.dataset_config["batch_size"],
                                      workers=self.dataset_config["workers"],
                                      collate_fn=val_collate_fn
                                      )
        return val_loader

    def training_step(self, batch, batch_idx):
        """Training step."""
        data, label= batch
        data = data.float()
        output = self.model(data)
        loss =  F.cross_entropy(output , label) 
        self.train_accuracy.update(output , label)
        self.log("train_loss", loss, on_step=False, on_epoch=True, prog_bar=True, sync_dist=True, batch_size=16)
        self.log("train_acc@1", self.train_accuracy, on_step=False, on_epoch=True, prog_bar=True, sync_dist=True, batch_size=16)
        return loss

    def validation_step(self, batch, batch_idx):
        """Validation step."""

        data, label, _, _= batch
        output = self.model(data)
        loss = F.cross_entropy(output, label)
        self.val_accuracy.update(output, label)
        self.log("val_loss", loss, on_step=True, on_epoch=True, prog_bar=True, batch_size=16 )
        self.log("val_acc@1", self.val_accuracy, on_step=True, on_epoch=True, prog_bar=True, batch_size=16 )
     
    def forward(self, x):
        output = self.model(x)
        return output

I am running this for 2 epochs.
So, far in my progress bar, I can only see a loss. When an epoch finishes I see train loss/accuracy but not validation loss/acc.

Any clue whats happening here?

[SamPusegaonkar]

Machine: Ubuntu 20.04
Pytorch Lightining Version: 1.6.0

[rohitgr7]

the configuration looks correct, can you reproduce this issue using BoringModel?

[SamPusegaonkar]

I am not able to reproduce it using the above script.
My progress bar also doesnt go entirely to 100%. It stops at 38% Could this be a issue with my dataloader?

[SamPusegaonkar]

Epoch 0: 38%|██████████████████████████████████████████▎ | 742/1949 [01:12<01:57, 10.28it/s, loss=9.66, v_num=29, train_loss=10.60, train_acc@1=0.00472] /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:1383: UserWarning: argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.

I get a warning like this and then it jumps to a new epoch

[rohitgr7]

the warning is not related to PL.

if not possible to reproduce using BoringModel, can you share your code with run instructions. Hard to debug without an example.

[SamPusegaonkar]

    trainer = Trainer(gpus=gpus_ids,
                      max_epochs=total_epochs,
                      check_val_every_n_epoch=1,
                      default_root_dir=results_dir,
                      enable_checkpointing=True,
                      accelerator=acc_flag,
                      gradient_clip_val=clip_grad)

    # setup checkpointer:
    checkpoint_callback = ModelCheckpoint(every_n_epochs=1,
                                          dirpath=results_dir,
                                          monitor=None,
                                          save_top_k=-1,
                                          filename='model_{epoch:02d}-{val_loss:.2f}')

    trainer.callbacks.append(checkpoint_callback)
    trainer.fit(my_model)
    trainer.save_checkpoint("example.ckpt")

I think the issue is with my dataloader. What are some steps I can perform to check its validity?

For example, this is a similar thread that discusses validation epoch skipped: #2675

Is is possible that the last batch size can cause the epoch jump/skip issue?
If yes, how can I add that to my custom sampler class?

[Klassikcat]

It seems your code is correct. However, defining custom .info in the lightning module would solve the problem.

    def info(self, dictionary: dict) -> None:
        for key, value in dictionary.items():
            self.log(key, value, prog_bar=True)

then, put this on your validation_step

        self.info(
            {
                f'{stage}_loss': loss, # can put anything you want to display on progress bar.
            }
        )

If this could not solve your problem, then check out the Nvidia NeMo's logger. your issue can be related to Nvidia NeMo's logger.

[SamPusegaonkar]

Unfortunately this did not work. My progress bar for training goes only till 50% and my validation progress bar doesnt even appear.

I'm still debugging with my dataloader

[SamPusegaonkar]

A small update on this: it seems like my train dataloader finishes its job but my val dataloder is not able to iterate.

I checked the size of val loader and its huge. Any way to debug this? @rohitgr7 @Klassikcat

[rohitgr7]

hey @SamPusegaonkar !

can you try running trainer.validate(model) to check the validation loop?

if it still doesn't work or gives any errors, please paste them here.
and if still doesn't work out, please share a reproducible example to check this with BoringModel or any other which is minimal. You can avoid sharing data by generating some random values of the same shape inside your dataset.

[ConnorHeggie]

I have the same issue, was able to work around it with this callback:

  class ResetValDataloader(pl.Callback):
      def on_validation_start(self, trainer, pl_module):
          trainer.reset_val_dataloader(pl_module)