training_step not defined, but it is implemented

[thistlillo]

Unfortunately, I cannot provide a complete running script due to the way the code is organized, but I will try to provide as many details as possible. The error I am getting is:

pytorch_lightning.utilities.exceptions.MisconfigurationException: No `training_step()` method defined. 
Lightning `Trainer` expects as minimum a `training_step()`, `train_dataloader()` and `configure_optimizers()` 
to be defined.

That is generated in ConfigValidator, specifically at:

def __verify_train_loop_configuration(self, model: 'pl.LightningModule') -> None:
      # -----------------------------------
      # verify model has a training step
      # -----------------------------------
      has_training_step = is_overridden('training_step', model)
      if not has_training_step:
          raise MisconfigurationException(
              'No `training_step()` method defined. Lightning `Trainer` expects as minimum a'
              ' `training_step()`, `train_dataloader()` and `configure_optimizers()` to be defined.'
          )

I modified the function is_overriden(.) to add some outputs (see the comment XXX MOD HERE):

def is_overridden(method_name: str, model: Union[LightningModule, LightningDataModule]) -> bool:
    # if you pass DataModule instead of None or a LightningModule, we use LightningDataModule as super
    # TODO - refector this function to accept model_name, instance, parent so it makes more sense
    super_object = LightningModule if not isinstance(model, LightningDataModule) else LightningDataModule

    if not hasattr(model, method_name) or not hasattr(super_object, method_name):
        # in case of calling deprecated method
        return False

    instance_attr = getattr(model, method_name)
    if not instance_attr:
        return False
    super_attr = getattr(super_object, method_name)

    # when code pointers are different, it was implemented
    if hasattr(instance_attr, 'patch_loader_code'):
        # cannot pickle __code__ so cannot verify if PatchDataloader
        # exists which shows dataloader methods have been overwritten.
        # so, we hack it by using the string representation
        is_overridden = instance_attr.patch_loader_code != str(super_attr.__code__)
    else:
        is_overridden = instance_attr.__code__ is not super_attr.__code__
        # XXX MOD HERE
        print(f"method_name is {method_name}, is_overriden is {is_overridden}")
    return is_overridden

Its output is:

method_name is train_dataloader, is_overriden is True
method_name is val_dataloader, is_overriden is True
method_name is test_dataloader, is_overriden is True
method_name is predict_dataloader, is_overriden is False
method_name is on_before_batch_transfer, is_overriden is False
method_name is transfer_batch_to_device, is_overriden is False
method_name is on_after_batch_transfer, is_overriden is False
method_name is training_step, is_overriden is False

I will now show how the classes are organized. First the model, that is a VGG:

import pytorch_lightning as pl
[...]

class VGG(pl.LightningModule):
   [...]

   def forward(self, x):
       x = self.conv_layers(x)
       x = x.reshape(x.shape[0], -1)
       x = self.output(x)
       return x

   def configure_optimizers(self):
    # XXX check self.paramters: two versions in official docs parameters() and parameters
    optimizer = torch.optim.SGD(self.parameters(), lr=self.config["learning_rate"], momentum=self.config["momentum"])
    lr_scheduler = ReduceLROnPlateau(optimizer, 'min', factor=0.05, patience=5, cooldown=0, verbose=True)
    return optimizer, lr_scheduler

   def training_step(self, batch, batch_idx):
       x, y = batch
       y_hat = self.forward(x)
       loss =  F.cross_entropy(y_hat, y)
       return {'loss': loss}

Data is provided by a DataModule:

from pytorch_lightning import LightningDataModule

class MyDataModule(LightningDataModule):
def train_dataloader(self):
        if self.train_DL is None:
            self.train_DL = self._from_indexes_to_dataloader("train_i.tsv")
        return self.train_DL
    
    def val_dataloader(self):
        if self.valid_DL is None:
            self.valid_DL = self._from_indexes_to_dataloader("valid_i.tsv")
        return self.valid_DL
    
    def test_dataloader(self):
        if self.test_DL is None:
            self.test_DL = self._from_indexes_to_dataloader("test_i.tsv")
        return self.test_DL

The DataModule relies on a pytorch Dataset, whose getitem is:

def __getitem__(self, idx):
      if torch.is_tensor(idx):
          idx = idx.tolist()

      img_name = os.path.join(self.data_dir, self.df.iloc[idx, 0])
      image = Image.open(img_name)
      encoding = self.df.iloc[idx, 1:]
      encoding = torch.tensor([encoding], dtype=torch.float)  # (118,) --> (1, 118), 118 corresponds to |labels|
      if self.transform:
          image = self.transform(image)
      
      return image, encoding

Then there is the script for training:

import pytorch_lightning as pl
[...]
model = VGG(config)
trainer = pl.Trainer(gpus=1, max_epochs=config["n_epochs"], progress_bar_refresh_rate=5)
data = MyDataModule(config, logger=log)
trainer.fit(model, datamodule=data)

I am aware if is very hard to help without running a sample code, but can you please suggest me what I should search in order to understand the cause of this strange error message? All of the previous code (training step, data loading, etc) run fine in pure pytorch.

[awaelchli]

Hmm, this is very odd. I can't see anything obviously wrong with your definitions.
Have you debugged this using the code from the master branch (the is_overridden issue)?