Where to transform and inverse-transform

[aurany]

Hi!

I’m working on a LSTM to predict price changes. The data has to be transformed (standardized) when training/validering and later inverse-transformed when predicting in production.

I’m using the LightningModule as well as the LightingDataModule, but I’m not sure where to apply the StandardScaler’s transform and more specifically; where to save the scaler-parameters and where to apply the inverse-transform on the predictions. And ideeas?

// R

[rohitgr7]

Assuming that you are using pytorch TensorDataset, you can apply transform inside setup and inverse transform inside predict_step itself.

[aurany]

Thanks @rohitgr7 for your answer!

Do you mean apply transform in setup 1) inside the LightningDataModule or 2) inside the LightningModule?

If 1 - how do I access the scaler from within the predict_step inside the LightningModule? I guess the best place to fit/save the scaler is somewhere inside the LightningDataModule where all other data related things live? Can I pass the scaler from the LightningDataModule to the LightningModule?

If 2 - how do I access the dataset/batch from within this setup-function? Inside predict_step we have the batch etc but here?

[rohitgr7]

if you are using datamodule so applying transform inside LightningDataModule.setup would be better.

there must be some place where you have all your training data fit train the scalar so you can do a .fit there on the scalar.
you can assign the scaler to some state.. eg.

class LitDataModule(LightningDataModule):
    def __init__(self):
        super().__init__()
        self.scalar = MinMaxScalar()  # or whatever
    def setup(self, stage):
        if state == 'fit':
            train_data = self.scalar.fit_transform(train_data)
            val_data =  self.scalar.transform(train_data)
        ....

and inside LightningModule you can access it using self.trainer:

class LitModel(LightningModule):
    def predict_step(self, batch, batch_idx):
        preds = ...
        preds = self.trainer.datamodule.scaler.inverse_transform(preds)
        return preds

[aurany]

Thank you very much @rohitgr7 !

[denisbeslic]

@rohitgr7 Related question: How can I save the scalar after training and use it when calling inference using LightningDataModule?

class LitDataModule(LightningDataModule):
    def __init__(self):
        super().__init__()
        self.scalar = MinMaxScalar()
    def setup(self, stage):
        if state == 'fit':
            train_data = self.scalar.fit_transform(train_data)
            val_data =  self.scalar.transform(val_data)
        if state =='test':
            test_data =  self.scalar.transform(test) 
            # This will throw "This MinMaxScaler instance is not fitted yet." exception, since we do not save the scalar during training