pl.Callback using a nested pl.Trainer

[gianmarcoaversanoenx]

In a project, I need to evaluate some metrics during training. Thus, I use a pl.Callback! Very natural. However, these metrics are evaluated by quickly training another model (pl.LightninModule), using a pl.Trainer. This creates a pl.Trainer nested in another running pl.Trainer via a pl.Callback.

trainer = pl.Trainer(
    ...,
    # this callback will log some metrics after training another model
    callbacks=[MyCallback()],
)

While training, all is good until MyCallback runs. And it runs fine. However, after it does, I get:

def _call_lightning_datamodule_hook(
        self,
        hook_name: str,
        *args: Any,
        **kwargs: Any,
    ) -> Any:
        if self.datamodule is None:
>           raise TypeError("No `LightningDataModule` is available to call hooks on.")
E           TypeError: No `LightningDataModule` is available to call hooks on.

If MyCallback does not use a (nested) pl.Trainer to train the model that estimates the metrics but some custom training loop, I do not get this error. Any reason why this happens? How do I fix it?

[justusschock]

Hi, Nested Trainers are currently not supported as there are a few global torch states we rely on in the trainer which could be mixed this way.

[gianmarcoaversanoenx]

Hello
thanks for replying! Now I know :) I will move on implementing my own "Trainer". I should be able to make do with something simple.