Getting "munmap_chunk(): invalid pointer Aborted (core dumped)" all of a sudden when script gets to self.logger.loghyperparams in trainer.py

[johnmarktaylor91]

Suddenly I am getting the following error when I've never gotten it before, and scripts that worked before no longer work. It happens in the following chunk of code in trainers.py:

    def pre_dispatch(self):
        self.accelerator.pre_dispatch()

        # log hyper-parameters
        if self.logger is not None:
            # save exp to get started (this is where the first experiment logs are written)
            self.logger.log_hyperparams(self.lightning_module.hparams_initial)
            self.logger.log_graph(self.lightning_module)
            self.logger.save()

It gets as far as the line with self.logger.log_hyperparams, and then I get the following error message:

munmap_chunk(): invalid pointer
/home/docker/interpreters/python: line 5: 76255 Aborted                 (core dumped) python "$@"

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Unfortunately it gives no line number so tracking this down has been painful. Does anyone have any idea where this could be coming from? The relevant version numbers are:

PyTorch: 1.6.0+cu101
Lightning: 1.2.4
Python 3.7.10
OS: Ubuntu 18.04

[tchaton]

Dear @johnmarktaylor91,

Any chance you could comment the lines in PyTorch Lightning and see which one triggers the error. My guess is self.logger.log_graph(self.lightning_module).

Best,
T.C

[johnmarktaylor91]

Reinstalling PyTorch fixed it! But before that I determined with a debugger that yes, that was the line that triggered it.

[tchaton]

Great it is working now.