How to perform ungraceful shutdown

[smolPixel]

My training code is as such:

self.trainer.fit(self, train_loader, dev_loader)
torch.save(self.model.state_dict(), path_model)

However, when I press ctrl+C during training I get a "attempting graceful shutdown", and then the model is saved. I'd like to not save the model or perform any of the code after trainer.fit if keyboard interrupt is performed.

The doc says that "The trainer will catch the KeyboardInterrupt and attempt a graceful shutdown, including running accelerator callback on_train_end to clean up memory. The trainer object will also set an attribute interrupted to True in such cases. If you have a callback which shuts down compute resources, for example, you can conditionally run the shutdown logic for only uninterrupted runs."

However, when I try to access the "interrupted" attribute, I get "no attribute 'interrupted' ".

Is there a way to force the model to perform ungraceful shutdown or to quit after having release resources?

Thank you

[akihironitta]

I'd like to not ... perform any of the code after trainer.fit if keyboard interrupt is performed.

1. Press C-c many times, or
2. Override on_exception and reraise the exception there
   • docs: https://pytorch-lightning.readthedocs.io/en/latest/extensions/callbacks.html?highlight=on_exception#on-exception
   • the error handling all happens at:
     https://github.com/Lightning-AI/lightning/blob/1d59b3f0ce185702d69eedef752363f63bc48012/src/pytorch_lightning/trainer/trainer.py#L649-L670

when I try to access the "interrupted" attribute, I get "no attribute 'interrupted' ".

Here's the trainer's property:

# status in ("initializing", "running", "finished", "interrupted")
trainer.state.status

https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#state