Is it possible to disable CheckpointConnector?

[PALuczak]

Even with checkpoint_callback=False, Trainer appears to be using CheckpointConnector for some reason. Since very occasionally (once every ~30 runs) the checkpoint is either deleted too early or is never created in the first place (no idea which one), the whole experiment fails, as shown in the log below. Since CheckpointConnector does not appear to be doing anything important when running locally, is it possible to eliminate it without breaking the training process?

Traceback (most recent call last):
  File ".\my_code\run_automated.py", line 95, in <module>
    main(experiment, config, dataset)
  File "D:\GIT\my_code\processing.py", line 117, in main
    trainer.fit(model, dataloader_train, dataloader_val)
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 513, in fit
    self.dispatch()
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 553, in dispatch
    self.accelerator.start_training(self)
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 74, in start_training
    self.training_type_plugin.start_training(trainer)
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 111, in start_training
    self._results = trainer.run_train()
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 609, in run_train
    self._pre_training_routine()
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 600, in _pre_training_routine
    self.checkpoint_connector.restore_weights()
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\connectors\checkpoint_connector.py", line 65, in restore_weights
    max_suffix = self.max_ckpt_in_folder(dir_path_hpc, "hpc_ckpt_")
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\connectors\checkpoint_connector.py", line 372, in max_ckpt_in_folder
    files = [os.path.basename(f["name"]) for f in fs.listdir(dir_path)]
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\fsspec\spec.py", line 1122, in listdir
    return self.ls(path, detail=detail, **kwargs)
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\fsspec\implementations\local.py", line 51, in ls
    return [self.info(f) for f in paths]
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\fsspec\implementations\local.py", line 51, in <listcomp>
    return [self.info(f) for f in paths]
  File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\fsspec\implementations\local.py", line 61, in info
    out = os.stat(path, follow_symlinks=False)
FileNotFoundError: [WinError 2] Nie można odnaleźć określonego pliku: 'D:/GIT/my_code/04d2a63662b34828946b5545646c063f.pt'

[carmocca]

So are you saying the file returned by fs.listdir(dir_path) gets removed?
If you can reproduce this, please open an Issue about it. Seems like a bug.

In the meantime, you can patch the problematic function like this:

trainer.checkpoint_connector.max_ckpt_in_folder = lambda *args, **kwargs: None