NeptuneLogger raises Error occurred during asynchronous operation processing: Cannot upload file during checkpointing

[dagap]

I am using pytorch lightning and using the Neptune logger in conjunction. So, my trainer setup has the following parameters:

gpus: 0
min_epochs: 1
max_epochs: 3
log_every_n_steps: 1

My ModelCheckpoint callback has the following parameters:

  _target_: pytorch_lightning.callbacks.ModelCheckpoint
  monitor: "train/loss" # name of the logged metric which determines when model is improving
  mode: "min" # "max" means higher metric value is better, can be also "min"
  save_top_k: 1 # save k best models (determined by above metric)
  save_last: False 
  verbose: False
  dirpath: "checkpoints/"
  filename: "epoch_{epoch:03d}"
  auto_insert_metric_name: False

Now, when I run the training, it is running fine but when trying to upload everything to Neptune, it is showing the following errors:

2022-03-31 12:24:12,095][neptune.new.internal.operation_processors.async_operation_processor][ERROR] - Error occurred during asynchronous operation processing: Cannot upload file /Users/infermatica/Dropbox/infermatica/code/ml/outputs/2022-03-31/12-17-06/.neptune/model_default/UN-19/checkpoints/epoch=1-step=9.ckpt: Path not found or is a not a file.
[2022-03-31 12:24:12,129][neptune.new.internal.operation_processors.async_operation_processor][ERROR] - Error occurred during asynchronous operation processing: Cannot delete training/model/checkpoints/epoch=1-step=9: Attribute does not exist
All 61 operations synced, thanks for waiting!

Not sure what is going on. I tried to set the log_every_n_steps to 0 to try and remove intermediate checkpoint logging but that does not seem to work either.

The trainer does upload the final model to the checkpoint store. it seems though it is the intermediate checkpoints that cannot be uploaded.

[Blaizzy]

Hi @dagap

Prince Canuma here, a Data scientist at neptune.ai,

I'm here to help you with this one,

Can you provide me a minimal reproducible code?

[Blaizzy]

Hi @dagap

I'm just checking in to see if you still need help with this one or if you have any other doubts I can help with?