What's the proper way to use auto-submit with SLURM?

[ShihengDuan]

Hi all,

I'm using PyTorch Lightning with SLURM on a server. The slurm submission script is the following:

#!/bin/bash
#SBATCH -A m2637_g
#SBATCH -C gpu
#SBATCH -q debug
#SBATCH -t 00:03:00
#SBATCH -N 1 
#SBATCH -J envir
#SBATCH --error=%x-%j.err
#SBATCH --output=%x-%j.out
#SBATCH --gpus 4
#SBATCH --signal=SIGUSR1@90

export SLURM_CPU_BIND="cores"
module load python
source activate duan-torch
echo "Your job is running on" $(hostname)
python -u train_unetframes.py

And the trainer is simply the following:

model = UNet(in_channels=len(forcings)*seq_len+len(static),
             out_channels=1, k=k, s=s, k_list=kl, mask_on=True, frames=True)

trainer = pl.Trainer(max_epochs=50, gpus=[0],
                     gradient_clip_val=1., callbacks=[checkpoint_callback], plugins=[SLURMEnvironment(auto_requeue=True)])
trainer.fit(model, loader)

But it fails to resubmit when the script ends due to the time limit. Any help will be appreciated!

Thanks!

[ShihengDuan]

I have checked the file for SLURM_environment and the parameter auto_requeue seems not used in the code. Any idea why?

Thanks!

[ananthsub]

It is used here: https://github.com/PyTorchLightning/pytorch-lightning/blob/0fe3379fa4fddd059df50979cab80d4f26fc7411/pytorch_lightning/trainer/connectors/signal_connector.py#L52

[ShihengDuan]

Thanks! And I see the slurm_sigusr1_handler_fn function to requeue the job. But it didn't work with my SLURM script. The job would stop due to the time limit and there is no resubmission. Any idea why?

Thank you!