Multi-Node distributed training with SLURM

[hachreak]

Hi everybody,
I'm wondering how can I run a distributed training on 2 machines from a cluster managed with SLURM?
I read that you have to manually specify the machine rank when you run the tools/train_net.py script, but in my case it should be managed by SLURM.
Somebody had the same issue? 😄

[ppwwyyxx]

You can obtain ranks from slurm and pass it to the script (or modify the script).

[hachreak]

You can obtain ranks from slurm and pass it to the script (or modify the script).

I tried this but without success:

python tools/train_net.py \
    --num-machines 2 \
    --machine-rank $SLURM_PROCID \
    --config-file $CONFIG_FILE \
    --num-gpus 2 \
    OUTPUT_DIR $WORKING_DIR

[JiweiMaster]

hello, Did you succeed to use slurm? @hachreak

[hachreak]

hello, Did you succeed to use slurm? @hachreak

Actually not.. in the end I used only one machine because I had no time to investigate better on how to run on multiple machines.. sorry.