Running on LUMI HPC?

[SilverSulfide]

Has anyone tried to run axolotl finetuning on an AMD HPC that uses slurm, in particular LUMI? If so, could you share your experience and provide some tips and suggestions? Context: I am currently pre-training a 30B model using GPT-neox on LUMI and it took a lot of work to configure that framework and it is still somewhat faulty, so a head-start on any potential axolotl issues would be appreciated. Thanks!

[NanoCode012]

We have a contributor that added their workflow to the docs for amd HPC https://axolotl-ai-cloud.github.io/axolotl/docs/amd_hpc.html

[SilverSulfide]

We have a contributor that added their workflow to the docs for amd HPC https://axolotl-ai-cloud.github.io/axolotl/docs/amd_hpc.html

Thanks, they seem to point out that deepspeed was not working. Do you know if any changes have been made to address that? LUMI does not allow torchrun and accelerate is too slow usually.

[NanoCode012]

They may have used FSDP instead ? I haven’t heard anything new on this front

[NanoCode012]

If you do give it a quick try, let us know how it goes please