GPU and special resource requests

[jennydaman]

Currently the ChRIS plugin spec requires the resource-related fields

• max_memory_limit or (MiB, GiB)
• min_cpu_limit (millicores)
• min_gpu_limit (int)

The spec has inadequate detail for representing resource request needs for ML apps (typically GPU). An app might require specifically...

• CPU supports AVX extensions
• GPU has a specific compute capability
• GPU has xGB VRAM

[jennydaman]

SLURM handles this using "generic resources (GRES)."

https://slurm.schedmd.com/gres.html

Similar to ChRIS, it is possible to request from SLURM a specific amount of CPU and RAM. But when it comes to GPU, the situation becomes too complicated so GPUs are supported as a "generic resource." SLURM admin can customize the definition of GRES so that users can request specific GPU models of other GPU properties.