skip gpu partitions when cpu job

[GaspardR]

Is your feature request related to a problem? Please describe.
When jobs are submitted with the automatic selection from the partition config file, score_job_fit is computed for each partition. The partition with the greatest score for that job is used for submission. There's a problem when a job is cpu-only, partitions with gpus are still considered for the partition selection. These partitions usually don't have as many max_threads than cpu-only partitions, thus have a "better" fit and can be selected for submission.
To me, this seems like a waste of compute resources, gpus will be allocated to the job and not be used at all.

Describe the solution you'd like
The function score_job_fit, should have a logic to skip/disadvantage gpu partitions when the job doesn't require a gpu.

Additional context
As an example here's a partition file:

cpubackfill:
    max_mem_mb_per_cpu: 4000
    max_runtime: "1d"
    max_threads: 192
gpubackfill:
    max_gpu: 312
    max_mem_mb_per_cpu: 4000
    max_runtime: "1d"
    max_threads: 64

Here, the partition gpubackfill will be used over cpubackfill.

[GaspardR]

I'm not very familiar with the codebase, but I'd be happy to try and open a PR if this issue is of interest.

[cmeesters]

Thanks for bringing this issue to my attention! I share your view.

The reason why I did not consider flagging gpu partitions as such is that we do not know when gpus are requested: administrators can choose to obfuscate this (i.e. with requiring --gres ).

Reconsidering, we could implement a gpu_only flag (akin to supports_mpi) to be considered. Now, what about people not using gpu? gres can point to anything. Not just a gpu. Should we live with that? (I am starting to consider it futile to support every deviation an admin can possibly think of ...)

If you want that feature soon, please go ahead. I am going on a little holiday after Easter and will not find the time before that. (But I might be able to merge smaller PRs which pass the CI.)

NB Your partition config should provide max_gpu: for the gpu partition. Without it, there is no upper limit.

[GaspardR]

Thanks for you response!

Maybe an additional flag is not necessary and we could infer it from the partition. We could add a naive check after:

snakemake-executor-plugin-slurm/snakemake_executor_plugin_slurm/partitions.py

Lines 547 to 555 in 8f7491f

	gpu_count, gpu_model = parse_gpu_requirements(job)
	if gpu_count > 0:
	if self.limits.max_gpu == 0 or gpu_count > self.limits.max_gpu:
	return None
	score += gpu_count / self.limits.max_gpu
	if gpu_model and self.limits.available_gpu_models:
	if gpu_model not in self.limits.available_gpu_models:
	return None

Something like:

if gpu_count == 0 and self.limits.max_gpu > 0:
    return None

Or if return Noneis too harsh, we could add a penalty to the score eg: score -= 1.

I do think trying to keep up with any and every admin implementation is too much to ask for, the partitions.yaml file should be used to alleviate the guessing game we have to do.

[cmeesters]

right, that would be a simpler solution ... I'm afraid, I will not get to any proper testing until Mid-April. So, certainly, if you test this and do a simple PR, that would be very much appreciated.