Slurm Scheduler doesn't support multi-cluster

[JimPaine]

Slurm Scheduler doesn't support multi-cluster

I am trying to run a series of tests across multiple clusters in Slurm.

• I have accounting configured and can interact with the clusters using sbatch, sinfo, sacct .... using -M flag to specify the cluster
• I use the access property to specify my cluster in my partitions in my reframe cluster config and can see the jobs are submitted to the correct cluster
• when running with verbose output I can see that sacct arguments are pre-defined and there is no way to pass additional values to sacct as seen
  here

Snippet of core/schedulers/slurm.py

completed = _run_strict(
  f'sacct -S {t_start} -P '
  f'-j {",".join(job.jobid for job in jobs)} '
  f'-o jobid,state,exitcode,end,nodelist'
)

Am I missing something obvious? If not could we do something that lets us set additional args in scheduler options per reframe system partition something similar to sched_access_in_submit or addidional_args?

Section of cluster.py

site_configuration = {
    'systems': [
        {

...

            'partitions': [
                {
                    'name': 'clusterA',
                    'descr': 'clusterA',
                    'scheduler': 'slurm',
                    'launcher': 'srun',
                    'environs': ['clusterA'],
                    'access': ['-M clusterA']
                },
                {
                    'name': 'clusterB',
                    'descr': 'clusterB',
                    'scheduler': 'slurm',
                    'launcher': 'srun',
                    'environs': ['clusterB'],
                    'access': ['-M clusterB']
                }
            ]
        }
    ],
    'environments': [
        {
            'name': 'clusterA',
            
            ...
        },
        {
            'name': 'clusterB',

            ...
        }
    ]
}

Cluster A is my default cluster so tests run fine here.

Output of sacct

No -M means it is running on default cluster.

Entering stage: run
Entering stage: run
Entering stage: run_wait
[CMD] 'sacct -S 2025-09-30 -P -j 54 -o jobid,state,exitcode,end,nodelist'

[JimPaine]

@vkarak If this is seen as a useful feature I am more than happy to have a look through the src and docs and contribute back to the project.

[vkarak]

This is certainly a limitation in the framework. You can't modify the sacct command. I'm trying to understand if there is a better way to do it apart from the obvious one of adding yet another configuration option.

[JimPaine]

Far from the final solution (i.e. a bit of an experimental hack), but sharing in the hope it might help with any creativity.

In my very specific example modifying the sacct command to include -L fixed it. That being said that assumes that no job ids cross over between clusters from the timeframe specified in -S

[vkarak]

I think eventually that adding a configuration option relevant only to the slurm backend is probably the best way. It could be something like sacct_extra_opts defined like this:

'partitions': [
    {
        'name': 'foo',
        'scheduler': 'slurm',
        'sched_options': {
            'sacct_extra_opts': ['-L']
        }
    }
]