Extend SLURM jobs ownership at user level

[vcozzolino]

How can we ensure that SLURM jobs submitted via Interlink through K8s by different users (like Bob) are attributed to the correct user within the SLURM system, rather than always appearing as owned by the Interlink process owner (Alice)?

This is particularly relevant when the Interlink SLURM plugin is active and bound to a specific local user on the edge node.

[dciangot]

Right, the model that we are running on Leonardo and Vega HPCs is with a "service" account.

• the k8s admin is responsible to track who owns the pods at any time
• HPC center handshake with the K8s admin (no a priori knowledge of k8s end users)

The service account can be linked to different SLURM accounts, and the plugin/the k8s cluster can authorize users to use one account or the other.

So there is the concept of groups more than single users in the most common use cases we have at the moment.

Nevertheless more fine grain delegation is possible, simply not coded yet into the current Slurm plugin. We have currently some use cases interested in passing down the interlink chain the sbatch option --uid to indicate the user that own the job. Be aware that this remove the capability of accepting k8s users that are not SLURM users (depending on the case it might be what you want to do though).

[vcozzolino]

I understand, could you give me some tech details about this?

By k8s administrator, I suppose you refer to a human operator, or something automated like an Admission Controller validating submitted tasks?

Also, by service account, you mean a proper k8s SA? So it involves the use of RBACs, context, cluster role etc..

I also didn't fully get how this part works in practice: "The service account can be linked to different SLURM accounts, and the plugin/the k8s cluster can authorize users to use one account or the other."

I'm just trying to understand how this all fits together :)

PS: When you talk about the --uid, is it the same as the --account parameter passed to the SLURM option in the YAML file?

[dciangot]

So, yes, by k8s admin I mean the human administrator.

Service account: I mean an account on the SLURM side, that is not linked to a single person, but rather a single group/organization. I can be linked to differente SLURM username, and can submit with differente SLURM accounts.

At that point, you expose a kubernetes cluster with the services users want. The responsibility to give the correct access to the right SLURM account, is "delegated" to the k8s administrator.

--uid is different from --account. --account is indicating which computing allocation you are accessing, but the UID of the job submitted will stay the one with which you do the srun or sbatch command. The --uid is changing the UID of the processes that you submit with srun and sbatch, saying nothing on the allocation of the computing resources you need.

Please, tell me if there is anything unclear, don't hesitate.

[vcozzolino]

Thanks a lot for the detailed explanation; it helped me better understand the SLURM/HPC side, which is quite new to me compared to k8s.

I will close the discussion for now, but if I have further questions, I'll return here :)

[dciangot]

Yeah don't worry, and don't mind closing this, I'd keep it for a proposal about UID management, that we need in any case!

[dciangot]

Proposal for UID management

This solution is to allow use cases where the SLURM plugin is running as root user, so it needs to "impersonate" a real UID to submit jobs.

The value should be taken like follows:

• Looking at default value of UID in a new slurm configuration field defaultUID
• Overwrite that value if explicitly set on spec.securityContext.runAsUser field in the pod

Manage all the filesystem operation of the plugin (in preparation of the sbatch command) setting the correct ownership of the folders. (Bonus) A useful thing would be to have a group with some GID, to allow interLink components the readonly access to such dirs.