Define a policy for requirements to plugins for PVC custom management and defaults

[landerlini]

Having multiple sidecars for multiple backend is fantastic, and kubernetes provides a detailed standard to which they will all do their best to adhere.

Unfortunately, some "special usecase" for our workflows escapes the standard and we need to define a common "language" to enable and/or configure each specific feature.

A few examples:

• in the slurm plugin, we define the queue of the target batch system using annotations
• in the docker plugin, we embed the ssh-forwarding for interactive jupyter sessions in the docker image
• in the kubernetes plugin, we mount cvmfs based on the name of the volume

It would be nice to converge towards a common mechanism and a well defined space of configurations we tend to move towards while developing plugins.

[dciangot]

@Bianco95 @Surax98 can you summarize the status on this? What's the plan for the first implementation?

[dciangot]

I think this is more appropriate as a github discussion, I don't think we ever reached a definiteve answer here.

[dciangot]

Time to resume this, @landerlini I'm drafting a first proposal for each of the points you mentioned above. Please let me know if they still valid or any other came up

[dciangot]

Starting from these:
https://agenda.infn.it/event/43867/contributions/246703/attachments/127247/188196/Offloading%20Flash%20Simulation%20steps%20with%20InterLink.pdf
https://codimd.infn.it/OesOleuQQFqEWkukpGAJlQ#

[dciangot]

custom volume management example here: https://github.com/interlink-hq/interlink-slurm-plugin/blob/main/pkg/slurm/prepare.go#L403-L422

[dciangot]

submission defaults based on requested resource (gpu, no gpu, cpu/mem ratio, other custom resource limits?)

[dciangot]

in general aiming to get a pod working with all backends without passing different annotations

[dciangot]

Proposal

Custom Volume logic implementation

We should allow custom CSI storage provider scripts

Implementation

• A configmap containing a bash (?) script (or a direct string with the script) to provision the volume inside the interLink payload.
• An interlink storage class can also be needed for other use case, in addition to a plain csi classs that just check for existing configmap o script string before creating the "dummy" Pod volume
• optional: The pod indicate a PVC with the indicated storage class to be attached.
• Virtual kubelet include the script into the "containers" field of the interlink requests (same thing we do for configmap) so that the plugin knows what to put in the job script before running

Example:

spec:
  volumes:
    - name: some-offloadable-volume
       csi: 
         driver: vk.io/interlink
         volume_attributes: 
           script: |
             ...
             sshfs ...

• hostPath volume should be added with a withelist of paths set at plugin level

Questions

• for k8s to k8s: we need a passthrough storage class. To allow PVC to be mounted and created only remotely
• cvmfs: a bit special, because different centers might already have something native. To be provisioned as the proposed implementation and override at plugin level if needed
  We should check if we can interpret the CSI provider already created for cvmfs and evaluate it at plugin level.

Tracking issue

https://github.com/interTwin-eu/interLink/issues/399

Plugin defaults

The submission/execution parameters should have defaults based on payload resource requests:

• N GPU vs no GPU workloads
• FPGA
• CPU/RAM ratio
• custom pod resource requirements: high-io (configure the job)

In other words, defining payload flavors. N.B. we will start from static defaults at plugin level, then adding dyamic flavor match, based on pod resources.

We should have already examples to be followed

Implementation

At plugin configuration there should be a dictionary with a set of default annotations (flavors) for each of the use cases above. e.g.:

defaults:
  GPU:
    singularity-options: --nv
    slurm-options: --gres gpu:$N_GPUS
  high-io:
    singularity-binds: --bind /nvme/:$SLURM_HOME

At first implementation we might need a simple concatenation of strings? Maybe later we will discover a smarter way to merge cases where multiple options from the list are met.

We will likely start from SLURM and Docker plugins.

Dynamic allocatable resources for virtual nodes

This is already supported by the worflow. The plugin implementation is not there yet though.
https://github.com/interTwin-eu/interLink/blob/main/pkg/virtualkubelet/virtualkubelet.go#L504-L509

Implementation

The plugin should answer to the ping requests with a valide allocatable resource dictionary.

[landerlini]

Hey,
thanks for putting this together. Some first-read comments.

Custom Volume logic implementation

We should allow a PVC storage class <-> PVC implementation script mapping.

I would not restrict to PVC, as the same mechanism may apply to other V1PodVolumes, but I agree.
I guess the point here is that apptainer and singularity are instrumented to enable mounting fuse volumes without root privileges and we should define a semantic to leverage that feature to represent volumes in a standard in-cluster implementation.

Implementation

* A configmap containing a bash (?) script to provision the volume inside the interLink payload.

ConfigMap make the submission part significantly more complicated.
Consider that the bash script will include user-specific secrets that should percolate from the user environment to the job at submission time. The simplest implementation would be that for each job the submission mechanism also generates and creates a configmap that is then bound to the job. I think this is an overkill and the (ba)sh script should be included as part of the V1Pod API.

The condorce-plugin used annotations in the V1Pod instance, which is a lazy catch-all solution, which is however not readily viable for a Kubernetes-based target (though I guess the kubernetes plugin could process that entity as well).
The idea here is that the annotation contains instructions to override the behavior of a volume. If the pod is executed in-cluster, the annotation is simply ignored and the V1VolumeSource in the pod-spec is used to provision the volume. If the job is off-loaded the plugin takes care of actively ignoring the volumes identified in the annotation, replacing them with a generic fuse-volumes.

An alternative probably cleaner option would probably be to pass the bash script as a field of the V1CSIVolumeSource API. This would allow to have in one's pod something on the line of:

spec:
  volumes:
    - name: some-offloadable-volume
       csi: 
         driver: vk.io/interlink
         volume_attributes: 
           script: |
             ...
             sshfs ...
       

However, this would require writing a (probably very simple) csi driver to ensure the pod is executable in-cluster.

* An interlink storage class that just check for existing configmap before creating the "dummy" PV

Here I don't understand what exactly you have in mind. In general I would avoid creating persistent volumes connected to interlink jobs.

* The pod indicate a PVC with the indicated storage class to be attached.

* Virtual kubelet include the script into the "containers" field of the interlink requests (same thing we do for configmap) so that the plugin knows what to put in the job script before running

Questions

* for k8s to k8s: we need a passthrough storage class. To allow PVC to be mounted and created only remotely

This is correct. k8s -> k8s is actually a simpler case because there is still a plugin in the middle that can interpret the volume specification, but we should support in-cluster execution of the V1Pod objects we generate.

* cvmfs: a bit special, because different centers might already have something native. To be provisioned as the proposed implementation and override at plugin level if needed

In general, one can mount up to a finite number of fuse volumes per Linux kernel instance (the limit should be 128 if my memory is correct). So in general if an agreement exists between the cluster-owner and the target provider, it is a better idea to provision volumes at host level and let containers to access them via simple binding. This is indeed customary with cvmfs.
We may design some kind of fall-back solution that mounts the fuse volume only if a given path is not available in the host.
However this may have severe security implications as we would be enabling the submitted Pods to mount arbitrary paths of the target provider.
I agree cvmfs may be treated separately, though a specification uniform among plugins for a pod to request cvmfs should exists.

Plugin defaults

I have the impression this section mixes two aspects (both correct, but with different implications).
On one hand, we want a job without site-specific annotations to run, taking the default partition, account and quality of service from the plugin configuration and not from the podspec. On the other, if we have options in the target site, we want a mechanism to make those default as cheap as possible. This yes: based on the podspec.

If pod configurations are unavoidable, they should be read from the pod tolerations (even if we do not define explicitely a taint in the plugin, it's only a matter of where the information is stored in podspec) because tolerations can be configured for example through Kueue after the matching between the podspec and a resource flavor, while annotations cannot.

The submission/execution parameters should have defaults based on payload resource requests:

* GPU vs no GPU workloads

Binary selection proved to be not sufficient as payloads may require more than a GPU. So this should really read as number of GPUs.

* FPGA

I have no experience of multi-FPGA setup, but why not... let's make this an integer as well.

* CPU/RAM ratio

* custom pod resource requirements: high-io (configure the job)

For consistency with Kubernetes (and Kueue) scheduling, this should be implemented with the taint-toleration mechanism. If a pod does needs high-io, it should not include low-io among its tolerances.
At least in Kueue, this can be managed out-of-the-box with "resource flavor tolerations".

Implementation

At plugin configuration there should be a dictionary with a set of default annotations for each of the use cases above. e.g.:

defaults:
  GPU:
    singularity-options: --nv
    slurm-options: --gres gpu:$N_GPUS
  high-io:
    singularity-binds: --bind /nvme/:$SLURM_HOME

At first implementation we might need a simple concatenation of strings? Maybe later we will discover a smarter way to merge cases where multiple options from the list are met.

We will likely start from SLURM and Docker plugins.

@Bianco95 has implemeted for another plugin, a "slurm flavor" mechanism that enables configuring the plugin with a list of "flavors" plus a default.
Every flavor declares a maximum amount of runtime, CPU, memory, number of GPUs and taints it would be capable of provisioning, and configuration keys.
Flavors must be sorted from the cheapest to the most expensive at configuration time.
When the pod reaches the plugin, the flavors are compared to the podspec in order, the first one matching is selected and its configuration keys are used to override the default.

For example, you may have:

 - max_gpu: 0
   max_cpu: 1
   taints: 
    - slow-io=true:NoSchedule
    partition: potato
    account: farmer

 - max_gpu: 0
   max_cpu: 1    
   partition: ssd
   account: farmer
 
 - max_gpu: 1
   max_cpu: 8
   partition: gpu
   account: hpc
   gres: "gpu:1"

# default (= The most expensive)   
partition: gpu
account: hpc
gres: "gpu:4"

If job requires a single cpu, but does not tolerate slow-io, it gets mapped to partition ssd with account farmer.
If it also requires a gpu, then it gets mapped to hpc.
If it requires 2 gpu, then it assigns 4 gpus to the slurm job.

This makes the configuration rather verbose, but enables a decent matching between kubernetes and slurm semantics.

In addition, one should pick all the available information from the standard V1PodSpec API. For example, --time in slurm means something very similar to what Kubernetes calls active_deadline_seconds (though --time is in minutes).

Dynamic allocatable resources for virtual nodes

This is already supported by the worflow. The plugin implementation is not there yet though. https://github.com/interTwin-eu/interLink/blob/main/pkg/virtualkubelet/virtualkubelet.go#L504-L509

Implementation

The plugin should answer to the ping requests with a valide allocatable resource dictionary.
Great news! Please include in the logic node labels and node taints that are used, for example, to communicate to the cluster which kind of hardware accelerator is available in the remote node. The most critical information for GPUs is the card memory as large models requires large memory and simply cannot tolerate low-memory cards. I guess analogy exist for FPGA, though the xilinx node-feature-discovery tool does not label nodes with standardized labels as nVidia does.

[dciangot]

Hey, thanks for putting this together. Some first-read comments.

Custom Volume logic implementation

We should allow a PVC storage class <-> PVC implementation script mapping.

I would not restrict to PVC, as the same mechanism may apply to other V1PodVolumes, but I agree. I guess the point here is that apptainer and singularity are instrumented to enable mounting fuse volumes without root privileges and we should define a semantic to leverage that feature to represent volumes in a standard in-cluster implementation.

Yes, as explained below, I think we are talking the same.

Implementation

* A configmap containing a bash (?) script to provision the volume inside the interLink payload.

ConfigMap make the submission part significantly more complicated. Consider that the bash script will include user-specific secrets that should percolate from the user environment to the job at submission time. The simplest implementation would be that for each job the submission mechanism also generates and creates a configmap that is then bound to the job. I think this is an overkill and the (ba)sh script should be included as part of the V1Pod API.

Submission knows nothing about the config map. It indicates only the kind of storage class you want. The storage class will look into a configmap to get the script now you pass in the annotation. So you keep that script consistent for all your workflows invoking the same storage class.

The condorce-plugin used annotations in the V1Pod instance, which is a lazy catch-all solution, which is however not readily viable for a Kubernetes-based target (though I guess the kubernetes plugin could process that entity as well). The idea here is that the annotation contains instructions to override the behavior of a volume. If the pod is executed in-cluster, the annotation is simply ignored and the V1VolumeSource in the pod-spec is used to provision the volume. If the job is off-loaded the plugin takes care of actively ignoring the volumes identified in the annotation, replacing them with a generic fuse-volumes.

An alternative probably cleaner option would probably be to pass the bash script as a field of the V1CSIVolumeSource API. This would allow to have in one's pod something on the line of:

spec:
  volumes:
    - name: some-offloadable-volume
       csi: 
         driver: vk.io/interlink
         volume_attributes: 
           script: |
             ...
             sshfs ...
       

However, this would require writing a (probably very simple) csi driver to ensure the pod is executable in-cluster.

That's exactly what I meant, with the exeption that I would rather support both plain script and configMap ref for the task.

* An interlink storage class that just check for existing configmap before creating the "dummy" PV

Here I don't understand what exactly you have in mind. In general I would avoid creating persistent volumes connected to interlink jobs.

well, with the CSI implementation above, you are actually binding a volume. So I don't get the point, would be a virtual volume in any casi.

At first implementation we might need a simple concatenation of strings? Maybe later we will discover a smarter way to merge cases where multiple options from the list are met.
We will likely start from SLURM and Docker plugins.

@Bianco95 has implemeted for another plugin, a "slurm flavor" mechanism that enables configuring the plugin with a list of "flavors" plus a default. Every flavor declares a maximum amount of runtime, CPU, memory, number of GPUs and taints it would be capable of provisioning, and configuration keys. Flavors must be sorted from the cheapest to the most expensive at configuration time. When the pod reaches the plugin, the flavors are compared to the podspec in order, the first one matching is selected and its configuration keys are used to override the default.

For example, you may have:

 - max_gpu: 0
   max_cpu: 1
   taints: 
    - slow-io=true:NoSchedule
    partition: potato
    account: farmer

 - max_gpu: 0
   max_cpu: 1    
   partition: ssd
   account: farmer
 
 - max_gpu: 1
   max_cpu: 8
   partition: gpu
   account: hpc
   gres: "gpu:1"

# default (= The most expensive)   
partition: gpu
account: hpc
gres: "gpu:4"

I think we agree, I see only matter of syntax and definition w.r.t. my example that we can address in a implementation issue.

Dynamic allocatable resources for virtual nodes

This is already supported by the worflow. The plugin implementation is not there yet though. https://github.com/interTwin-eu/interLink/blob/main/pkg/virtualkubelet/virtualkubelet.go#L504-L509

Implementation

The plugin should answer to the ping requests with a valide allocatable resource dictionary.
Great news! Please include in the logic node labels and node taints that are used, for example, to communicate to the cluster which kind of hardware accelerator is available in the remote node. The most critical information for GPUs is the card memory as large models requires large memory and simply cannot tolerate low-memory cards. I guess analogy exist for FPGA, though the xilinx node-feature-discovery tool does not label nodes with standardized labels as nVidia does.

Extra taints and label should be already there in the latest pre release

[dciangot]

@landerlini updated the original summary with your comments, please go through it and let me know if it closer to what you pointed out.

[dciangot]

related to csi vs storage class, I think it would be easy to start from csi (more urgent need) then extend to a generic storage class for other use cases

[landerlini]

Ciao @dciangot

Submission knows nothing about the config map. It indicates only the kind of storage class you want. The storage class will look into a configmap to get the script now you pass in the annotation. So you keep that script consistent for all your workflows invoking the same storage class.

The script to be passed may depend on the job. It may include user's secrets or sub-paths, that cannot be generalized in a catch-all valid for all users and all jobs. ConfigMap can contain a template if you wish, but that needs to be parsed with job-specific information, by the jobscript builder, before it is passed to apptainer --fusermount.
My point is that if you implement a mechanism to pass job-specific information to the VolumeSource, then it might be easier to pass the entire script itself as an "argument" (the template would be { .script }).
But I have no objection to the script-template being distributed as a ConfigMap.

well, with the CSI implementation above, you are actually binding a volume. So I don't get the point, would be a virtual volume in any case.
Yes, ok. For in-cluster execution, you would need a helper pod defining a volume which is claimed by pod running the job. To reduce the number of helper pods and volumes, we should probably hash the script that would be passed to apptainer in the interlink scenario and create a helper+volume per hash value with some automatic clean-up policy.

related to csi vs storage class, I think it would be easy to start from csi (more urgent need) then extend to a generic storage class for other use cases
I guess most of the effort would be coding the csi driver which should apply to both scenarios, however V1PersistentVolumeClaimVolumeSource does not define a field for additional arguments, so one should pass job-specific information in another way...

As for what is urgent, consider that the jobscript builder mechanism does currently solve this with annotations overriding any volumesource when PodSpec is processed by the plugin (while using the original one in in-cluster). So as long as we stick to bouncing to the jobscript builder service, this is not preventing any use case.

In the edit of the original document, you have now

hostPath volume should be added with a withelist of paths set at plugin level
but we have not defined what is hostPath.
I suggest editing the previous code block with

spec:
  volumes:
    - name: some-cvmfs-volume
       csi: 
         driver: vk.io/interlink
         volume_attributes: 
           # hostPath is used to look for host-generic mounts of the volume
           hostPath: /cvmfs
           script: |
             ...
             cvmfs2 ...

    - name: some-other-volume
       csi: 
         driver: vk.io/interlink
         volume_attributes: 
           # hostPath is used to look for host-generic mounts of the volume
           hostPath: null
           script: |
             ...
             eos mount ...

We should check if we can interpret the CSI provider already created for cvmfs and evaluate it at plugin level.
plugins should not rely on the name of the persistentVolumeClaim used to mount cvmfs by the cluster to identify a cvmfs volume.
You can identify cvmfs volumes as JobSpec annotations: e.g. https://github.com/landerlini/interlink-condorce-plugin/blob/main/natsprovider/apptainer_cmd_builder/from_kubernetes.py#L135

Once you identify the VolumeSource mounting cvmfs, plugins can override it (setting a hostPath or whatever) before passing it to singularity.

Plugin defaults

Implementation

Please add a reminder that we are leaving unresolved discussion on the flavor-selection strategy for the implementation issue.