Provide serviceAccount token as token file in Slurm

[antoinetran]

Short Description of the issue

When serviceAccount field is filled (even for default one called "default"), the pod in Kubernetes automatically receive the token to talk to kubernetes API

ls -al /var/run/secrets/kubernetes.io/serviceaccount/
drwxr-xr-x    2 root     root           100 Dec 19 13:21 ..2024_12_19_13_21_43.446791399
lrwxrwxrwx    1 root     root            31 Dec 19 13:21 ..data -> ..2024_12_19_13_21_43.446791399
lrwxrwxrwx    1 root     root            13 Dec 16 18:01 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root            16 Dec 16 18:01 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root            12 Dec 16 18:01 token -> ..data/token

Additionally, the kubelet automatically add environment variables that contains Kubernetes API address and port (thanks to https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/kubelet/envvars/envvars.go#L45-L48):

KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_HOST=10..... (some ip)

There are other KUBERNETES variables, but these two are the minimum for my test case, Argo, to be able to communicate to Kubernetes API.

Environment

• Operating System: Debian 13
• Other related components versions: Kind 1.25

Steps to reproduce

deploy any pod, even with the default serviceAccount, this should not be in error

ls -al /var/run/secrets/kubernetes.io/serviceaccount/
env | grep KUBERNETES_

Logs, stacktrace, or other symptoms

Summary of proposed changes

See below

[antoinetran]

Some useful info thanks to @dciangot

• where virtual kubelet seems to provide such tokens:
  https://github.com/interTwin-eu/interLink/blob/main/pkg/virtualkubelet/execute.go#L445
• the end of this mechanism Support Bound Service Account Token virtual-kubelet/virtual-kubelet#1012
• maybe a solution with the token API https://kubernetes.io/docs/concepts/security/service-accounts/#get-a-token

[antoinetran]

Ok I see a (long painful) path.

1. Add in InterLink current Role the ability to request serviceAccount token

- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]

2. In presence of this volume (this JSON is generated from volumes.String() and is a bit rearranged)

{
   "Name":"kube-api-access-kzzwz",
   "VolumeSource":"VolumeSource"{
      "Projected":"&ProjectedVolumeSource"{
         "VolumeProjection"{
            "VolumeProjection"{
               "ServiceAccountToken":"&ServiceAccountTokenProjection"{
                  "Audience":,
                  "ExpirationSeconds":*3607,
                  "Path":"token"
               },
            },
            "VolumeProjection"{
               "ConfigMap":"&ConfigMapProjection"{
                  "LocalObjectReference":"LocalObjectReference"{
                     "Name":"kube-root-ca.crt"
                  },
                  "Items":[
                     
                  ]"KeyToPath"{
                     "KeyToPath"{
                        "Key":"ca.crt",
                        "Path":"ca.crt",
                     }
                  },
               },
            },
            "VolumeProjection"{
               "DownwardAPI":"&DownwardAPIProjection"{
                  "DownwardAPIVolumeFile"{
                     "DownwardAPIVolumeFile"{
                        "Path":"namespace",
                        "FieldRef":"&ObjectFieldSelector"{
                           "APIVersion":v1,
                           "FieldPath":"metadata.namespace"
                        },
                     }
                  }
               },
            }
         },
         "DefaultMode":*420
      },
   }
}

• for token, in InterLink VK, request a service account token (see https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/1205-bound-service-account-tokens#example-flow) . For simplicity, boundObjectRef can be null and validityDuration should hard-coded to 99999h, but it could be great if the first is attached to the pod UID. The second variable should stay at 99999h because otherwise it would be a PITA to have InterLink VK to API to Slurm Plugin renew the token periodically.
  This token file should be put in the pod creation request from InterLink VK to API, and API to (InterLink Slurm) Plugin.
  The InterLink Slurm Plugin should put this content to pod directory ~/.interlink/pod-uid-blabla/volumes/kube-api-access-66fbd/token
• for ca.crt, InterLink VK should request content of configMap kube-root-ca.crt present in its namespace (something like this below, but in HTTP request)

kubectl get configmap -n [namespace name] kube-root-ca.crt
NAME               DATA   AGE
kube-root-ca.crt   1      97m

The content should be in pod creation request from VK to API to Plugin.
The InterLink Slurm Plugin should put the content to a file in ~/.interlink/pod-uid-blabla/volumes/kube-api-access-66fbd/ca.crt

• for namespace, in InterLink VK, if we have
  volume.Projected.Sources[0].DownwardAPI.Items[0].FieldRef.FieldPath == "metadata.namespace" then we put the namespace where we are in pod request creation, from VK to API to Plugin.
  The InterLink Slurm Plugin should put this content to pod directory ~/.interlink/pod-uid-blabla/volumes/kube-api-access-66fbd/namespace
  Note: this namespace is not interlink but it is the one where the serviceAccount exist.

3. Then the InterLink Slurm Plugin should add bind path to Singularity/Apptainer, from these file to the inside where the VolumeMounts want. Eg

    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-66fbd
      readOnly: true

is translated to Singularity --bind ~/.interlink/pod-uid-blabla/volumes/kube-api-access-66fbd:/var/run/secrets/kubernetes.io/serviceaccount

[antoinetran]

Ok first draft attempt to get token works!

• modified rbac to add servicesaccounts/token (in interlink helm repo)
• in VK, read pods volumes, and when it is a projected volume of type serviceaccount, token is created thanks to InterLink VK own serviceaccount (https://github.com/antoinetran/interLink/blob/fix_issue45_serviceaccount_token/pkg/virtualkubelet/execute.go#L502)

Now @dciangot I think I need to create token, ca.crt, namespace files and put these in PodCreateRequest (https://github.com/interTwin-eu/interLink/blob/main/pkg/interlink/types.go#L10), I need to add a volumes field I think. What do you think?

No urgency, I will continue the work next year!

[dciangot]

Just one quick comment. The projected accounts token will be rotated each hour.
I think there is no way around it, unless you pre create a token for a service account manually diabling the rotation (and the projection as well)

[antoinetran]

Just one quick comment. The projected accounts token will be rotated each hour. I think there is no way around it, unless you pre create a token for a service account manually diabling the rotation (and the projection as well)

Hi Diego, the correct implementation would be for InterLink VK to do the same as what kubelet does: when 80% of expiration time is passed, then it should reask a new token, and provide them to InterLink API, then Plugin. The Slurm Plugin should have a way to trigger or undergo the rotation (either by listening to API with a port, or by polling regularly, or because it knows when a token should expire).
However that would be too much of work and need further discussion. The benefit of honoring the expiration time is, for the time being (InterLink still being experimental), less than just setting infinite time (something like 100 year). Which is what I am planning to do for now.

[antoinetran]

One thing I am not sure though is, after giving token to Slurm job, how to give it access to Kubernetes API? I suggest exposing Kubernetes API :

• to a NodePort Service. A random port 3XXXX is given
• to an Ingress Service. This is highly dependent on which Kubernetes cluster we are (AWS, GCP, baremetal, etc.)., what loadbalancing service it provides
• though a reverse SSH tunnel from VK to API, then the Slurm job would need access to API to a given port

I feel the easiest way and most compatible way to deploy is the first one (NodePort).

[dciangot]

Just one quick comment. The projected accounts token will be rotated each hour. I think there is no way around it, unless you pre create a token for a service account manually diabling the rotation (and the projection as well)

Hi Diego, the correct implementation would be for InterLink VK to do the same as what kubelet does: when 80% of expiration time is passed, then it should reask a new token, and provide them to InterLink API, then Plugin. The Slurm Plugin should have a way to trigger or undergo the rotation (either by listening to API with a port, or by polling regularly, or because it knows when a token should expire). However that would be too much of work and need further discussion. The benefit of honoring the expiration time is, for the time being (InterLink still being experimental), less than just setting infinite time (something like 100 year). Which is what I am planning to do for now.

As you say @antoinetran , it is so much work and so much of a burden to maintain that I'm not sure it is worth. Just to be aligned, did you check the path where you create the "infinite" service token manually so to avoid the projectedVolume one? I suspect anything else would be an hack around it.

Also, regarding argo workflows, I still think that a custom executor engine would be something to look at, I'll try to give it a look later this week.

[dciangot]

One thing I am not sure though is, after giving token to Slurm job, how to give it access to Kubernetes API? I suggest exposing Kubernetes API :

• to a NodePort Service. A random port 3XXXX is given
• to an Ingress Service. This is highly dependent on which Kubernetes cluster we are (AWS, GCP, baremetal, etc.)., what loadbalancing service it provides
• though a reverse SSH tunnel from VK to API, then the Slurm job would need access to API to a given port

I feel the easiest way and most compatible way to deploy is the first one (NodePort).

Yes, either 1 or 2 I think are good options. That btw, are outside the plugin and interlink ballpark, right?

[antoinetran]

As you say @antoinetran , it is so much work and so much of a burden to maintain that I'm not sure it is worth. Just to be aligned, did you check the path where you create the "infinite" service token manually so to avoid the projectedVolume one? I suspect anything else would be an hack around it.

I don't think we can avoid projectedVolume. To me, the kubelet automatically converts serviceAccount to a projected volumes in the yaml. So to answer your question, the path field always exist.

[antoinetran]

Also, regarding argo workflows, I still think that a custom executor engine would be something to look at, I'll try to give it a look later this week.

Would gladly receive more help, maybe in Slack, but from what I see, it seems to me that maybe some hack can be done with argo plugin executor but it needs lot of work.

[antoinetran]

Yes, either 1 or 2 I think are good options. That btw, are outside the plugin and interlink ballpark, right?

Yes, the more I think of it, the more I think any way to talk to Kubernetes API should be outside InterLink scope. Of course it can provide some example. I initially thought something could be done in the way InterLink already provides bastionSSH, so with a node port and ssh tunnel, we can access Kubernetes API. But that is not always the cleanest way (eg in Kind, the nodeport does not work unless we also map the host port).

[dciangot]

Alright @antoinetran , I converted this into a discussion, so we can define the way to go in here and then open the issues where its needed.

So, let me recap and see if we can agree on this (then I'll circulate the topic in slack to have comments from other interested people):

1. A pod with a projectedVolume is created (default for a service account token)
2. VK will be able to retrieve the required information from the the Kubernetes API server
3. VK will include new volumes in the PodRequest with service account token content (and the expiration time)
4. VK GetStatus will check for (each pod with projected volume) the expiration time (keeping an in memory cache?)
5. If close to the expiration, will call a new "projectedVolumeUpdate" API of interLink that will forward the content of the new token to the plugin.

The implementation form 1 to 3 is the step 0 that we can insert without API changes, IF we make VK killing the pod when token lifetime comes to an end.

Then, 4-5 can come later, in a 0.4.x where also other apis are likely to change.

What do you think?

[dciangot]

And then this provides the token, ca.crt and namespace, but the kubernetes address and port are provided by kubelet with an implicit env variables providing host and port. In the context of InterLink, it is highly dependant of the setup of Kubernetes cluster (ingress, node port, etc). The easiest way to implement that would be, to my mind, to add a static configuration in interlink yaml to provide IP/FQDN and port. if these are present, VK add the two env var.

Yes, what about living it to the plugin config? So it knows how to set that required variables?

[antoinetran]

Regarding the change of API I refer to make the logic of the refresh token generic for all the plugin. I was not referring to the k8s api, but rather the interLink apis... not sure to follow your answer.

Ok, if you meant "without the need to add a new API for refresh token", then yes I agree!

While for the "infinite" expiration as step0 is a big +1 for me. I had missed your suggestion!

Nice!

Yes, what about living it to the plugin config? So it knows how to set that required variables?

The logic of putting the required variables, either for docker or slurm plugin, seems the same for me. We need on one hand, the address(es) and port information that only the k8s admin can provide, that is static, and on the other hand we just add (at least) two new env variables. The env variables of a container is already implemented in VK and propagated to plugins. So to me this configuration is not specific to a plugin and should not be duplicated in each plugin. It should be factorized on VK and be added by kubelet (like the real kubelet does I think) as env vars to each containers. I don't think the pod objects are immutable, so VK should be able to enrich it?

[dciangot]

What about vk pod adding an annotation for kubernete_api_url when creating the pod? Because I was thinking that interLink can be used by multiple vk in principle.

Then the plugins should know where to get it and what to do.

At that point maybe vk could set the env directly maybe? Wdyt @antoinetran ?

[antoinetran]

I don't follow you @dciangot . I didn't know InterLink API could have multiple VK. But let's say we have:
VK1 (url k8s = url_k8s_1)
VK2 (url k8s = url_k8s_2)
VK1 and VK2 sends to one single InterLink API, which sends to one single InterLink Slurm Plugin.

That is the scenario you thought?
Then if one pod from VK1 is sent to API, this pod should already contain url_k8s_1 as url in env var, set by VK1. API will have nothing to do about this env except letting it passed to Slurm plugin. Then the Slurm job will know where is the k8s API URL related to VK1.
In this context, if one pod from VK2 is sent to API, it will be the exact same thing as VK1, except the env will contain the url_k8s_1. The container launched by Slurm will be know that it needs to talk to url_k8s_2.

I don't see how you meant to use the annotation and for what purpose?

[dciangot]

Exactly @antoinetran , you're right, my first tought was about annotations. But VK setting url env it's definitely the way to go as you describe.

P.s. regarding the multiple vk for a single interLink server on the edge. It's actually one of the feature we wanted to allow authenticated share of a single api instance from different k8s clusters

[antoinetran]

What are the environment variables related to Kubernetes API to add? Here are those from any pod, from a Kind cluster:

KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_HOST=10.96.0.1

Some of them are described in official Kubernetes docs. The kubelet code responsible for these: https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/kubelet/envvars/envvars.go#L45-L48

So I will add these to InterLink Config (https://github.com/interTwin-eu/interlink-helm-chart/blob/main/interlink/templates/virtual-kubelet-config.yaml#L7):

KubernetesApiAddr: 10.96.0.1
KubernetesApiPort: 443

If these values are not present, no env will be added by VK.

[dciangot]

Sounds good! @antoinetran sorry for late replying!

Let me know when you have something to test, we have a RayCluster case study for which this will be game changer

[antoinetran]

I am currently testing it. All the mechanism works. But I have an issue with the token right now: it seems it is incorrectly created, or it does not have sufficient right.
When I test the token inside a kubernetes container, I get

root@helloworld-bm7q4:/# kubectl get pod
E0124 12:24:31.459905      23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0124 12:24:31.472631      23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0124 12:24:31.483906      23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0124 12:24:31.497543      23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0124 12:24:31.507695      23 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)

This is a container where only the token file has been replaced at /run/secrets/kubernetes.io/serviceaccount/token

[dciangot]

Which are the permissions bound to the service account in question?