KUBE_PING throws error if kubernetes token is refreshed

[vinokurig]

Kubernetes refreshes the service account token (located at the path /var/run/secrets/kubernetes.io/serviceaccount/token) before the token expires, see https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#bound-service-account-token-volume. After such refresh a cyclic error appears:

2024-12-09 08:47:24,674[89b4-l7dv4-9011]  [WARN ] [o.j.p.kubernetes.KUBE_PING 115]      - failed getting JSON response from Kubernetes Client[masterUrl=https://****:443/api/v1, headers={Authorization=#MASKED:1590#}, connectTimeout=5000, readTimeout=30000, operationAttempts=3, operationSleep=1000, streamProvider=org.jgroups.protocols.kubernetes.stream.TokenStreamProvider@60c64dcb] for cluster [WorkspaceStateCache], namespace [****], labels [app.kubernetes.io/component=****,app.kubernetes.io/instance=****,app.kubernetes.io/managed-by=****,app.kubernetes.io/name=****,app.kubernetes.io/part-of=****]; encountered [java.lang.Exception: 3 attempt(s) with a 1000ms sleep to execute [OpenStream] failed. Last failure was [java.io.IOException: Server returned HTTP response code: 401 for URL: https://****:443/api/v1/namespaces/****/pods?

To reproduce the issue it is possible to decrease the kubernetes service account token expiration time to 600 seconds, see https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#tokenrequest-api

[rhusar]

Thanks for the report and reproducer @vinokurig, this must be fixed in the next version (and backported).

However, the general recommendation is to use DNS_PING instead.