Is strimzi incompatible with K8s 1.21 because of service account token expiry?

[raflesw]

After our K8s environment was upgraded to 1.21, we saw annotations like the following appear:

annotations.authentication.k8s.io/stale-token
subject: system:serviceaccount:eventbus:my-cluster-entity-operator, seconds after warning threshold: 3989

This is due to changes in token expiry in K8s 1.21 as described here:
https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html#troubleshooting-boundservicetoken

It would appear that there is 90d grace period, after which tokens will be rejected.

It looks like the entity-operator needs to use a later client SDK version, or is there a workaround?

[scholzj]

What version of Strimzi you use? I haven't seen this annotation anywhere and I'm not aware of any issues with Kubernetes 1.21 or newer.

[psemeniuk]

@scholzj because fabric8 maintainers decided to not to follow official guides, but instead they implemented fallback on 401. Details here fabric8io/kubernetes-client#3445 (comment)

So finally you will be still getting stale-token warnings, and there is no way to test it without container uptime longer than grace period (90d in EKS / one year in bare k8s).

[tyagian]

What should be the latest version to have this fix? Strimzi 0.25 ?
Any version after Strimzi 0.25 will be getting stale-token warnings?
It seems fabric updated in 0.27.1 also.
https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.27.1

But I see warnings in 0.27.1 also. Any workaround yet?

[chadlangston]

anyone know the answer to @tyagian's question?

thanks!

[scholzj]

I think everything is said above. I'm not aware of any issues with any supported Kubernetes version.

[chadlangston]

cool. so, am i reading the above correctly that the token refresh issue should be fixed in the strimzi kafka operator v0.26 or higher, but that we may still see stale-token warnings due to the method that fabric8 is using to refresh the token?

you seem to suggest that this fix should be present in 0.31 or higher here
#6845

am i confusing two different issues?