ImpersonationProxy strategy configuration

[romanfurst]

From the documentation, I understood that there are two supported strategies: Token Credential Request API and Impersonation Proxy (https://pinniped.dev/docs/reference/supported-clusters/).

I assumed that the required strategy is selected based on the impersonationProxy.mode value in the CredentialIssuer spec (auto, enabled, or disabled).

However, even when impersonationProxy.mode: enabled is set, the pinniped-concierge server still attempts to create a kube-cert-agent Pod on the control plane node.
I briefly looked into the implementation, and it seems that impersonationProxy.mode=enabled does not prevent this behavior. If a control plane with a running kube-controller-manager exists, the first strategy is always chosen.

There might be a use case where a so-called “self-hosted” cluster is used, but the cluster is still not truly yours - for example, it may be managed by another infrastructure team or an outsourced team. In this scenario, you may not be supposed to deploy any workloads on the control plane.

What is the difference between impersonationProxy.mode=enabled and impersonationProxy.mode=auto then? Is this behavior expected, or is it more of a bug/improvement opportunity? Or am I just missing something ?

I tried a quick-and-dirty solution, and it seems it works as I expect:
romanfurst@3d8296b

Thank you for the clarification and discussion.
Best regards,

romanfurst

[cfryanr]

Hi @romanfurst,

Thanks for your question. As you saw, you can force-enable the impersonation proxy by setting its mode to enabled. This will cause it to run even when it seems that it should not be needed. Also as you observed, this does not impact the kube cert agent, which independently does its best to run if possible. So you can have both working approaches on the same cluster, and both will be advertised in the CredentialIssuer's status as successfully working. By default, the pinniped get kubeconfig command will prefer using the kube cert agent mode (also known as the perhaps poorly named TokenCredentialRequestAPI mode), but you can tell it that you would rather use the impersonation proxy mode with the flag pinniped get kubeconfig --concierge-mode ImpersonationProxy.

As you saw, there is currently no way to ask the kube cert agent strategy not to attempt to work. This is because we assume that if it can work, then you probably want to use it. If you wanted to prevent it from working, you could take away one of the permissions that its controller needs to auto-detect that it can work. For example, if you take away the Pinniped Concierge's permission to create Deployments when you're installing the Concierge (see https://github.com/vmware/pinniped/blob/v0.43.0/deploy/concierge/rbac.yaml#L152), then the kube cert agent controller will fail, it would not deploy a kube cert agent Deployment, and it would list the strategy as failed in the CredentialIssuer. Taking away its permission to create Deployments shouldn't hurt anything else because it won't be trying to create any other Deployments. That would be a reasonable workaround. Or you could use a solution like the one that you outlined above.

There might be a use case where a so-called “self-hosted” cluster is used, but the cluster is still not truly yours - for example, it may be managed by another infrastructure team or an outsourced team. In this scenario, you may not be supposed to deploy any workloads on the control plane.

I think we had envisioned that it would probably be that other team that installed and configured Pinniped for you on the cluster, since you need a very high level of permissions on the cluster to install Pinniped. It sounds like you have a slightly different use case than we had imagined, where you are the admin of a cluster but there are certain things that you are not supposed to do by convention even though you have permission to do them. I'd be curious to learn more and I wonder how common that case might be.

[romanfurst]

I think we had envisioned that it would probably be that other team that installed and configured Pinniped for you on the cluster, since you need a very high level of permissions on the cluster to install Pinniped. It sounds like you have a slightly different use case than we had imagined, where you are the admin of a cluster but there are certain things that you are not supposed to do by convention even though you have permission to do them. I'd be curious to learn more and I wonder how common that case might be.

Imagine an enterprise environment where Kubernetes clusters are provisioned and managed by an infra team. The infra team can be internal or outsourced - it doesn’t matter. Their responsibility is to provide and manage Kubernetes and own its lifecycle, ensuring that it is up and running.
Then there is a platform team that turns the plain, provisioned Kubernetes cluster(s) into development and production environments. Platform teams might install and configure things like ingress controllers, service mesh, CI pipelines, CD tools, logs and metrics collectors, user RBAC, Pinniped 🙂, etc. The platform team may require higher cluster permissions to provide these components, but they still do not own the cluster if we are talking about infrastructure setup. There is an agreement with the infra team that you don’t intentionally break the cluster - in other words, the infra team might say: don’t touch the control plane; it’s not your business.
You can think of this setup as managed Kubernetes in a private cloud.
Besides the infra and platform teams, there are developer teams, who are the real end users of the cluster(s). They have restricted permissions and are allowed to perform only a subset of operations, access non-system namespaces, and so on.

That’s kind if the case I was referring to in origin post.

Taking away its permission to create Deployments shouldn't hurt anything else because it won't be trying to create any other Deployments. That would be a reasonable workaround.

To be honest, I tried playing with Pinniped RBAC a little bit before creating this ticket. Removing permissions for listing pods in kube-system resulted in Pinniped Pod going into CrashLoopBackOff, so I thought this was not the right way.
Your advice to cut off the Deployment creation RBAC permission is good idea , though. It’s a bit hacky, add some errors noise in container's logs, but it works - and that’s all I need right now.

So from my side, this discussion can be closed.

If there is anything else I can do for the case discussed in this thread (maybe try to mature my quick-and-dirty solution commit ), or if I can provide more information to make it clear, please let me know.

Thank you.

Best regards,
romanfurst