Non-leader concierge pods permanently fail to load impersonation proxy TLS certificate (impersonation-proxy-serving-cert)

[sedflix]

What happened?

In multi-replica Pinniped Concierge deployments using the impersonation proxy (mode: auto on GKE), non-leader pods permanently fail to load the impersonation-proxy-serving-cert TLS certificate. The DynamicServingCertificateController retries every 60 seconds but never succeeds. This causes tls: internal error for clients when the LoadBalancer routes to a non-leader pod.

Important distinction: There are two separate empty-certificate errors in the logs. Only one is a bug:

Certificate	Behavior	Bug?
concierge-serving-cert	Transient at startup (<2 seconds), resolves once leader populates cert	No — normal startup race
impersonation-proxy-serving-cert	Persistent every 60 seconds on non-leader pods, never resolves	Yes — this is the bug

Non-leader pod logs (persistent, never resolves):

Repeats every 60 seconds indefinitely:

  {"level":"error","message":"Unhandled Error",
   "error":"key failed with : not loading an empty serving certificate from "impersonation-proxy-serving-cert""}

Repeats every ~3 minutes indefinitely:

  {"level":"error","message":"Unhandled Error",
   "error":"impersonator-config-controller: { } failed with:
     [...] write attempt rejected as client is not leader,
     failed to update CredentialIssuer status: [...] write attempt rejected as client is not leader"}

Client-side error (intermittent, depends on LoadBalancer routing):

  remote error: tls: internal error

Leader pod works correctly — acquires lease, loads certs, handles TokenCredentialRequests.

What did you expect to happen?

Non-leader pods should load the existing impersonation-proxy-serving-cert from the Kubernetes Secret (created by the leader) and serve TLS successfully. The ensureTLSSecretIsCreatedAndLoaded() function in impersonator_config.go already has a read-only path for this case — when the Secret exists, it reads from the informer cache (zero writes) and calls loadTLSCertFromSecret(). This path would succeed on non-leader pods, but it is never reached.

What is the simplest way to reproduce this behavior?

1. Deploy Pinniped Concierge on GKE with replicas: 2 and impersonation proxy mode: auto
2. Wait for leader election to complete
3. Check the non-leader pod logs:
   • "impersonation-proxy-serving-cert" errors every 60 seconds (persistent, never resolves)
   • "write attempt rejected as client is not leader" errors every ~3 minutes
4. Run kubectl commands repeatedly — some succeed (hit leader), some fail with tls: internal error (hit non-leader)

In what environment did you see this bug?

• Pinniped server version: v0.40.0 - (also with v0.44.0)
• Pinniped client version: v0.44.0
• Pinniped container image: ghcr.io/vmware/pinniped/pinniped-server:v0.40.0 (also with v0.44.0)
• Pinniped configuration: OIDCIdentityProvider (Dex) → Pinniped Supervisor → Concierge JWTAuthenticator. Impersonation proxy mode: auto. Service type: LoadBalancer (Internal GKE).
• Kubernetes version: GKE 1.31.x (managed control plane)
• Cloud provider: Google Cloud (GKE)

What else is there to know about this bug?

Root Cause Analysis

The doSync() method in internal/controller/impersonatorconfig/impersonator_config.go executes steps sequentially. Service write operations execute before the TLS cert-loading step. On non-leader pods, the leader election middleware rejects the Service write with ErrNotLeader, causing doSync() to return early — the cert-loading code is never reached.

  Step 1: ensureImpersonatorIsStarted()          ✅ read-only, works on non-leader
  Step 2: ensureLoadBalancerIsStarted()           ❌ WRITES to Service → ErrNotLeader → RETURNS EARLY
  Step 3: ensureClusterIPServiceIsStarted()       ⛔ never reached
  Step 4: ensureCAAndTLSSecrets()                 ⛔ never reached ← cert loading here
           └─ ensureTLSSecretIsCreatedAndLoaded()
                └─ loadTLSCertFromSecret()        ⛔ never reached ← read-only, would succeed!

The dynamiccert.Private provider (tlsServingCertDynamicCertProvider) is entirely passive — no informer, no file watcher, no background goroutine. The only way it gets populated is via SetCertKeyContent() inside loadTLSCertFromSecret(), which is gated behind the failing write operations.

On subsequent sync cycles, the informer re-enqueues the controller. But each retry hits the same early-return: Service update → ErrNotLeader → return. The cert-loading code is never reached, regardless of how many retries occur.

Workaround

Setting replicas: 1 on the pinniped-concierge Deployment eliminates the issue by ensuring only the leader pod exists. The startup-transient concierge-serving-cert errors still occur but resolve within seconds once the single pod acquires the lease.

[cfryanr]

@sedflix Thank you for providing all these excellent details in this issue, and thanks for reporting it.

Do you have any idea why this line is not eventually returning nil and therefore is causing the non-leader pod to try to update the Service? After some time (maybe a few seconds), I would except that the leader pod has successfully created or updated the Service such that the non-leader pod would decide that there is nothing that needs update on the Service anymore, and so the non-leader pod would then decide to continue through the rest of the read-only logic that you outlined above without even trying to update the Service.

Unfortunately, there's not a lot of logging of the details of what it is trying to update on that line. It would be nice if the line that says c.log.Info("updating service for impersonation proxy") right before the call to Update also logged the values of the existingService and updatedService variables, so we can see why it thinks there is a difference which requires an update to the Service. Would you like to try to add some more logging there, build your own container image, and try again with that container image deployed? Building the container image can be done using docker build at the top-level directory of the project (which contains the Dockerfile for the server container image).

Of you might even be able to spot the difference without more logging if you're familiar enough with the logic in the createOrUpdateService function.

[sedflix]

Hi @cfryanr, thank you for the detailed follow-up question!

I was able to reproduce and identify why createOrUpdateService never returns nil on the non-leader pod.

The answer to your question: The equality.Semantic.DeepEqual(existingService, updatedService) check at https://github.com/vmware/pinniped/blob/main/internal/controller/impersonatorconfig/impersonator_config.go#L709 always
sees a difference because of the full label replacement on line ~1095:

  updatedService.Labels = desiredService.Labels

In our environment, a Kyverno ClusterPolicy (mutating admission webhook) adds a label to all Services. This creates a permanent diff cycle:

1. existingService from the informer cache has labels: {app: pinniped-concierge, labelx: labelx}
2. updatedService = existingService.DeepCopy() — starts with both labels
   3.updatedService.Labels = desiredService.Labels— overwrites with only {app: pinniped-concierge}, removing the external label
3. equality.Semantic.DeepEqual returns false — the label diff is detected
4. The controller attempts Update → non-leader gets ErrNotLeader → doSync() returns early → TLS cert loading is never reached
5. Even when the leader succeeds and removes the label, the mutating webhook re-adds it immediately
6. Repeat forever

The underlying code issue: The createOrUpdateService function uses two different strategies for externally-managed metadata:

• Annotations: Merge-overwrite with bookkeeping (annotationKeysKey) — preserves external annotations correctly
• Labels: Full replacement (updatedService.Labels = desiredService.Labels) — strips any externally-added labels

This means any external actor that adds labels to the impersonation proxy Service (admission webhooks, Kyverno policies, Gatekeeper, OPA, etc.) will trigger an infinite reconciliation loop, preventing non-leader pods from ever reaching the TLS cert-loading step.

Our workaround: We're excluding Service resources in pinniped namespaces from the Kyverno policy.

Suggested fix: The label handling in createOrUpdateService could use the same merge-overwrite pattern as annotations — only set desired labels additively, and use a bookkeeping annotation to track which labels the controller manages, so it can clean up removed labels without interfering with labels set by other controllers.

[sedflix]

We can close this issue or label it as an enhancement request.

[cfryanr]

@sedflix, thanks for this excellent debugging. Your explanation makes perfect sense and I can see how this Pinniped controller would be accidentally fighting to remove labels that are getting added and re-added by any other controller. Back when we fixed the similar problem for annotations in 2021, I'm not sure why we didn't notice that labels deserved the same fix. That was clearly an oversight.

I'm glad that you found a workaround. Thanks for sharing that too with the community.

I think this should be fixed as you suggested. I hope to find time to do it soon. Let me know if you'd be interested in submitting a PR for it, with units tests similar to those for the annotations. Since labels and annotations are both map[string]string, the production code could maybe be refactored to handle both with some shared helper function.