redpanda: `Client auth failure: user 'kubernetes-controller' wrong password`

[chrisseto]

It seems the chart (and/or operator) can get itself into a state where it generates the following log lines:

WARN  2024-11-12 11:32:11,242 [shard 5:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:11,253 [shard 6:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:13,121 [shard 5:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:13,127 [shard 6:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:13,136 [shard 0:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:13,140 [shard 5:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password

This has been discovered in a cloud cluster (5.9.8) and has been reported by a self hosted user in the community slack in both 5.9.8 and 5.9.7.

In both cases, SASL was enabled, admin_api_require_auth was NOT enabled, and users were being specified via a secret created out of band of the chart itself rather than the inline sasl.users option.

---

Manual Recovery for Affected Clusters

Good news: It's pretty easy to fix provided admin_api_require_auth is not set.

• Ensure that admin_api_require_auth is NOT set.
  • If it is please contact support for assistance.
• Make sure redpanda-bootstrap-user has the desired password
• Perform a rolling restart of the redpanda cluster to ensure $RPK_PASS is up to date
• From any redpanda Pod, run rpk acl user update $RPK_USER --mechanism $RPK_SASL_MECHANISM --new-password $RPK_PASS

JIRA Link: K8S-415

[chrisseto]

See also this thread in the community slack. It's been pointed out that fullnameOverride may be the cause of this issue as removing fullnameOverride appears to make the problem go away.

[chrisseto]

This may also be related to #1536

[chrisseto]

So I've got most of the cause of this though I'm not yet certain how the kubernetes-controller user's password ends up incorrect. I know it could happen if the bootstrap user's password is not being set and a tool that doesn't set the helm's upgrade flag (useFlux: false at the time of writing, possible ArgoCD, etc) is used as that would result in generating a new random password.

Otherwise, we see this log spam because:

• The RPK_{USER,PASS,SASL_MECHANISM} environment variables are set whenever auth.sasl.enabled is true
  • The condition should actually be contingent on the admin_api_require_auth cluster configuration.
• Redpanda fully checks authentication regardless of whether or not it's actually enabled.
  • The log line Client auth failure: user '{}', wrong password will only surface when the password stored redpanda-bootstrap-user drifts from what's in the redpanda cluster.
• Our readiness probe shells out to rpk cluster health which is erroneously sending auth at a consistently interval.

Good news: It's pretty easy to fix provided admin_api_require_auth is not set.

1. Make sure redpanda-bootstrap-user has the desired password
2. Perform a rolling restart of the redpanda cluster to ensure RPK_PASS is up to date
3. From any redpanda Pod, run rpk acl user update $RPK_USER --mechanism $RPK_SASL_MECHANISM --new-password $RPK_PASS

Now to figure out how fullnameOverride affects this issue.

[chrisseto]

Working theory: fullnameOverride affects the names of the generated PVCs and the STS itself. Removal of the fullnameOverride parameter effectively creates a new cluster which will have a new password generated and it will therefore be correct.

[chrisseto]

The best mitigation for this issue is to explicitly set auth.sasl.bootstrapUser.password as the core of the issue is helm charts' inability to guarantee secure idempotent generation of random values.

[chrisseto]

Closing as this has been fixed in the linked PR/commits.