Secure ClickHouse Password Management Using Environment Variables and External Secrets

[slyt]

Issue

Currently the clickhouse.config.users.appUserPassword and otelUserPassword are configured in plaintext in values.yaml (likely stored in github for config-as-code). They are then templated into users.xml and stored in a ConfigMap on the cluster.

The above approach is secure if Clickhouse remains a ClusterIP, however, if the service type for Clickhouse is changed to NodePort or LoadBalancer, then it will be exposed outside of the K8s cluster. Doing so causes Clickhouse be insecure because the passwords are stored in plaintext, both in values.yaml (likely in github) and in the users.xml ConfigMap (not a secure K8s Secret).

Proposed Solution

Add support for external secrets while maintaining backward compatibility with the current inline password approach.

Recommended Changes

1. Update values.yaml
   Add external secret configuration options:

clickhouse:
  config:
    users:
      # Existing inline password support (for backward compatibility)
      appUserPassword: "hyperdx"
      otelUserPassword: "otelcollectorpass"
      otelUserName: "otelcollector"
      
      # NEW: External secret support
      useExternalSecret: false
      externalSecret:
        name: ""
        appUserPasswordKey: "app-user-password"
        otelUserPasswordKey: "otel-user-password"

2. Modify ClickHouse Deployment clickhouse-deployment.yaml
   Add environment variables that read from secrets when external secrets are enabled:

# Add to the ClickHouse container env section:
env:
  - name: CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT
    value: "1"
  {{- if .Values.clickhouse.config.users.useExternalSecret }}
  - name: CLICKHOUSE_APP_USER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: {{ .Values.clickhouse.config.users.externalSecret.name }}
        key: {{ .Values.clickhouse.config.users.externalSecret.appUserPasswordKey }}
  - name: CLICKHOUSE_OTEL_USER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: {{ .Values.clickhouse.config.users.externalSecret.name }}
        key: {{ .Values.clickhouse.config.users.externalSecret.otelUserPasswordKey }}
  {{- end }}

3. Update users.xml
   Modify to support both password methods:

<app>
    {{- if .Values.clickhouse.config.users.useExternalSecret }}
    <password from_env="CLICKHOUSE_APP_USER_PASSWORD" />
    {{- else }}
    <password>{{ .Values.clickhouse.config.users.appUserPassword }}</password>
    {{- end }}
    <!-- rest of app user config -->
</app>

<{{ .Values.otel.clickhouseUser | default .Values.clickhouse.config.users.otelUserName }}>
    {{- if .Values.clickhouse.config.users.useExternalSecret }}
    <password from_env="CLICKHOUSE_OTEL_USER_PASSWORD" />
    {{- else }}
    <password>{{ .Values.otel.clickhousePassword | default .Values.clickhouse.config.users.otelUserPassword }}</password>
    {{- end }}
    <!-- rest of otel user config -->
</{{ .Values.otel.clickhouseUser | default .Values.clickhouse.config.users.otelUserName }}>

4. Update OTEL collector Deployment otel-collector-deployment.yaml
   Replace the plain text password with secret reference:

# Replace the existing CLICKHOUSE_PASSWORD env var:
- name: CLICKHOUSE_PASSWORD
  {{- if .Values.clickhouse.config.users.useExternalSecret }}
  valueFrom:
    secretKeyRef:
      name: {{ .Values.clickhouse.config.users.externalSecret.name }}
      key: {{ .Values.clickhouse.config.users.externalSecret.otelUserPasswordKey }}
  {{- else }}
  value: {{ .Values.otel.clickhousePassword | default .Values.clickhouse.config.users.otelUserPassword }}
  {{- end }}

5. Remove the now unused secrets.yaml

# DELETE this entire secret block:
---
apiVersion: v1
kind: Secret
metadata:
  name: {{ include "hdx-oss.fullname" . }}-clickhouse-secrets
  labels:
    {{- include "hdx-oss.labels" . | nindent 4 }}
data:
  appUserPassword: {{ .Values.clickhouse.config.users.appUserPassword | toString | b64enc }}
  otelUserPassword: {{ .Values.clickhouse.config.users.otelUserPassword | toString | b64enc }}

6. Update Default Connections values.yaml
   Modify the default connections to support external secrets:

defaultConnections: |
  [
    {
      "name": "Local ClickHouse",
      "host": "http://{{ include "hdx-oss.fullname" . }}-clickhouse:8123",
      "port": 8123,
      "username": "app",
      {{- if .Values.clickhouse.config.users.useExternalSecret }}
      "passwordFromSecret": {
        "name": "{{ .Values.clickhouse.config.users.externalSecret.name }}",
        "key": "{{ .Values.clickhouse.config.users.externalSecret.appUserPasswordKey }}"
      }
      {{- else }}
      "password": "{{ .Values.clickhouse.config.users.appUserPassword }}"
      {{- end }}
    }
  ]

7. Update documentation
   Add examples in the chart's README or values.yaml comments showing:

# Example usage with external secrets:
clickhouse:
  config:
    users:
      useExternalSecret: true
      externalSecret:
        name: "my-clickhouse-passwords"
        appUserPasswordKey: "app-password"
        otelUserPasswordKey: "otel-password"

[dhable]

Thanks for the detailed write up @slyt.

A minor suggestion would be allowing multiple external secrets so the external name/key can be different for app password and otel password. Something like:

# Example usage with different external secrets:
clickhouse:
  config:
    users:
      useExternalSecrets: true
      externalSecrets:
        appUserRef:
           name: "clickhouse-app-passwords"
           key: "app-password"
        otelUserRef:
           name: "clickhouse-otel-passwords"
           key: "otel-password"

I suspect that restricting the external references to use the same named object would be insufficient in all cases.

It seems like this is a risk of helping users make security vulnerable deployments. We should also be flagging the older properties as deprecated and make the breaking change to remove support for them in the followup release.

Let me create a backlog ticket for the team. PR's also welcome 🙂

[bueti]

I'd prefer to keep external secrets as an optional feature rather than a requirement. We should still use valueFrom.secretKeyRef to prevent passwords from being exposed in logs or output.
This approach would already address the needs of users who rely on helm-secrets for encrypting sensitive values.

[brightsparc]

+1 for this feature, is there a PR for this yet? Right now i'm merging an external secret over the initial installed secret which is not desirable.