Decision: Allow adding secrets to the OpenSearch keystore

[siegfriedweber]

Description

OpenSearch uses a keystore (see OpenSearch keystore) to store sensitive data like S3 credentials.

Part of #44

Solution in the OpenSearch Helm chart

The Helm chart provides the setting keystore to specify secrets whose contents are stored in the keystore (see https://github.com/opensearch-project/helm-charts/blob/opensearch-3.1.0/charts/opensearch/values.yaml#L454-L456). For instance:

# secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
stringData:
  s3.client.default.access_key: ...
  s3.client.default.secret_key: ...

# opensearch-helm-values.yaml
---
keystore:
  - secretName: s3-credentials

Solution in the OpenSearch Kubernetes Operator

general:
  keystore:
    - secret:
        name: credentials
      keyMappings:
        # Renames key accessKey in secret to s3.client.default.access_key in keystore
        accessKey: s3.client.default.access_key
        password: s3.client.default.secret_key

If a keyMappings is set, then keys which are not mapped, are also not added to the keystore. This allows to add only a selection of keys.

Proposed solution for the Stackable OpenSearch operator

There are already specific solutions in the SDP, e.g. S3Connection referencing a SecretClass to define a secured S3 connection. But there always must be an option (like podOverrides) to add arbitrary values, e.g. to use a feature which is not already implemented. This proposal is about this general approach, but it should not prevent more specific implementations later.

Proposal (updated on 2025-10-27 after a discussion in the daily meeting):

---
apiVersion: opensearch.stackable.tech/v1alpha1
kind: OpenSearchCluster
spec:
  clusterConfig:
    keystore:
      - key : <string> # key in the OpenSearch keystore
        secretKeyRef: # reference to the value which will be stored in the OpenSearch keystore
          name: <string> # name of a Secret
          key: <string> # key in the Secret

If desired, SecretClasses can be supported later if either a secretKeyRef or a secretClass can be set.

Example

---
apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
stringData:
  AWS_ACCESS_KEY_ID: opensearchAccessKey
  AWS_SECRET_ACCESS_KEY: opensearchSecretKey
---
apiVersion: opensearch.stackable.tech/v1alpha1
kind: OpenSearchCluster
spec:
  clusterConfig:
    keystore:
      - key : s3.client.default.access_key
        secretKeyRef:
          name: s3-credentials
          key: AWS_ACCESS_KEY_ID
      - key : s3.client.default.secret_key
        secretKeyRef:
          name: s3-credentials
          key: AWS_SECRET_ACCESS_KEY

$ bin/opensearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

[razvan]

Nice. May I suggest keystoreSource or keystoreData for the property name?

[NickLarsenNZ]

I think Secrets should be enough, though I would add Name to the property (like how Kubernetes does it for volumeMounts):

    keystore:
      - secretName: <string>
      - secretName: <string>

[siegfriedweber]

Added the solution in the OpenSearch Kubernetes Operator to the description

The option to map secret keys to keystore keys is interesting:

general:
  keystore:
    - secret:
        name: credentials
      keyMappings:
        # Renames key accessKey in secret to s3.client.default.access_key in keystore
        accessKey: s3.client.default.access_key
        password: s3.client.default.secret_key

Keys which are not mapped, are also not added to the keystore. This allows to add only a selection of keys.

A more Kubernetes-like but wordy way (see Define container environment variables with data from multiple Secrets) could be:

spec:
  clusterConfig:
    keystore: # better: notJavaKeystore :P We need docs what this is
      - key : s3.client.default.access_key
        secretKeyRef:
          name: s3-credentials
          key: AWS_ACCESS_KEY_ID
      - key : s3.client.default.secret_key
        secretKeyRef:
          name: s3-credentials
          key: AWS_SECRET_ACCESS_KEY

[siegfriedweber]

Proposal updated according to the discussion in the daily meeting and moved to voting

Please vote on this comment