Deploy: updated charts.

Author: vladd-bit

deploy/charts/README.md

@@ -43,4 +43,4 @@ helm upgrade --install cogstack-opensearch ./deploy/charts/opensearch \
 ```
 
 The OpenSearch and Dashboards config files should come from `services/`, and the security files from `security/`, so Docker and Kubernetes use the same source files.
-Only keys in `envFile.includeKeys` and `usersEnvFile.includeKeys` are imported.
+Only keys in `envFile.includeKeys`, `usersEnvFile.includeKeys`, and `certificatesEnvFile.includeKeys` are imported.

deploy/charts/opensearch/README.md

@@ -12,15 +12,15 @@ Helm chart for deploying OpenSearch and/or OpenSearch Dashboards using the CogSt
   - `log4j2.properties` (when `opensearch.enabled=true`)
   - OpenSearch Security files (`config.yml`, `internal_users.yml`, `roles.yml`, `roles_mapping.yml`) (when `opensearch.enabled=true`)
   - `opensearch_dashboards.yml`
+- PVC-backed `data`, `logs`, and performance-analyzer storage for OpenSearch by default
 
 ## Prerequisites
 
 1. Kubernetes cluster with dynamic PV provisioning (if `opensearch.enabled=true` and `persistence.enabled=true`).
 2. Kubernetes Secrets containing TLS materials for enabled components.
 3. If `credentials.create=false`, an existing Secret with:
-   - `OPENSEARCH_INITIAL_ADMIN_PASSWORD`
-   - `KIBANA_USER`
-   - `KIBANA_PASSWORD`
+   - `OPENSEARCH_INITIAL_ADMIN_PASSWORD` when `opensearch.enabled=true`
+   - `KIBANA_USER` and `KIBANA_PASSWORD` when `dashboards.enabled=true`
 
 ## Required certificate secrets
 
@@ -120,14 +120,15 @@ helm template cogstack-opensearch ./deploy/charts/opensearch \
 - In this repo, the chart `files/` entries are symlinked to the shared `services/` and `security/` sources so Docker and Kubernetes stay aligned.
 - The standard install/render commands still use `--set-file` explicitly to make the source-of-truth paths obvious at invocation time.
 - If you run Helm from `deploy/charts/opensearch`, the equivalent relative paths are `../../../services/...` and `../../../security/...`.
-- `envFile.raw` can be set from `deploy/elasticsearch.env` and is loaded via `envFrom` into OpenSearch and Dashboards.
-- `usersEnvFile.raw` can be set from `security/env/users_elasticsearch.env` and feeds the credentials Secret (`OPENSEARCH_INITIAL_ADMIN_PASSWORD`, `KIBANA_USER`, `KIBANA_PASSWORD`).
+- `envFile.raw` can be set from `deploy/elasticsearch.env`; the chart reads shared values from it (`ELASTICSEARCH_CLUSTER_NAME`, `ELASTICSEARCH_JAVA_OPTS` / `OPENSEARCH_JAVA_OPTS`, `KIBANA_SERVER_NAME`) and still generates Kubernetes-specific discovery and publish-host settings itself.
+- `usersEnvFile.raw` can be set from `security/env/users_elasticsearch.env` and feeds only the credential keys required by the enabled components.
 - `certificatesEnvFile.raw` can be set from `security/env/certificates_elasticsearch.env`; currently `ES_CLIENT_CERT_NAME` is used to resolve Dashboards cert secret keys (`<name>.pem` / `<name>.key`).
-- `deploy/elasticsearch.env` shared values are used where they make sense on Kubernetes (`ELASTICSEARCH_CLUSTER_NAME`, `OPENSEARCH_JAVA_OPTS`, `KIBANA_SERVER_NAME`), while pod IP and discovery hosts remain Kubernetes-specific.
+- `deploy/elasticsearch.env` shared values are used where they make sense on Kubernetes (`ELASTICSEARCH_CLUSTER_NAME`, `ELASTICSEARCH_JAVA_OPTS` / `OPENSEARCH_JAVA_OPTS`, `KIBANA_SERVER_NAME`), while pod IP and discovery hosts remain Kubernetes-specific.
 - By default, `certificates.opensearchNodeFiles[*]` maps pod ordinals `0/1/2` to repo-style node cert keys `elasticsearch-1/2/3`.
+- `opensearch.logPersistence` and `opensearch.performanceAnalyzerPersistence` default to PVC-backed storage to stay closer to the Docker Compose deployment.
+- `opensearch.snapshotBackups` adds shared PVC-backed mounts for `/mnt/es_data_backups` and `/mnt/es_config_backups`; use RWX storage or set `existingClaim` values, and still set `path.repo` in the shared OpenSearch config if you want the cluster to use them.
 - `configFiles.opensearchRaw` can be set from `services/elasticsearch/config/opensearch.yml`.
 - `configFiles.log4jRaw` can be set from `services/elasticsearch/config/log4j2_opensearch.properties`.
 - `configFiles.dashboardsRaw` can be set from `services/kibana/config/opensearch.yml`.
 - `securityFiles.*Raw` can be set from `security/es_roles/opensearch/*.yml` and overrides the chart-bundled OpenSearch security files.
-- Only keys listed in `envFile.includeKeys` are imported (to avoid leaking secrets from env files into ConfigMaps).
 - Review security and certificate settings before production use.

deploy/charts/opensearch/templates/_helpers.tpl

@@ -64,9 +64,13 @@ app.kubernetes.io/component: dashboards
 {{- end -}}
 {{- end -}}
 
-{{/* Env ConfigMap name */}}
-{{- define "cogstack-opensearch.envConfigMapName" -}}
-{{- printf "%s-elasticsearch-env" (include "cogstack-opensearch.fullname" .) -}}
+{{/* Snapshot backup PVC names */}}
+{{- define "cogstack-opensearch.snapshotBackupDataPvcName" -}}
+{{- printf "%s-snapshot-backup-data" (include "cogstack-opensearch.fullname" .) -}}
+{{- end -}}
+
+{{- define "cogstack-opensearch.snapshotBackupConfigPvcName" -}}
+{{- printf "%s-snapshot-backup-config" (include "cogstack-opensearch.fullname" .) -}}
 {{- end -}}
 
 {{/* Parse deploy/elasticsearch.env into a filtered YAML map */}}

deploy/charts/opensearch/templates/configmap-elasticsearch-env.yaml

@@ -44,11 +44,6 @@ spec:
             - name: http
               containerPort: {{ .Values.dashboards.service.port }}
               protocol: TCP
-          {{- if .Values.envFile.raw }}
-          envFrom:
-            - configMapRef:
-                name: {{ include "cogstack-opensearch.envConfigMapName" . }}
-          {{- end }}
           env:
             - name: KIBANA_SERVER_NAME
               value: {{ $serverName | quote }}

deploy/charts/opensearch/templates/deployment-dashboards.yaml

@@ -0,0 +1,41 @@
+{{- if and .Values.opensearch.enabled .Values.opensearch.snapshotBackups.enabled }}
+{{- if not .Values.opensearch.snapshotBackups.data.existingClaim }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: {{ include "cogstack-opensearch.snapshotBackupDataPvcName" . }}
+  labels:
+    {{- include "cogstack-opensearch.labels" . | nindent 4 }}
+    app.kubernetes.io/component: opensearch
+spec:
+  accessModes:
+    {{- toYaml .Values.opensearch.snapshotBackups.data.accessModes | nindent 4 }}
+  resources:
+    requests:
+      storage: {{ .Values.opensearch.snapshotBackups.data.size }}
+  {{- if .Values.opensearch.snapshotBackups.data.storageClassName }}
+  storageClassName: {{ .Values.opensearch.snapshotBackups.data.storageClassName | quote }}
+  {{- end }}
+{{- end }}
+{{- if and (not .Values.opensearch.snapshotBackups.data.existingClaim) (not .Values.opensearch.snapshotBackups.config.existingClaim) }}
+---
+{{- end }}
+{{- if not .Values.opensearch.snapshotBackups.config.existingClaim }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: {{ include "cogstack-opensearch.snapshotBackupConfigPvcName" . }}
+  labels:
+    {{- include "cogstack-opensearch.labels" . | nindent 4 }}
+    app.kubernetes.io/component: opensearch
+spec:
+  accessModes:
+    {{- toYaml .Values.opensearch.snapshotBackups.config.accessModes | nindent 4 }}
+  resources:
+    requests:
+      storage: {{ .Values.opensearch.snapshotBackups.config.size }}
+  {{- if .Values.opensearch.snapshotBackups.config.storageClassName }}
+  storageClassName: {{ .Values.opensearch.snapshotBackups.config.storageClassName | quote }}
+  {{- end }}
+{{- end }}
+{{- end }}

deploy/charts/opensearch/templates/pvc-snapshot-backups.yaml

@@ -1,4 +1,4 @@
-{{- if and .Values.credentials.create (not .Values.credentials.existingSecret) }}
+{{- if and .Values.credentials.create (not .Values.credentials.existingSecret) (or .Values.opensearch.enabled .Values.dashboards.enabled) }}
 {{- $usersData := include "cogstack-opensearch.parsedUsersEnvFile" . | fromYaml | default (dict) -}}
 {{- $adminPassword := .Values.credentials.adminPassword -}}
 {{- if hasKey $usersData "OPENSEARCH_INITIAL_ADMIN_PASSWORD" -}}
@@ -20,7 +20,11 @@ metadata:
     {{- include "cogstack-opensearch.labels" . | nindent 4 }}
 type: Opaque
 stringData:
+  {{- if .Values.opensearch.enabled }}
   OPENSEARCH_INITIAL_ADMIN_PASSWORD: {{ $adminPassword | quote }}
+  {{- end }}
+  {{- if .Values.dashboards.enabled }}
   KIBANA_USER: {{ $kibanaUser | quote }}
   KIBANA_PASSWORD: {{ $kibanaPassword | quote }}
+  {{- end }}
 {{- end }}

deploy/charts/opensearch/templates/secret-credentials.yaml

@@ -92,11 +92,6 @@ spec:
             - name: analyzer
               containerPort: {{ .Values.opensearch.service.analyzerPort }}
               protocol: TCP
-          {{- if .Values.envFile.raw }}
-          envFrom:
-            - configMapRef:
-                name: {{ include "cogstack-opensearch.envConfigMapName" . }}
-          {{- end }}
           env:
             - name: POD_IP
               valueFrom:
@@ -212,6 +207,12 @@ spec:
               mountPath: /usr/share/opensearch/logs
             - name: performance-analyzer
               mountPath: /usr/share/opensearch/config/opensearch-performance-analyzer/plugins-stats-metadata
+            {{- if .Values.opensearch.snapshotBackups.enabled }}
+            - name: snapshot-backup-data
+              mountPath: {{ .Values.opensearch.snapshotBackups.data.mountPath }}
+            - name: snapshot-backup-config
+              mountPath: {{ .Values.opensearch.snapshotBackups.config.mountPath }}
+            {{- end }}
       volumes:
         - name: opensearch-config
           configMap:
@@ -230,14 +231,26 @@ spec:
           secret:
             secretName: {{ .Values.certificates.opensearchSecretName }}
         {{- end }}
+        {{- if .Values.opensearch.snapshotBackups.enabled }}
+        - name: snapshot-backup-data
+          persistentVolumeClaim:
+            claimName: {{ default (include "cogstack-opensearch.snapshotBackupDataPvcName" .) .Values.opensearch.snapshotBackups.data.existingClaim }}
+        - name: snapshot-backup-config
+          persistentVolumeClaim:
+            claimName: {{ default (include "cogstack-opensearch.snapshotBackupConfigPvcName" .) .Values.opensearch.snapshotBackups.config.existingClaim }}
+        {{- end }}
         {{- if not .Values.opensearch.persistence.enabled }}
         - name: data
           emptyDir: {}
         {{- end }}
+        {{- if not .Values.opensearch.logPersistence.enabled }}
         - name: logs
           emptyDir: {}
+        {{- end }}
+        {{- if not .Values.opensearch.performanceAnalyzerPersistence.enabled }}
         - name: performance-analyzer
           emptyDir: {}
+        {{- end }}
       {{- with .Values.opensearch.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
@@ -250,8 +263,9 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-  {{- if .Values.opensearch.persistence.enabled }}
+  {{- if or .Values.opensearch.persistence.enabled .Values.opensearch.logPersistence.enabled .Values.opensearch.performanceAnalyzerPersistence.enabled }}
   volumeClaimTemplates:
+    {{- if .Values.opensearch.persistence.enabled }}
     - metadata:
         name: data
       spec:
@@ -263,5 +277,32 @@ spec:
         {{- if .Values.opensearch.persistence.storageClassName }}
         storageClassName: {{ .Values.opensearch.persistence.storageClassName | quote }}
         {{- end }}
+    {{- end }}
+    {{- if .Values.opensearch.logPersistence.enabled }}
+    - metadata:
+        name: logs
+      spec:
+        accessModes:
+          {{- toYaml .Values.opensearch.logPersistence.accessModes | nindent 10 }}
+        resources:
+          requests:
+            storage: {{ .Values.opensearch.logPersistence.size }}
+        {{- if .Values.opensearch.logPersistence.storageClassName }}
+        storageClassName: {{ .Values.opensearch.logPersistence.storageClassName | quote }}
+        {{- end }}
+    {{- end }}
+    {{- if .Values.opensearch.performanceAnalyzerPersistence.enabled }}
+    - metadata:
+        name: performance-analyzer
+      spec:
+        accessModes:
+          {{- toYaml .Values.opensearch.performanceAnalyzerPersistence.accessModes | nindent 10 }}
+        resources:
+          requests:
+            storage: {{ .Values.opensearch.performanceAnalyzerPersistence.size }}
+        {{- if .Values.opensearch.performanceAnalyzerPersistence.storageClassName }}
+        storageClassName: {{ .Values.opensearch.performanceAnalyzerPersistence.storageClassName | quote }}
+        {{- end }}
+    {{- end }}
   {{- end }}
 {{- end }}

deploy/charts/opensearch/templates/statefulset-opensearch.yaml

@@ -15,14 +15,10 @@ envFile:
   # Pass deploy/elasticsearch.env via:
   #   --set-file envFile.raw=./deploy/elasticsearch.env
   raw: ""
-  # Only these keys are imported into a ConfigMap (avoid leaking secrets from env files).
+  # Only these shared values are read from deploy/elasticsearch.env.
+  # Kubernetes-specific discovery and publish-host values are generated by the chart.
   includeKeys:
-    - ELASTICSEARCH_NETWORK_HOST
-    - ELASTICSEARCH_NETWORK_PUBLISH_HOST
     - ELASTICSEARCH_CLUSTER_NAME
-    - ELASTICSEARCH_SEED_HOSTS
-    - ELASTICSEARCH_INITIAL_CLUSTER_MANAGER_NODES
-    - ELASTICSEARCH_HOSTS
     - ELASTICSEARCH_JAVA_OPTS
     - OPENSEARCH_JAVA_OPTS
     - KIBANA_SERVER_NAME
@@ -67,6 +63,37 @@ opensearch:
       - ReadWriteOnce
     size: 20Gi
     storageClassName: ""
+  logPersistence:
+    enabled: true
+    accessModes:
+      - ReadWriteOnce
+    size: 10Gi
+    storageClassName: ""
+  performanceAnalyzerPersistence:
+    enabled: true
+    accessModes:
+      - ReadWriteOnce
+    size: 1Gi
+    storageClassName: ""
+  snapshotBackups:
+    # Optional parity mounts for Docker's shared backup paths.
+    # These should use shared storage accessible from all OpenSearch pods.
+    # Note: OpenSearch will only use these if path.repo is also configured in opensearch.yml.
+    enabled: false
+    data:
+      mountPath: /mnt/es_data_backups
+      existingClaim: ""
+      accessModes:
+        - ReadWriteMany
+      size: 20Gi
+      storageClassName: ""
+    config:
+      mountPath: /mnt/es_config_backups
+      existingClaim: ""
+      accessModes:
+        - ReadWriteMany
+      size: 5Gi
+      storageClassName: ""
   service:
     type: ClusterIP
     httpPort: 9200

deploy/charts/opensearch/values.yaml

@@ -98,8 +98,8 @@ helm upgrade --install cogstack-opensearch ./deploy/charts/opensearch \
 
 > The chart expects pre-created Kubernetes Secrets for TLS materials (see the chart README).
 > The `--set-file configFiles.*Raw=...` flags point Helm at the same OpenSearch and Dashboards config files used by Docker Compose.
-> The `--set-file envFile.raw=...` flag injects values from `deploy/elasticsearch.env` into pod environment variables.
-> The `--set-file usersEnvFile.raw=...` flag feeds credentials (`OPENSEARCH_INITIAL_ADMIN_PASSWORD`, `KIBANA_USER`, `KIBANA_PASSWORD`) into the chart Secret.
+> The `--set-file envFile.raw=...` flag lets the chart read shared values from `deploy/elasticsearch.env` while still generating Kubernetes-specific discovery and publish-host settings itself.
+> The `--set-file usersEnvFile.raw=...` flag feeds only the credential keys required by the enabled chart components into the chart Secret.
 > The `--set-file certificatesEnvFile.raw=...` flag loads certificate metadata from `security/env/certificates_elasticsearch.env` (`ES_CLIENT_CERT_NAME` currently).
 > The `--set-file securityFiles.*Raw=...` flags use `security/es_roles/opensearch/*.yml` as the source of OpenSearch security config.