Cluster stuck in "Security not initialized" loop after TLS certificate rotation (2.11.1) #1283
Description
Description:
Environment:
OpenSearch Version: 2.11.1
Deployment: OpenSearch Kubernetes Operator
Replicas: 3 Masters (currently trying to recover with 1)
The Issue: My transport and http certificates expired. I attempted to rotate them by deleting the Kubernetes secrets and letting the Operator recreate them. While the secrets were recreated successfully, the cluster is now stuck in a deadlock:
Masters are not Ready: The master pods are running but not "Ready" because the Security Plugin is not initialized.
Quorum Blocked: With only 1 replica active for troubleshooting, the node refuses to elect itself as master because it remembers the old 3-node quorum (requires at least 2 nodes).
Security Initialization Loop: I cannot run securityadmin.sh because the REST API is blocked (Security not initialized), and the script times out because the cluster state is RED/Not Elected.
Circular Dependency: I can't initialize security because the cluster isn't up, and the cluster won't stay up/ready because security isn't initialized.
What I've tried:
Deleting secrets to force certificate regeneration.
Setting discovery.type: single-node (rejected by Operator/Configuration conflicts).
Running securityadmin.sh manually from within the pod (SocketTimeout/Connection refused).
Request: How can I force the Security Plugin to initialize or bypass the quorum check to let securityadmin.sh apply the new certificates to the .opendistro_security index when the cluster is in this state?