Using elasticsearch DHI image in ECK

[jgoodall]

Has anyone successfully used the elasticsearch docker hardened image with Elastic Cloud on Kubernetes (ECK)? I first ran into an issue where the elasticsearch pod would not start because: container has runAsNonRoot and image has non-numeric user (elasticsearch), cannot verify user is non-root. I addressed this by setting the securityContext:

    podTemplate:
      spec:
        securityContext:
          runAsUser: 1000
          fsGroup: 1000

This leads the container to start with status of RUNNING, but READY is 0/1 and kubectl describe shows this:

  Warning  Unhealthy  5s (x9 over 45s)  kubelet            spec.containers{elasticsearch}: Readiness probe failed: Elasticsearch is not ready yet. Check the server logs.

But the logs look fine, no errors, few warnings related to "Not entitled: component". The ECK operator logs show a Reconciler error with the message:

elasticsearch client failed for http://elastic-es-default-0.elastic-es-default.serve:9200/_security/api_key?active_only=true&name=eck-*: Get \"http://elastic-es-default-0.elastic-es-default.serve:9200/_security/api_key?active_only=true&name=eck-*\": dial tcp 192.168.194.25:9200: connect: connection refused

Can anyone help me figure this out?

[jgoodall]

I am pretty sure this image will never work with ECK, because the readinessProbe. Here is the readiness probe for elasticsearch:

  Readiness:  exec [bash -c /mnt/elastic-internal/scripts/readiness-port-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3

The script in container:

cat /mnt/elastic-internal/scripts/readiness-port-script.sh
#!/usr/bin/env bash
if ! nc -z -v -w5 127.0.0.1 8080 >/dev/null 2>&1; then
    echo "Elasticsearch is not ready yet. Check the server logs."
    exit 1
fi

But nc is not available:

elasticsearch@elastic-es-default-0:/$ nc -z -v -w5 127.0.0.1 8080
bash: nc: command not found

I was unable to use a custom readinessProbe. The readinessProbe ECK docs say:

We do not recommend overriding the default readiness probe on Elasticsearch 8.2.0 and later. ECK configures a socket-based readiness probe using the Elasticsearch readiness port feature which is not influenced by the load on the Elasticsearch cluster.

I was unable to get that to work. But as is I dont think this image will ever work with ECK, which is one of the primary deployment strategies where a hardened image would be useful.