Deployed broker doesn't pass readiness checks

[dj-wasabi]

I have the following manifest to create a new broker:

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: activemg-broker-activemq
spec:
  acceptors:
    - name: tlsaccept
      protocols: all
      port: 61617
      sslEnabled: false
      needClientAuth: true
      expose: false
  deploymentPlan:
    size: 2
    messageMigration: true
    persistenceEnabled: true
    enableMetricsPlugin: true
    podDisruptionBudget:
      minAvailable: 1
    podSecurityContext:
      fsGroup: 0
    containerSecurityContext:
      runAsNonRoot: true
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "2"
        memory: 4Gi
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      tcpSocket:
        port: 8161
      timeoutSeconds: 10
    readinessProbe:
      exec:
        command:
        - /bin/bash
        - -c
        - /opt/amq/bin/readinessProbe.sh 1
      failureThreshold: 3
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    storage:
      size: 10G
    extraVolumes:
      - name: jms-server-certs
        secret:
          secretName: artemis-active-secret
          defaultMode: 256
    extraVolumeMounts:
      - name: jms-server-certs
        mountPath: /etc/ssl/activemq
        readOnly: true
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
              - key: lifecycle
                operator: In
                values:
                  - OnDemand
  brokerProperties:
    - "metricsConfiguration.jvmGc=true"
    - "metricsConfiguration.jvmMemory=true"
    - "metricsConfiguration.jvmThread=true"
    - 'addressSettings.#.addressFullMessagePolicy=PAGE'
    - "connectorConfigurations.tlsaccept.params.sslEnabled=true"
    - "connectorConfigurations.tlsaccept.params.needClientAuth=true"
    - "connectorConfigurations.tlsaccept.params.keyStorePath=/etc/ssl/activemq/broker.ks"
    - "connectorConfigurations.tlsaccept.params.keyStorePassword=${KEYSTORE_PASSWORD}"
    - "connectorConfigurations.tlsaccept.params.trustStorePath=/etc/ssl/activemq/client.ts"
    - "connectorConfigurations.tlsaccept.params.trustStorePassword=${TRUSTSTORE_PASSWORD}"
  env:
    - name: JAVA_ARGS_APPEND
      value: '-XX:InitialRAMPercentage=80.0 -XX:MaxRAMPercentage=80.0'
    - name: KEYSTORE_PASSWORD
      valueFrom:
        secretKeyRef:
          key: keyStorePassword
          name: artemis-active-secret
    - name: TRUSTSTORE_PASSWORD
      valueFrom:
        secretKeyRef:
          key: trustStorePassword
          name: artemis-active-secret

The liveness and readiness probe configuration comes somewhere from the documentation, but the pod doesn't come available:

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  24m                  default-scheduler  Successfully assigned smhe-test/activemg-broker-activemq-ss-0 to ip-10-31-56-169.eu-west-1.compute.internal
  Normal   Pulled     24m                  kubelet            Container image "quay.io/arkmq-org/activemq-artemis-broker-init@sha256:f2c853ff91809282c04d5b1381ceae366518bf07b16649a61aa6c60c47f10714" already present on machine
  Normal   Created    24m                  kubelet            Created container: activemg-broker-activemq-container-init
  Normal   Started    24m                  kubelet            Started container activemg-broker-activemq-container-init
  Normal   Pulled     22m                  kubelet            Container image "quay.io/arkmq-org/activemq-artemis-broker-kubernetes@sha256:a4a0d0453f0761b9e114d0d4cc7c14091a2a9c30790f18ffa6bf1e365d5083fe" already present on machine
  Normal   Created    22m                  kubelet            Created container: activemg-broker-activemq-container
  Normal   Started    22m                  kubelet            Started container activemg-broker-activemq-container
  Warning  Unhealthy  21m (x2 over 21m)    kubelet            Liveness probe failed: dial tcp 10.31.62.21:8161: connect: connection refused
  Warning  Unhealthy  100s (x92 over 21m)  kubelet            Readiness probe failed: command timed out: "/bin/bash -c /opt/amq/bin/readinessProbe.sh 1" timed out after 10s

The pod is started, without any errors:

2025-05-19 11:37:46,184 INFO  [org.jgroups.protocols.pbcast.GMS] activemg-broker-activemq-ss-0-17567: no members discovered after 2005 ms: creating cluster as coordinator
2025-05-19 11:37:46,205 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2025-05-19 11:37:46,205 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2025-05-19 11:37:46,205 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2025-05-19 11:37:46,205 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2025-05-19 11:37:46,205 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2025-05-19 11:37:46,206 INFO  [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2025-05-19 11:37:46,230 INFO  [org.apache.activemq.artemis.core.server] AMQ221034: Waiting indefinitely to obtain primary lock
2025-05-19 11:37:46,230 INFO  [org.apache.activemq.artemis.core.server] AMQ221035: Primary Server Obtained primary lock
2025-05-19 11:37:46,508 INFO  [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2025-05-19 11:37:46,509 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2025-05-19 11:37:46,530 INFO  [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2025-05-19 11:37:46,530 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2025-05-19 11:37:46,748 INFO  [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at activemg-broker-activemq-ss-0.activemg-broker-activemq-hdls-svc.smhe-test.svc.cluster.local:61616 for protocols [CORE]
2025-05-19 11:37:46,749 INFO  [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at activemg-broker-activemq-ss-0.activemg-broker-activemq-hdls-svc.smhe-test.svc.cluster.local:61617 for protocols [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
2025-05-19 11:37:46,751 INFO  [org.apache.activemq.artemis.core.server] AMQ221007: Server is now active
2025-05-19 11:37:46,751 INFO  [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.40.0 [amq-broker, nodeID=d91c9d83-34a3-11f0-ae53-3a9f68320556]
2025-05-19 11:37:46,756 INFO  [org.apache.activemq.artemis] AMQ241003: Starting embedded web server
2025-05-19 11:37:47,134 INFO  [io.hawt.HawtioContextListener] Initialising Hawtio services
2025-05-19 11:37:47,137 INFO  [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 4.2.0
2025-05-19 11:37:47,148 INFO  [io.hawt.web.auth.AuthenticationConfiguration] Authentication throttling is enabled
2025-05-19 11:37:47,162 INFO  [io.hawt.web.auth.AuthenticationConfiguration] Starting Hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "admin" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2025-05-19 11:37:47,163 INFO  [io.hawt.web.auth.AuthenticationConfiguration] Looking for OIDC configuration file in: /home/jboss/amq-broker/etc/hawtio-oidc.properties
2025-05-19 11:37:47,233 INFO  [io.hawt.web.auth.ClientRouteRedirectFilter] Hawtio ClientRouteRedirectFilter is using 1800 sec. HttpSession timeout
2025-05-19 11:37:47,261 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://activemg-broker-activemq-ss-0.activemg-broker-activemq-hdls-svc.smhe-test.svc.cluster.local:8161
2025-05-19 11:37:47,262 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://activemg-broker-activemq-ss-0.activemg-broker-activemq-hdls-svc.smhe-test.svc.cluster.local:8161/console/jolokia
2025-05-19 11:37:47,262 INFO  [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://activemg-broker-activemq-ss-0.activemg-broker-activemq-hdls-svc.smhe-test.svc.cluster.local:8161/console

But even when I do a curl to fqdn vs to console 1 works the other not (and i think that is probably reason why the liveness is a tcp port check?):

[jboss@activemg-broker-activemq-ss-0 ~]$ curl http://activemg-broker-activemq-ss-0.activemg-broker-activemq-hdls-svc.mynamespace.svc.cluster.local:8161/console/jolokia -I
HTTP/1.1 403 Forbidden
Date: Mon, 19 May 2025 12:02:29 GMT
Cache-Control: max-age=0, no-cache, must-revalidate, proxy-revalidate, private
Pragma: no-cache
X-Frame-Options: DENY
X-XSS-Protection: 1
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self' data: https://cdn.jsdelivr.net/npm/[email protected]/min/; img-src 'self' data:; connect-src 'self'; frame-src 'self'; manifest-src 'self'; media-src 'self'; object-src 'self'; worker-src 'self' blob:; frame-ancestors 'none'; script-src-elem 'self' https://cdn.jsdelivr.net/npm/[email protected]/min/; style-src-elem 'self' 'unsafe-inline' https://cdn.jsdelivr.net/npm/[email protected]/min/
Strict-Transport-Security: max-age=31536000;includeSubDomains;preload
Referrer-Policy: strict-origin
Hawtio-Forbidden-Reason: NONE
Content-Length: 0

[jboss@activemg-broker-activemq-ss-0 ~]$ curl localhost:8161/console/jolokia -i
curl: (7) Failed to connect to localhost port 8161: Connection refused
[jboss@activemg-broker-activemq-ss-0 ~]$

And even the readinessprobe does take a long time:

[jboss@activemg-broker-activemq-ss-0 ~]$ time /opt/amq/bin/readinessProbe.sh 1

real    0m49.763s
user    0m51.346s
sys     0m0.369s

Unfortunately (but from a security p.o.v. I totally understands!) I am not able to check my other port (61617) to see if it is open, but a curl failure doesn't convince me that it is open.

So at the moment, I am not sure what to do. Well, I can change the periodSeconds to a much higher value, but that doesn't sound right to me. Any one else has ideas?

CPU Load is quite high to be honest (dotted line is the limits):

[brusdev]

@dj-wasabi this seems an environment issue because I created the same ActiveMQArtemis CR and the probes are not failing.

Anyway, I found the following unrelated issues in your CR:

• the broker properties to set the params of the tlsaccept acceptor should start with acceptorConfigurations.tlsaccept.params.;
• the JVM settings -XX:InitialRAMPercentage=80.0 -XX:MaxRAMPercentage=80.0 could cause the broker pods to be killed because they use more memory than their limits. This is because netty native buffers doesn't use JVM heap.

[dj-wasabi]

ok crabs, I was already afraid of that. I'll try to dive deeper, thank you for confirming this.

Thank you, I havent noticed it till I saw your message! With regards to the JVM, so from your p.o.v. we should not set these kinds of parameters? So just keep it to the "defaults" that the operator sets?

[brusdev]

The operator use the safest value 25 but the value 50 should work in most of the use cases. For higher values I would recommend to monitor the JVM metrics and netty-pool metrics, i.e.

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: artemis-with-metrics
spec:
  console:
    expose: true
  deploymentPlan:
    enableMetricsPlugin: true
  brokerProperties:
    - "metricsConfiguration.jvmGc=true"
    - "metricsConfiguration.jvmMemory=true"
    - "metricsConfiguration.jvmThread=true"
    - "metricsConfiguration.nettyPool=true"