ERROR ActiveMQArtemisReconciler updated Failed

[lnthai2002]

Hi, artemis broker seem seem to restart every 5 min in my cluster and I saw its pod event log complain about readiness and liveness check fail so i kubectl edit ActiveMQArtemis and add the rediness and liveness but then i see the controller of the operator throwing these error non stop:
2024-04-01T18:01:04.896Z ERROR ActiveMQArtemisReconciler updated Failed {"kind ": "v1.StatefulSet", "named ": "activemq-artemis-broker-ss", "error": "failed to update *v1.StatefulSet, Operation cannot be fulfilled on statefulsets.apps \"activemq-artemis-broker-ss\": the object has been modified; please apply your changes to the latest version and try again"} github.com/go-logr/logr.Logger.Error /go/pkg/mod/github.com/go-logr/[email protected]/logr.go:279 github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisReconcilerImpl).updateRequestedResource /workspace/app/controllers/activemqartemis_reconciler.go:1368 github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisReconcilerImpl).updateResource /workspace/app/controllers/activemqartemis_reconciler.go:1348 github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisReconcilerImpl).ProcessResources /workspace/app/controllers/activemqartemis_reconciler.go:1312 github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisReconcilerImpl).Process /workspace/app/controllers/activemqartemis_reconciler.go:160 github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisReconciler).Reconcile /workspace/app/controllers/activemqartemis_controller.go:167 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

I'm not sure what impact would it cause but when i check the statefulset of the broker i see my readiness and liveness values there. Can someone explain this error and how to get rid of it?

[brusdev]

@lnthai2002 could you share the ActiveMQArtemis CR and the operator version that you are using?

[lnthai2002]

Here is my helm chart template defining ActiveMQArtemis:

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: activemq-artemis-broker
spec:
  acceptors:
    - name: "all-protocols-acceptor"
      bindToAllInterfaces: true
      port: 61617
#      protocols: "CORE" # not defining protocols make broker enable all protocols (CORE,STOMP, AMQP,...) support
  addressSettings:
    applyRule: merge_all
    addressSetting:
    - match: '#'
      deadLetterAddress: DLQ
      expiryAddress: ExpiryQueue
      redistributionDelay: 60000
      redeliveryDelay: 0
      maxSizeBytes: "-1"
      messageCounterHistoryDayLimit: 10
      addressFullPolicy: PAGE
      autoCreateQueues: true
      autoCreateAddresses: true
      autoCreateJmsQueues: true
      autoCreateJmsTopics: true
      autoDeleteQueues: false
      autoDeleteCreatedQueues: false
      autoDeleteAddresses: false
      defaultConsumerWindowSize: 0
    - match: 'JOB_QUEUE.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
# wait 5 MIN before delete queues
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: 0
      autoDeleteAddresses: true
    - match: 'CRON_JOB_QUEUE.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
# wait 5 MIN before delete queues
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: 0
      autoDeleteAddresses: true
    - match: 'QUEUE.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
# wait 5 MIN before delete queues
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: 0
      autoDeleteAddresses: true
    - match: 'TOPIC.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
# wait 5 MIN before delete queues
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: 0
      autoDeleteAddresses: true
    - match: 'TOPIC.internal.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
# wait 5 MIN before delete to avoid delete due to temporary lost connection
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: -1
      autoDeleteAddresses: true
    - match: 'TOPIC.retro.#'
      retroactiveMessageCount: 1
      autoDeleteQueues: true
      autoDeleteQueuesDelay: 0
      autoDeleteQueuesMessageCount: -1
    - match: '$.artemis.internal.#'
      autoDeleteQueues: false
      autoDeleteAddresses: true
    - match: 'internal-ncp-deployment'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
# wait 5 MIN before delete queues
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: -1
      autoDeleteAddresses: true
    - match: 'internal-cs-log-level-#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
# wait 5 MIN before delete queues
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: -1
      autoDeleteAddresses: true
    - match: 'internal_core_service_interruption-#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
# wait 5 MIN before delete queues
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: -1
      autoDeleteAddresses: true
  console:
    expose: true
    sslEnabled: false
  deploymentPlan:
    size: {{ .Values.replicas }}
    livenessProbe:
      initialDelaySeconds: 60
      periodSeconds: 10
      timeoutSeconds: 30
    readinessProbe:
      initialDelaySeconds: 60
      periodSeconds: 10
      timeoutSeconds: 30
    requireLogin: true
    messageMigration: true
    persistenceEnabled: {{ .Values.persistence.enabled }}
    {{- if .Values.persistence.enabled }}
    podSecurityContext:
      fsGroup: 0
    storage:
      size: {{ .Values.persistence.size }}
      storageClassName: {{ .Values.persistence.storageClass }}
    {{- end }}
    resources:
      requests:
        cpu: {{ .Values.resources.requests.cpu }}
        memory: {{ .Values.resources.requests.memory }}
  adminUser: {{ .Values.artemisUser }}
  adminPassword: {{ .Values.artemisPassword }}

The operator controller image is quay.io/artemiscloud/activemq-artemis-operator:1.0.16

[lnthai2002]

Hi, is there any news on this? I still get this error in the controller often

[brusdev]

Your ActiveMQArtemis CR LGTM, do you see any error in the broker pod log before it is restarted?