Scale down not working?

[maxysio]

Hello,

• I was following the instructions for a basic deployment using a minikube on my mac with the following yaml. Ignore the formatting.

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
name: ex-aao
spec:
deploymentPlan:
size: 2
image: placeholder
persistenceEnabled: true
messageMigration: true

• The first time i used deployment size as 2
• Sent 100 messages to both the brokers
• Then applied the new yaml file with deployment size as 1
• It shut down the second pod with but pod1 stayed with only 100 messages.
• Providing the log from pod 1 below

[org.apache.activemq.artemis.core.server] AMQ224091: Bridge ClusterConnectionBridge@4bc24f48 [name=$.artemis.internal.sf.my-cluster.5a14c62f-abe3-11ed-b9e3-0242ac110006, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.5a14c62f-abe3-11ed-b9e3-0242ac110006, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@110be0ff targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@4bc24f48 [name=$.artemis.internal.sf.my-cluster.5a14c62f-abe3-11ed-b9e3-0242ac110006, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.5a14c62f-abe3-11ed-b9e3-0242ac110006, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@110be0ff targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=ex-aao-ss-1], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@627519623[nodeUUID=52568132-abe3-11ed-8708-0242ac110005, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=ex-aao-ss-0-ex-aao-hdls-svc-activemq-artemis-operator-svc-cluster-local, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=ex-aao-ss-1], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying 2023-02-13 21:49:45,063 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection

What am I doing wrong? :(

[gaohoward]

The logs seems to show that the pod lost the connection to the other broker pod. What should happen is that the operator will then start up a "drainer" pod to migrate the messages to the living pod. It may take some seconds to do it.
Can you share the full operator pod'd log?

[maxysio]

Thank you @gaohoward for your response. I checked the logs looks like the netty connector is not working. Attaching the entire log. Would deeply appreciate any guidance.

log.txt

[gaohoward]

@maxysio the log looks fine. When the other broker pod is down this broker will retry connect to it. What's not normal is that the operator should kick off a "drainer" pod so that the messages can be migrated to this broker. Can you share the full operator pod log?

[maxysio]

Thank you @gaohoward for taking a look, here is the artemis-controller-manager pod's log
log_controller.txt

[gaohoward]

@maxysio I think I reproduced the issue. That could be a regression, I'll take a look. Thanks !

[rvais]

Please create the Github issue and downstream jira with the details filled in. Thanks!

[gaohoward]

https://github.com/artemiscloud/activemq-artemis-operator/issues/510

[gaohoward]

https://github.com/artemiscloud/activemq-artemis-operator/pull/527

[gaohoward]

@maxysio Please try the latest tag, it's been fixed.