scaledown - making it more deterministic

[gtully]

The logic around scale down, takes the pod out of the statefulset and starts a deployment that takes over the PVC and makes use of the jmx shutdown(scaledown=true) feature of the broker, which leverages cluster support. In the event of failures, this procedure can leave messages pending or lost. We loose visibility of the draining broker and the availability of the statefulset.

I am thinking to change the behaviour. The guiding principal would be to only scale down empty brokers and to block the statefulset size adjustment till the broker is empty.

This means that the broker needs to quiesce, producers need to stop and consumers probably to, as connections can share producers and consumers.
Then existing messages on queues need to be drained. Updating the configuration to add an AMQP queue-bridge-to-the-broker-group can drain any pending messages.

If there are pending messages on any cluster snf queue, these need to be allowed to drain over an existing cluster connection. This may be a challenge if we also want to stop message production. This is only the case when clustered=true and size > 1. It will still be ok to scale to zero with pending messages as ordinal will be 0.

I am thinking conditions and reasons.

ScaleDownPending
ReasonPendingMessages count=?
ReasonPendingSNFMessages count=?

If a broker is in state ScaleDownPending=true with reason ReasonPendingSNFMessages count > 0, some management intervention may be needed to purge the relevant snf or move the messages elsewhere.

Eventually, the operator will see an empty quiesced broker and will scaledown statefulset and release the PVC in the normal way.

I need to check the detail on quiescing, to see what is currently available through config reload and JMX, ideally config reload will suffice to effect the necessary restrictions. That is the plan.