operator v1: do not decom broker if it's missing in RP

[birdayz]

avoid setting decommissionBrokerID for an already-decommissioned-broker by also checking in redpanda if that broker exists.

lots of context can be found in: #314

[chrisseto]

Thus far, I like the approach here the most but it still feels like we're papering over the larger issue here: A lack of idempotency and crash safety.

Stale reads are a symptom of the above but we could easily hit the same problem if the operator crashes (or gets evicted) at an inopportune time.

Couldn't we largely eliminate the problem if the handleScaling function pulled state information from the StatefulSet and adminAPI and pushed that into .Status instead of relying on .Status? (Similar to what Rafal has mentioned in Slack).

It doesn't seem like that large of a change to me and I suspect a lot of logic in this code path could be simplified.

For example, handleDecommissionInProgress could first fetch the brokers list and brokers uuid list from the admin API. If there's a broker that's in the process of decommissioning, you know there's a decommission in progress. With additional information from the StatefulSets/Pod list you should be able to deduce if the decommission hasn't yet started or if it's finished.

[birdayz]

Regarding crash safety, the same check could be added in the cluster_controller.go, and clear the status decommissionBrokerID if required.