Migrate kafka-operator to use listener-operator for exposing brokers (#443)

* Migrate kafka-operator to use lb-operator for exposing brokers

This is far from final (for example, we'd need to expose a configurable
LoadBalancerClass, for example), but I wanted to see whether lb-op would be a
good fit for this operator first.

* regenerated charts

* Update for lb-operator -> listener-operator rebrand

* Use ListenerOperatorVolumeSourceBuilder

* Use shortcut for creating the listener volume

* Allow git source for operator-rs temporarily

* Rename node_cmd -> node_address_cmd

* Remove operator-rs entry in deny.toml

* Make listenerclass configurable

* Use operator-rs from git

* Use listener secret scope

* Use listener for bootstrap and discovery

* Separate out kcat secret management now that it is the only PEM user

* Remove unused KafkaTlsSecurity::kafka

* Regenerate CRD

* Switch to main op-rs

stackabletech/operator-rs#858 was merged

* Switch to per-rolegroup bootstrap listeners

* make crds

* CRD docs

* Delete old test files

* Documentation

* Try to un-angry rustdoc

* Re-enable orphan cleanup

* Changelog

* Fix configuration test

* Update docs/modules/kafka/examples/getting_started/kafka.yaml

Co-authored-by: Sebastian Bernauer <[email protected]>

* Rename TLS volume variables for consistency

* Rename volumes/paths too, correspondingly

* Remove stale FIXME

* Add RBAC permission to delete listeners (orphan clenaup)

* Make service_ports return ListenerPorts instead

* Recommend using the same ListenerClass across rolegroups

* Apply recommended labels to broker listeners

* Apply the correct rolegroup to bootstrap listeners, too

* Apply the correct rolegroup label to the rolegroup configmaps

* Apply recommended labels to the bootstrap listener PVC too

* Remove version from persistent PVCs

It broke upgrading

* Add note about upgrading to changelog

* docs: Add note on what bootstrap listeners return

* Update futures to 0.3.31

0.3.30 was yanked due to soundness issues.

* Drop unused futures compat feature

* Reformat bootstrap rolegroup note slightly

---------

Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Siegfried Weber <[email protected]>
Co-authored-by: Sebastian Bernauer <[email protected]>
Co-authored-by: Sebastian Bernauer <[email protected]>

Author: 4 people

CHANGELOG.md

@@ -13,6 +13,10 @@ All notable changes to this project will be documented in this file.
 - Reduce CRD size from `479KB` to `53KB` by accepting arbitrary YAML input instead of the underlying schema for the following fields ([#750]):
   - `podOverrides`
   - `affinity`
+- Migrate to exposing Kafka using Listener Operator ([#443]).
+  - BREAKING: The existing services will be migrated to the new format. Clients will need to re-read settings from the discovery configmap.
+  - BREAKING: Kafka is now only accessible from within the Kubernetes cluster by default. Set listener classes manually to expose it to the outside world (again).
+  - BREAKING: To complete an upgrade to this kafka-operator, all existing Kafka StatefulSets must be deleted manually. This will cause some downtime.
 
 ### Fixed
 
@@ -24,6 +28,7 @@ All notable changes to this project will be documented in this file.
 
 - Remove versions `3.4.1`, `3.6.1`, `3.6.2` ([#753]).
 
+[#443]: https://github.com/stackabletech/kafka-operator/pull/443
 [#741]: https://github.com/stackabletech/kafka-operator/pull/741
 [#750]: https://github.com/stackabletech/kafka-operator/pull/750
 [#753]: https://github.com/stackabletech/kafka-operator/pull/753