fix: clean up listeners and document the intended purpose

[razvan]

Description

Currently the Kafka op creates a whole bunch of listeners (both at Kafka as well as Kubernetes level) without a clear understanding of their purpose.

The term listener here is a compound term including:

• a Kafka listener and advertised listerner property
• the Kubernetes listener volume associated with it
• the Kubernetes service associated with the listener volume.

The problem is that the code has become extremely complicated to maintain and extend, there are no tests for external access and no clear understanding if the status quo is actually correct or not.

For example:

• There are client and client_auth listeners. This is redundant. The name of the client listeners should not change because a different authentication mechanism is used.

• The naming is used interchangeably. In one part of the code client refers to client listeners and in others to internal broker listeners.

• Bug: The bootstrap SVC is always defined but the corresponding Kafka (advertised) listener is only defined when Kerberos enabled.

• When kerberos enabled, client listeners are left hanging. There is no client jaas configuration, no keytab, etc.

• There is no bootstrap-controller listener. This would be client interface for kraft controllers similar to the bootstrap listener for brokers.

Implementation Questions

• There is one internal service per pod but only one bootstrap service per role . Is this correct ?
  • The Kafka advertised bootstrap listener configuration is actually using the broker advertised listener configuration for the host name. Why?
• Users can configure a cluster_internal class for brokers and an external_* for the bootstrap service.
  • Will external clients be able to connect to the brokers?

Acceptance criteria

• There is a clear minimal list of (advertised) listeners that is independent of the authentication mechanisms.
• The listeners are partitioned by access patterns: internal and clients
• The listeners are partitioned by role: broker and controller (for kraft)
• The purpose and intended usage is clearly documented for all listeners created by the operator.
• If possible: add integration tests bootstrap listeners from outside the k8s cluster.