Spec of the feature:

A new top level entity should be added to the schema registry: KafkaCluster. This should have a name (the one referred in the subscription), and some free form map<string, string> of properties (subscription calls that metadata, uhmmmm maybe that wasn't a good name...). When being created, we should validate this map contains at least either bootstrap.servers or metadata.broker.list, as that's the basic requirement to connect to kafka (see https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md)
Everytime the Admin API gets queried and returns a KafkaCluster, or a Kafka subscription, we MUST redact (just replace with 3 asterisks) the following metadata/properties entries: sasl.username, sasl.password, ssl.keystore.location, ssl.keystore.password, ssl.key.pem, ssl_key, ssl.key.password, sasl.oauthbearer.client.credentials.client.secret, sasl.oauthbearer.client.secret, sasl.oauthbearer.assertion.private.key.passphrase, sasl.oauthbearer.assertion.private.key.pem. Double check all the secrets in https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md
We still need to keep supporting the static configuration of kafka clusters, so when a kafka cluster gets added through admin api, check there isn't conflict with the static configuration (conflict on using the same name is enough).
At this point, everything should work as it does today for static configuration: when adding a subscription, we should check the cluster exists, then when using the subscription, we should merge the properties from the KafkaCluster with the metadata from the subscription.

Dynamically configure Kafka clusters #964

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions