Plugin does not reload SSL certificates dynamically

[ProtemS]

Overview

When we update the SSL certificates on disk for our self hosted Kafka brokers, the brokers correctly pick up the new certificates without requiring a restart, thanks to the dynamic reload feature.

However, the Strimzi Quotas Plugin does not appear to reload the new certs, it continues using the old ones, which causes SSLHandshakeException failures until we restart the broker, and causes the plugin to stop working.

This is delaying us from deploying the plugin on production grade clusters, since we are unable to run rolling restarts frequently.

Expected behavior:

The plugin should watch the certificate and key files for changes, or provide a mechanism to reload them dynamically without requiring a broker restart.

This behavior should align with how Kafka broker configs support different update modes, according to Official Confluent Documentation

• read-only: Requires a broker restart for update.

• per-broker: May be updated dynamically for each broker.

• cluster-wide: May be updated dynamically as a cluster-wide default, or overridden per broker for testing.

Currently, the plugin’s AdminClient SSL config behaves like read-only. We would like it to support dynamic update just like a per-broker config.

Steps to reproduce:

Deploy the plugin with the AdminClient SSL enabled, e.g.
client.quota.callback.static.kafka.admin.ssl.keystore.location

Rotate the server certificates on disk, and wait until the old ones expire.

Observe that the broker’s listener keeps working, but the plugin’s AdminClient starts failing SSL handshakes until the broker is restarted.

Possible solution:

Make the plugin implement a file watcher over the keystore/truststore file locations and re-create the AdminClient when files change.

Or reuse the broker’s SslFactory so the AdminClient benefits from the same dynamic reload behavior.

Or allow ssl.* properties to be updated dynamically, similar to:

bin/kafka-configs --bootstrap-server BROKER_LISTENER --entity-type brokers --entity-name BROKER_ID --alter --add-config clinet.quota.callback.static.kafka.admin.ssl.keystore.location=LOCATION

[scholzj]

This is a reasonable feature request. But just watching some files and reloading them does not really work in most situations and is not usable for Strimzi (+ watching files for change is actually pretty non-trivial task if you want it to work across all different environments). So any solution would need to be more along the way of dynamic reconfiguration and would need to be in sync with whatever Kafka does one day to handle this properly.

[scholzj]

Triaged on 4.9.2025: This would be a useful feature. But we currently do not support reloading of certificates in Strimzi Cluster Operator. And when it is implemented, we need to make sure it is in sync with what is implemented in the Quotas plugins. So we should keep the issues. But it would need a proposal and it should be blocked until it is clear how we do certificate reloading in Strimzi Cluster Operator.