How to update KafkaConnect from 0.46.0 to 0.47.0?

[marcosschroh]

Hi,

I have installed the strimzi operator 0.46.0 in Kubernetes, I am using KafkaConnect, Connectors, a custom docker image which extends from quay.io/strimzi/kafka:0.46.0-kafka-3.9.0. Everything has work like a charm!!!.

I saw that there is a new operator release, 0.47.0, then I updated the operator. I did not see breaking changes in the CHANGELOG, so I was expecting a smooth update, however the KafkaConnect pods are continuously crashing CrashLoopBackOff and restarting. The pods are Ready False

Kubernetes version: v1.31.10
Client Version: v1.30.2
Strimzi operator: 0.47.0
New base docker image: quay.io/strimzi/kafka:0.47.0-kafka-3.9.0

Is there a proper way to update KafkaConnect from 0.46.0 to 0.47.0?

The only thing that I see in the logs is an ERROR related to log4j, which has always been there, and it is ignored, including version 0.46.0 :

log4j:ERROR Could not read configuration file from URL [file:/opt/kafka/custom-config/log4j.properties].
log4j:ERROR Could not read configuration file from URL [file:/opt/kafka/custom-config/log4j.properties].
java.io.FileNotFoundException: /opt/kafka/custom-config/log4j.properties (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
	at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
	at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:532)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:485)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:115)
	at org.slf4j.impl.Reload4jLoggerFactory.<init>(Reload4jLoggerFactory.java:67)
	at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
	at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
	at org.apache.kafka.connect.cli.AbstractConnectCli.<clinit>(AbstractConnectCli.java:48)
log4j:ERROR Ignoring configuration file [file:/opt/kafka/custom-config/log4j.properties].
log4j:WARN No appenders could be found for logger (org.apache.kafka.connect.cli.AbstractConnectCli).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

[im-konge]

Hey, so if you have new custom image, you should also have the version of Kafka mentioned in the KafkaConnect's .spec section.

[marcosschroh]

I think you are right, but how come the exact same KafkaConnect manifest works fine with 0.46.0 but not with 0.47.0?

The secrets and template are the same for both scenarios.

The following values are related with the authentication, which it seems working fine (according to the logs). The certificate and keys are properly loaded from the k8s secret

ssl.keystore.certificate.chain = [hidden]
ssl.keystore.key = [hidden]

ssl.keystore.certificate.chain=${strimzisecrets:des-strimzi-connect-acc/des-strimzi-connect:certificate.pem}
ssl.keystore.key=${strimzisecrets:des-strimzi-connect-acc/des-strimzi-connect:certificate.key}
ssl.keystore.type=PEM
producer.ssl.keystore.certificate.chain=${strimzisecrets:des-strimzi-connect-acc/des-strimzi-connect:certificate.pem}
producer.ssl.keystore.key=${strimzisecrets:des-strimzi-connect-acc/des-strimzi-connect:certificate.key}
producer.ssl.keystore.type=PEM
consumer.ssl.keystore.certificate.chain=${strimzisecrets:des-strimzi-connect-acc/des-strimzi-connect:certificate.pem}
consumer.ssl.keystore.key=${strimzisecrets:des-strimzi-connect-acc/des-strimzi-connect:certificate.key}
consumer.ssl.keystore.type=PEM
admin.ssl.keystore.certificate.chain=${strimzisecrets:des-strimzi-connect-acc/des-strimzi-connect:certificate.pem}
admin.ssl.keystore.key=${strimzisecrets:des-strimzi-connect-acc/des-strimzi-connect:certificate.key}
admin.ssl.keystore.type=PEM**

[scholzj]

Just to make sure, I have been using the PEM format for certificate and key with 0.46.0

...

-----BEGIN RSA PRIVATE KEY-----
...Base64 data...
-----END RSA PRIVATE KEY-----

This should normally be only:

-----BEGIN PRIVATE KEY-----
...Base64 data...
-----END PRIVATE KEY-----

No idea if this could be the issue or not. I think we never supported RSA keys. No idea whether they just worked by coincidence in any previous versions. But certainly not intentionally supported.

Could you share the BREAKING changes introduced by operator 0.47.0 only for KafkaConnect? :-) . My theory is that if there are not ANY breaking changes introduced in 0.47.0 then I should be able to run the exact same KafkaConnect template without any problem.

Again, breaking changes imply public interfaces. There were no changes to public interfaces I'm aware of. We were aware that some users were hacking into the internal configurations around the authentication, so that is why it was included in the release notes => sadly, looking at the code, it looks like that was added tot he 0.46.0 release notes 🤦. Sorry for that.

[scholzj]

This should normally be only:

-----BEGIN PRIVATE KEY-----
...Base64 data...
-----END PRIVATE KEY-----

No idea if this could be the issue or not. I think we never supported RSA keys. No idea whether they just worked by coincidence in any previous versions. But certainly not intentionally supported.

Just to add for clarity -> I believe there is a difference in the base64 part as well - not only in the header/footer. It is simply wrong certificate format.

[marcosschroh]

Thanks for your reply. What should be the proper format for certificate and key in authentication? Is it PEM, JKS, pkcs12?

[scholzj]

PEM ... but although I'm not sure how well defined PEM is. The correct format here is I think PKCS8. Whereas what you use is PKCS1.

[marcosschroh]

I have good news, it is working!! The problem indeed was the format. Changing key format from PKCS1 to PKCS8 did the trick.

I can confirm that for the operator version <= 0.46.0 then the key format works with PKCS1 and PKCS8. From 0.47.0 the format MUST be PKCS8

Operator version	SSL Key format
<= 0.46.0	PKCS1, PKCS8
0.47.0	PKCS8

So, it is a BREAKING CHANGE, but as discussed here, the authentication module was a bit messy so we did not know about this.

And it gets more interesting. My assumption was that IF there are not ANY BREAKING CHANGES in the new version, then the code should keep working, even though the base image was the old one (0.46.0). It turns out that I was correct.

Operator version	Custom Image based on	Status with SSL KEY FORMAT
0.46.0	quay.io/strimzi/kafka:0.46.0-kafka-3.9.0	WORKS (with PKCS1 and PKCS8)
0.47.0	quay.io/strimzi/kafka:0.47.0-kafka-3.9.0	WORKS (only with PKCS8)
0.47.0	quay.io/strimzi/kafka:0.46.0-kafka-3.9.0	WORKS (only with PKCS8)
0.46.0	quay.io/strimzi/kafka:0.47.0-kafka-3.9.0	FAILS as expected

[scholzj]

I have good news, it is working!! The problem indeed was the format. Changing key format from PKCS1 to PKCS8 did the trick.

I can confirm that for the operator version <= 0.46.0 then the key format works with PKCS1 and PKCS8. From 0.47.0 the format MUST be PKCS8

Operator version SSL Key format
<= 0.46.0 PKCS1, PKCS8
0.47.0 PKCS8
So, it is a BREAKING CHANGE, but as discussed here, the authentication module was a bit messy so we did not know about this.

Glad we figured this out. I opened #11784 to make sure it is properly documented. (+ already yesterday I oepned #11781 to fix the CHANGELOG and list the move to PEM certificates in the right version)

And it gets more interesting. My assumption was that IF there are not ANY BREAKING CHANGES in the new version, then the code should keep working, even though the base image was the old one (0.46.0). It turns out that I was correct.

Operator version Custom Image based on Status with SSL KEY FORMAT
0.46.0 quay.io/strimzi/kafka:0.46.0-kafka-3.9.0 WORKS (with PKCS1 and PKCS8)
0.47.0 quay.io/strimzi/kafka:0.47.0-kafka-3.9.0 WORKS (only with PKCS8)
0.47.0 quay.io/strimzi/kafka:0.46.0-kafka-3.9.0 WORKS (only with PKCS8)
0.46.0 quay.io/strimzi/kafka:0.47.0-kafka-3.9.0 FAILS as expected

I think that this is more or less coincidence. In 0.47.0, we noved to the configuration generated by the operator. So it has less stuff it needs from the container image and thus it is more likely to work with the older image. However:

• This migth depend on the exact configuration
• This is not a pattern to rely for in other versions as the future versions might bring different types of changes

[marcosschroh]

Thanks for creating #11784. Maybe I will send a PR.

I think that this is more or less coincidence.

I think we should try to make k8s operators backward compatible so when we update operators then users are not force to change their code. Of course this is not always possible, but it is good to keep it in mind.

How would you handle an operator update with many teams having their own KafkaConnect deployments in different namespaces? If I update the operator with breaking changes, then all the KafkaConnect deployments will fail.

[im-konge]

I think we should try to make k8s operators backward compatible so when we update operators then users are not force to change their code. Of course this is not always possible, but it is good to keep it in mind.

Backwards compatibility is for us one of the main things we are taking care of - even during proposals (it's one of the sections we have there). But not always it's a really something that can be done, so in that case, we are mentioning it in our changelog and our documentation. Unfortunately, in this case it wasn't visible and this feature was added to wrong release, causing this issue.

TBH - and it's my opinion - you cannot write code that will work for all of the releases for like 10 years, it's because of other things we for example cannot influence - K8s support, Kafka features etc.

How would you handle an operator update with many teams having their own KafkaConnect deployments in different namespaces? If I update the operator with breaking changes, then all the KafkaConnect deployments will fail.

You can possibly use the pause reconciliation on the KafkaConnect clusters and change the certs, then you can just un-pause the reconciliation and continue?

[marcosschroh]

You can possibly use the pause reconciliation on the KafkaConnect clusters and change the certs, then you can just un-pause the reconciliation and continue?

Good one! I will try that. Thanks

[scholzj]

I think we should try to make k8s operators backward compatible so when we update operators then users are not force to change their code. Of course this is not always possible, but it is good to keep it in mind.

How would you handle an operator update with many teams having their own KafkaConnect deployments in different namespaces? If I update the operator with breaking changes, then all the KafkaConnect deployments will fail.

I understand the concern. But strictly speaking, I do not see any compatibility issue here. There are two things which I guess you might see as breaking. One of them clearly is not. The other one is IMHO also not, but I admit it is a bit more complicated.

1. You have to use the container image which belongs to the correct Strimzi release. This is intentional by design. If you do not want to build new image for every upgrade, you can use Kafka Connect Build or Kubernetes Image Volumes. So there is nothing breaking here.
2. The PKCS#1 thing is a bit more tricky. As it seems that it worked and does not work anymore. But there was never any intentional support for PKCS#1, it is not tested and we were not even aware anything like that works. So while I can see that this is in a way a breaking change, there is no way how we can prevent breaking features we do not know about. And we cannot freeze the whole codebase forever just to prevent any breaking changes we might not be aware because while that might give you stability, but there will be no more development and no more upgrades anyway. However, you yourself can help to at least improve the awareness of any unknown breaking changes like this by testing the release candidates (or even the builds from the main branch) in your environments and with your settings. While that might not always help to prevent everything as sometimes it might not be possible, being aware of it is at least a start.