Log4j2 CVE (CVE-2021-44228)

[scholzj]

Info for everyone asking about the Log4j2 CVE (CVE-2021-44228) ...

• Kafka and ZooKeeper are using Log4j1. Originally, it was suggested that Log4j1 might affected when you configured it to use the JMSAppender (which I doubt anyone did with Kafka, since that would be a bit weird) - but it seems this is not the case according to the latest information. So your Kafka brokers, ZooKeeper nodes, Kafka Connects etc. should be most probably fine.
• The affected components are the Operators, the Bridge and Cruise Control which all use Log4j2. We did new releases which fixes the CVE - we recommend everyone to upgrade to it:
  • The Bridge 0.21.0 release
  • The Operators 0.26.1 patch release which includes the new version of the Strimzi Kafka Bridge and updates the Log4j2 dependency in our Operators and in Cruise Control.
• Strimzi Drain Cleaner uses Log4j2 only as a test dependency and should not be affected.

---

If you for any reason cannot upgrade to Strimzi Operators 0.26.1 / Strimzi Kafka Bridge 0.21.0, you should be able to mitigate the CVE in existing versions by setting the Java system property -Dlog4j2.formatMsgNoLookups=true ...

• For Cluster Operator, edit the Deployment and set the environment variable JAVA_OPTS to -Dlog4j2.formatMsgNoLookups=true
• For User and Topic operators and for the Kafka Bridge, you can use the jvmOptions to set the property:

        jvmOptions:
          javaSystemProperties:
          - name: log4j2.formatMsgNoLookups
            value: "true"

  (You can do this in the following places: in the Kafka resource in .spec.cruiseControl, .spec.entityOperator.topicOperator and .spec.entityOperator.userOperator and in the KafkaBridge resource in .spec )
• For Cruise Control, you can pass the environment variable KAFKA_OPTS using the .spec.cruiseControl.template section of the Kafka custom resource:

    # ...
    cruiseControl:
      template:
        cruiseControlContainer:
          env:
          - name: KAFKA_OPTS
            value: -Dlog4j2.formatMsgNoLookups=true
    # ...

• For Kafka Bridge, set the JAVA_OPTS environment variable using the .spec.template section:

    template:
      bridgeContainer:
        env:
        - name: JAVA_OPTS
          value: -Dlog4j2.formatMsgNoLookups=true

When the pods are starting, you should see a line like this:

exec /usr/bin/tini -w -e 143 -- java ...

and there, somewhere after the java keyword, you should see the option being set. So you know that it was passed to the pods.

---

(13.12.2021: Updates with a working way for Cruise Control)
(13.12.2021: Updates with a working way for Kafka Bridge)
(13.12.2021: Update for Drain Cleaner which has Log4j2 only as a test dependency)
(14.12.2021: 0.26.1 release)

[scholzj]

Just FYI ... Strimzi Kafka Bridge 0.21.0 is now available. It will be also used in the upcoming 0.27.0 and 0.26.1 releases of Strimzi Operators. The main changes in this release are support for Arm64 / AArch64 architecture and the fix for the Log4j2 CVE-2021-44228. For more details, go to https://github.com/strimzi/strimzi-kafka-bridge/releases/tag/0.21.0

If you use an older version of the Strimzi Cluster Operator which uses Strimzi Kafka Bridge 0.20.x, you should be able to run the 0.21.0 version without any problems. Just edit the Cluster Operator deployment and set the STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE environment variable to quay.io/strimzi/kafka-bridge:0.21.0.

Thanks to everyone who contributed to this release!

[vutkin]

Hi @scholzj , could you please advise for those who uses older versions kafka strimzi: 0.22.1 for example?
Will quay.io/strimzi/kafka-bridge:0.21.0 image work too?

[TairItzhak]

Is there a way to reconfigure the log4j2 property for the Cluster Operator when using a helm chart to deploy it? (using the values.yaml)

[scholzj]

Hi @scholzj , could you please advise for those who uses older versions kafka strimzi: 0.22.1 for example?
Will quay.io/strimzi/kafka-bridge:0.21.0 image work too?

I think it should. But I haven't tested it TBH. So you will need to give it a try.

Is there a way to reconfigure the log4j2 property for the Cluster Operator when using a helm chart to deploy it? (using the values.yaml)

I do not use Helm, so I do not know, sorry.

[steadyk]

Maybe this hint might be helpful?
For a temporary fix, you can try adjust the Strimzi deployment in the helm template directly.

[scholzj]

FYI: It looks like the original way did not worked properly for Cruise Control. Please check the updated document above with the new working way to set the mitigating system property.

[scholzj]

And I updated the Kafka Bridge in the same way.

[jrivers96]

Thanks for the information!

Do you happen to know what a potential attack would look like? We only have a broker endpoint exposed to clients. Is there a way to write to the cluster operator log?

[scholzj]

I'm not aware of any way how a Kafka user would be able to trigger this issue in the operator. But someone who has already access to your Kubernetes cluster might be able to trigger it through some specially crafted custom resource or through the healthcheck / metrics HTTP endpoint. TBH, I do not have an example of how to do this, but I do not feel like I can completely exclude it either.

[Typositoire]

Anything that goes through log4j with a malicious payload (staying vague, let's not feed the internet) can start a remote code execution. No matter where the payload lives, as soon has the magic string hit log4j is it executed. User Agent, User Input, Error msg anything. If your client send that magic string and for what ever reason it hits log4j, you get insta-owned.

DISCLAIMER: Vague and exaggeration included. I'm no security expert, just someone with a lot of time on his hands and a minecraft server. You should not listen to me and this is my own understanding of the problem.

[Typositoire]

Any ETA on Operator release > 0.26.0 for CruiseControl fix ? (And the helm chart)

[scholzj]

Well ... not sure ... I spent the afternoon and evening so far today trying to release the Java artifacts ... but a lot of infrastructure is has probably issues handling the load 🤷‍♂️ .

[scholzj]

Strimzi Operators release 0.26.1 is now available. The main change in this release is the fix for the Log4j2 CVE-2021-44228. For more details, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.26.1

The updates to OperatorHub.io and OpenShift OperatorHub are still work in progress. The Java artifacts might need some additional time to propagate.

If you for some reason cannot upgrade to Strimzi 0.26.1, you can check the possible mitigations in the top post.

Thanks to everyone who contributed to this release!

[leosunmo]

For Helm users, the commit you need in order to add a custom/extra environment variable (JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true, looking at this launch script) is 06f7032 which is slated for release 0.27.0, which is one release too late since the log4j2 fix is in 0.26.1 :P

For now the best move would presumably be to upgrade your Operator release in your existing 0.26.0 Helm chart or upgrade to 0.26.1.

[giskou]

[scholzj]

Kafka and ZooKeeper do not use Log4j2. All you need to do for them is make sure you do not use the JmsAppender.

[giskou]

Ah sorry accidentally removed the comment.

[zencircle]

I see that kafka-log4j-appender-2.7.0.jar used by kafka and zookeeper. Our vulnerability scanner is flagging it for CVE-2021-44228. Any thoughts ?

[scholzj]

It is not necessarily used by Kafka or ZooKeeper - it is a component which is part of Apache Kafka but would not be used unless you configure it to be used in your logging configuration. Plus it is Log4j1 appender AFAIK, so it has no relation to CVE-2021-44228 I think.

[zencircle]

t is not necessarily used by Kafka or ZooKeeper - it is a component which is part of Apache Kafka but would not be used unless you configure it to be used in your logging configuration. Plus it is Log4j1 appender AFAIK, so it has n

Thanks, I created a new issue #6045

[dweber019]

Will settings the environmental variable LOG4J_FORMAT_MSG_NO_LOOKUPS=true have the same effect for the operators pods. More specific for Strimzi version 0.20.x?
As stated here https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/ for log4j 2.10.x to 2.14.1

[dweber019]

POM for 0.20.1 regarding log4j https://github.com/strimzi/strimzi-kafka-operator/blob/0.20.1/pom.xml#L94

[scholzj]

If Log4j2 supports it, I guess it should.

[vutkin]

Just add step into your cd for helm:

#!/usr/bin/env bash
set -x

kubectl set env deploy/strimzi-cluster-operator \
    JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true" \
    LOG4J_FORMAT_MSG_NO_LOOKUPS="true" \
    -n ${namespace}

[RoachMJ]

The release notes for 0.26 states removal of kafka 2.7, is there a release patch for any versions involving kafka 2.7.x ?

[scholzj]

No, you should upgrade or follow the mitigations recommendations using the Java system property.

[scholzj]

We put together a blog post summarising the Log4j2 issue impact on Strimzi and the available mitigations. so if you are interested, have a read: https://strimzi.io/blog/2021/12/14/strimzi-and-log4shell/

[yxiang92128]

CVE-2021-45046 was found during mitigation of CVE-2021-44228
It was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations. This could allows attackers with control over Thread Context Map (MDC) input data when the logging configuration uses a non-default Pattern

[scholzj]

As far as I understood, the new CVE is significantly less critical. So we currently do not plan another patch release.

[yxiang92128]

Thanks for the reply. Would it be address in 0.27 then?

[scholzj]

It would definitely be addressed in 0.27 (original timeline for 0.27 was before holidays, but I don't think we will make it due to the work on 0.26.1 etc.). If it proves more critical than it currently seems or if the demand is too big, we might do 0.26.2, but not sure about it yet.

[scholzj]

FYI: We discussed this on the community call today. The plan is to start working on a 0.27 release (containing Log4j2 2.16.0). It is expected to be out next week. Please follow Slack or the mailing list for the progress on that

[abrad3]

Thanks for all this info!
I haven't been able to confirm that KafkaBridge has used the -Dlog4j2.formatMsgNoLookups=true setting though- unlike the cluster, topic and user operators, I can't see exec /usr/bin/tini -w -e 143 -- java ... appearing in the bridge pod's startup logs.
Is there another way I can confirm it's picked up the setting?

[scholzj]

Yeah sorry for that, it is possible the bridge doesn't print it at start up. You should be able to check it by execing into the Pod and running a command like this:

cat /proc/47/cmdline

The number 74 might be different in your case. Bur under one of the processes, you should see a command starting with java. And there you should see the system property.

[abrad3]

We're using version 0.19.0 of the bridge, 0.21.0 of the strimzi operator btw. From looking at the source code, it's clear that it doesn't print the executing line by default https://github.com/strimzi/strimzi-kafka-bridge/blob/d3e309e5028a129a61c9090c21a1a9898fa90761/bin/kafka_bridge_run.sh

It'd be awesome if anyone knew of a log file or something where I could see what it ran though, just to be sure it's using MsgNoLookup

[abrad3]

ah thanks @scholzj ! I'll try that now

[abrad3]

incredible, I can see it in there :) Cheers

[scholzj]

Yeah, sorry for the confusion. It looks like even the new version doesn't print it unlike the other pods. And bridge does nto have the ps utility installed which would made it easier. I guess I forgot I used this method when checking it for Bridge. (anyway, printing it is useful I guess, so I will have a look why is it not printed.)

[yxiang92128]

sorry for the repeat post. But after I upgraded our cluster from 0.25 to 0.26.1 using helm:
_helm upgrade --install strimzi-kafka strimzi/strimzi-kafka-operator --version="0.26.1" -n kafka
the logs collected seemed to stop updating/refreshing (cluster log, zookeeper logs and operator logs). Do I need to do something else? And also on pomenthus side, the consumer metrics stopped collecting as well.

[scholzj]

I'm not aware of any change - I did not had any problems (not a Helm user my self, so I do not know if it can be in any way connected to that). So you see the logs with kubectl logs ...?

[yxiang92128]

yes. I ended up "kubectl delete pod" for all kafka pods and let them restart and that seemed to have solve the logging issue. But now all my consumer clients start to show random SSL handshake errors and terminate.
It complains about 5/5 brokers are down but my kubectl tells me they are up:
%6|1639587173.961|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://a6979e86f0c0a4a4abdf5fa6f3d7ff81-1621923149.us-west-]: sasl_ssl://a6979e86f0c0a4a4abdf5fa6f3d7ff81-1621923149.us-west-2.elb.amazonaws.com:9094/2: Disconnected (after 226263ms in state UP)
% Error: Local: Broker transport failure: sasl_ssl://a6979e86f0c0a4a4abdf5fa6f3d7ff81-1621923149.us-west-2.elb.amazonaws.com:9094/2: Disconnected (after 226263ms in state UP)
%3|1639587177.955|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://a6979e86f0c0a4a4abdf5fa6f3d7ff81-1621923149.us-west-]: sasl_ssl://a6979e86f0c0a4a4abdf5fa6f3d7ff81-1621923149.us-west-2.elb.amazonaws.com:9094/2: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 3992ms in state SSL_HANDSHAKE)
% Error: Local: Broker transport failure: sasl_ssl://a6979e86f0c0a4a4abdf5fa6f3d7ff81-1621923149.us-west-2.elb.amazonaws.com:9094/2: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 3992ms in state SSL_HANDSHAKE)
%6|1639587177.958|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: a6979e86f0c0a4a4abdf5fa6f3d7ff81-1621923149.us-west-2.elb.amazonaws.com:9094: Disconnected (after 237618ms in state UP)
% Error: Local: Broker transport failure: GroupCoordinator: a6979e86f0c0a4a4abdf5fa6f3d7ff81-1621923149.us-west-2.elb.amazonaws.com:9094: Disconnected (after 237618ms in state UP)
%3|1639587178.070|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://a20816610be1543abac1f0b42ea3a438-743958010.us-west-2]: sasl_ssl://a20816610be1543abac1f0b42ea3a438-743958010.us-west-2.elb.amazonaws.com:9094/0: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 110ms in state SSL_HANDSHAKE)
% Error: Local: Broker transport failure: sasl_ssl://a20816610be1543abac1f0b42ea3a438-743958010.us-west-2.elb.amazonaws.com:9094/0: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 110ms in state SSL_HANDSHAKE)
%3|1639587178.637|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://a6fc7110c9ea84e1583ee1be10c01c84-57973359.us-west-2.]: sasl_ssl://a6fc7110c9ea84e1583ee1be10c01c84-57973359.us-west-2.elb.amazonaws.com:9094/1: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 427ms in state SSL_HANDSHAKE)
% Error: Local: Broker transport failure: sasl_ssl://a6fc7110c9ea84e1583ee1be10c01c84-57973359.us-west-2.elb.amazonaws.com:9094/1: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 427ms in state SSL_HANDSHAKE)
% Error: Local: All broker connections are down: 5/5 brokers are down

The cluster seemed to be broken after 0.25 to 0.26.1 upgrade.

[2muchgit]

Hi there! For your general information the mitigation used on the operator for CVE-2021-44228 is no longer consider safe. Lunasec has been giving valuable information on the disclosure of the newly found CVE-2021-45046 which implies a complete removal of the JDNI and the messagelookup feature. Please see here:
https://www.lunasec.io/docs/blog/log4j-zero-day-mitigation-guide/

[2muchgit]

This done my updating the log4j to version 2.16 or following the other options on Lunasec link above (Live service mitigation)

[vutkin]

Ref: https://kafka.apache.org/cve-list

Summary:
CVE-2021-45046
Users should NOT be impacted by this vulnerability
CVE-2021-44228
Users should NOT be impacted by this vulnerability
CVE-2021-4104
Version 1.x of Log4J can be configured to use JMS Appender, which publishes log events to a JMS Topic. Log4j 1.x is vulnerable if the deployed application is configured to use JMSAppender.

[jains20171]

Is Kafka Connect (v0.26.0) impacted by the log4j issue? Pls ignore, see the answer in the blog that kafka connect is not impacted.

[scholzj]

Kafka Connect (and Kafka brokers, ZooKeeper, and MirrorMaker 1 & 2 etc.) is using Log4j1 not Log4j2.

[ssimonexfo]

Hi
what about CVE-2021-45105 ?

It seems that Log4j 2.17.0 is required to fix it.
Is there any new strimzi release planned?

[scholzj]

It will be part of 0.27.0.

[ssimonexfo]

Any expected release date ?

[Sidhardha95]

Hi Team,

We were on very older version of Strmizi (0.18.0 ) and we are planning to move to 0.27.1.

As part of the upgrade , we got to know that we need to first upgrade to 0.22.x and then to desired version. As part of upgrade to 0.22.0 there is an api-conversion utility that needs to be ran : api-conversion-0.22.0.zip.

But this util is using an affected log4J core jar (2.13.3), So it is flagged as a issue in our environment. What can be done to avoid this? Can we manually replace log4J core jar to latest version in libs directory and try the upgrade ? If that works we can package it internally and use it

[scholzj]

Yeah, I think you can simply just replace the JARs if you want. Also, it is a utility you run basically twice from command line, not some server where others connect. So I'm not sure if the Log4j2 issues can be exploited there in any easy way.