trivy scan failure for strimzi-kafka-operator

[ns-jhu]

I did some trivy security scan (https://github.com/aquasecurity/trivy) with strimzi-kafka-operator image quay.io/strimzi/operator:0.34.0 , found these errors:

┌───────────────────────────────────────────────────────────┬───────────────┬──────────┬───────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├───────────────────────────────────────────────────────────┼───────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ net.minidev:json-smart │ CVE-2023-1370 │ HIGH │ 2.4.7 │ 2.4.9 │ Json-smart is a │
│ (com.nimbusds.nimbus-jose-jwt-9.10.jar) │ │ │ │ │ performance f ... │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-1370 │
├───────────────────────────────────────────────────────────┤ │ │ │ │ │
│ net.minidev:json-smart (net.minidev.json-smart-2.4.7.jar) │ │ │ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ │
├───────────────────────────────────────────────────────────┼───────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ org.yaml:snakeyaml (org.yaml.snakeyaml-1.33.jar) │ CVE-2022-1471 │ CRITICAL │ 1.33 │ 2.0 │ SnakeYaml: Constructor Deserialization Remote Code Execution │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-1471 │
└───────────────────────────────────────────────────────────┴───────────────┴──────────┴───────────────────┴───────────────┴──────────────────────────────────────────────────────────────┘

basically com.nimbusds.nimbus-jose-jwt-9.10.jar need to be upgraded to
version https://repo1.maven.org/maven2/com/nimbusds/nimbus-jose-jwt/9.24.4/nimbus-jose-jwt-9.24.4.jar

net.minidev.json-smart-2.4.7.jar need to be upgraded to version

wget https://repo1.maven.org/maven2/net/minidev/json-smart/2.4.9/json-smart-2.4.9.jar

org.yaml.snakeyaml-1.33.jar need to upgrade to version
(https://repo1.maven.org/maven2/org/yaml/snakeyaml/2.0/snakeyaml-2.0.jar)

however, when I made the jar switch, I got this exception:

Exception in thread "main" java.lang.NoSuchMethodError: 'void org.yaml.snakeyaml.parser.ParserImpl.(org.yaml.snakeyaml.reader.StreamReader)'
at com.fasterxml.jackson.dataformat.yaml.YAMLParser.(YAMLParser.java:191)
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:509)
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:413)
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:15)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3698)
at io.strimzi.operator.cluster.model.KafkaVersion.parseKafkaVersions(KafkaVersion.java:48)
at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.(KafkaVersion.java:113)
at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.(KafkaVersion.java:100)
at io.strimzi.operator.cluster.ClusterOperatorConfig.parseKafkaVersions(ClusterOperatorConfig.java:457)
at io.strimzi.operator.cluster.ClusterOperatorConfig.fromMap(ClusterOperatorConfig.java:293)
at io.strimzi.operator.cluster.Main.main(Main.java:81)

it turns out, upgrade org.yaml.snakeyaml-1.33.jar to version 2.0 doesn't work because the internal API changed. I then tried all the versions listed in https://repo1.maven.org/maven2/org/yaml/snakeyaml/ , the problem is , snakeyaml version lower than 2.0 can not pass the scan, snakeyaml version 2.0 or above doesn't work because of the API change.

Can somebody help to upgrade the version for strimzi-kafka-operator ?

I also found similar issue for docker image quay.io/strimzi/kafka:0.34.0-kafka-3.3.2
and I end up have to download these

RUN wget https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.9/scala-library-2.13.9.jar
RUN wget https://repo1.maven.org/maven2/com/nimbusds/nimbus-jose-jwt/9.24.4/nimbus-jose-jwt-9.24.4.jar
RUN wget https://repo1.maven.org/maven2/net/minidev/json-smart/2.4.9/json-smart-2.4.9.jar
RUN wget https://repo1.maven.org/maven2/org/yaml/snakeyaml/2.0/snakeyaml-2.0.jar
RUN wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.18.0/jmx_prometheus_javaagent-0.18.0.jar

to replace the jar files inside the images

RUN /bin/rm -rf /opt/cruise-control/libs/scala-library-2.13.6.jar /opt/kafka/libs/scala-library-2.13.8.jar
RUN cp /download/scala-library-2.13.9.jar /opt/cruise-control/libs/
RUN cp /download/scala-library-2.13.9.jar /opt/kafka/libs/
RUN /bin/rm -rf /opt/cruise-control/libs/nimbus-jose-jwt-9.21.jar /opt/kafka/libs/nimbus-jose-jwt-9.10.jar
RUN cp /download/nimbus-jose-jwt-9.24.4.jar /opt/cruise-control/libs/
RUN cp /download/nimbus-jose-jwt-9.24.4.jar /opt/kafka/libs/
RUN /bin/rm -rf /opt/kafka/libs/json-smart-2.4.7.jar
RUN cp /download/json-smart-2.4.9.jar /opt/kafka/libs/
RUN /bin/rm -rf /opt/cruise-control/libs/snakeyaml-1.33.jar /opt/kafka/libs/snakeyaml-1.33.jar
RUN cp /download/snakeyaml-2.0.jar /opt/kafka/libs/
RUN cp /download/snakeyaml-2.0.jar /opt/cruise-control/libs/
RUN /bin/rm -rf /opt/kafka/libs/jmx_prometheus_javaagent-0.17.2.jar
RUN cp /download/jmx_prometheus_javaagent-0.18.0.jar /opt/kafka/libs/

so pass the security scan. luckily , this actually worked. Can we update the jar file version inside the pom file so we don't have to make this change manually ?

[scholzj]

This is not really how it works that you just randomly go and replace some JARs without any testing and so on. The dependencies need to be fixed at source where they understand how they are used, can properly test the fix etc.

[ns-jhu]

yes, exactly, that is what I request. can the maintainer help to change the jar version and make the corresponding java code change, so that the built image come with the trivy compliant jar file version ?

[scholzj]

I'm not sure you understand how the dependencies work. There are different projects which bring their own dependencies which bring their own dependencies and so on. And these all end up in the image. So you need to first trace which project actually pulls the dependency. That might be Strimzi or some other completely different project => that depends on the dependency. Once you trace where it needs to be updated, you can try to have a look at what can be done about it. You can open a PR to help to get it fixed and tested for example. And once the fix is released, it can bubble up the chain into Strimzi. You should also expect that it is not always just bumping some dependnecy as there might be breaking changes and code fixes etc.

[ns-jhu]

Yes, I understand how it works.

Here is the dependency for current main branch:
[INFO] +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.10.5:compile
[INFO] | - org.yaml:snakeyaml:jar:1.26:compile

unfortunately, the newest version of released official version of jackson-dataformat-yaml is 2.14.2

[INFO] +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.14.2:compile
[INFO] | - org.yaml:snakeyaml:jar:1.33:compile

and it the snakeyaml of version 1.33 doesn't meet the security requirement either.

and we know that we can't swap snakeyaml:jar to version 2.0 because the API change. so the only solution seems to be have a fix of snakeyaml:jar in 1.x.x version with GHSA-mjmj-j48q-9wg2 fixed.

[ns-jhu]

the issue related to snakeyaml is opensearch-project/security#2363, unfortunately, they only released 2.0 version with this fix and did not back port it to 1.x.x

[scholzj]

I don't think the 1.x.x backport will happen -> for many reasons, but among others, because AFAIU the fix itself breaks the backward compatibility. So the right solution is to move everything to use SnakeYAML 2.x. We did some of that in Strimzi main branch already where we could. Other projects need to do that as well. It will eventually happen, it just takes time.

[ns-jhu]

I tried the latest release of quay.io/strimzi/operator:0.34.0, swapped the 1.3 version of snakeyaml with the 2.0, got this error:

Exception in thread "main" java.lang.NoSuchMethodError: 'void org.yaml.snakeyaml.parser.ParserImpl.(org.yaml.snakeyaml.reader.StreamReader)'
at com.fasterxml.jackson.dataformat.yaml.YAMLParser.(YAMLParser.java:191)
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:509)
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:413)
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:15)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3698)
at io.strimzi.operator.cluster.model.KafkaVersion.parseKafkaVersions(KafkaVersion.java:48)
at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.(KafkaVersion.java:113)
at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.(KafkaVersion.java:100)
at io.strimzi.operator.cluster.ClusterOperatorConfig.parseKafkaVersions(ClusterOperatorConfig.java:457)
at io.strimzi.operator.cluster.ClusterOperatorConfig.fromMap(ClusterOperatorConfig.java:293)
at io.strimzi.operator.cluster.Main.main(Main.java:81)

and then I did some investigation in https://github.com/orgs/strimzi/discussions/8360#discussioncomment-5556810

unfortunately, the dependency is via com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.10.5:compile
I checked the versions of com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar
the latest official release is 2.14.2, which still depends on 1.3.3 version of snakeyaml.

to solve this problem, we have to release a newer version of jackson-dataformat-yaml:jar changing it to 2.x version of snakeyaml, then change the strimzi kafka operator, then we can have a docker image with this security issue fixed. because of the above exception,
the strimzi kafka operator java code probably also need to change too. this is a long chain.

[ns-jhu]

I discussed the required change for snakeyaml in https://github.com/orgs/strimzi/discussions/8360#discussioncomment-5557425. turns out the is a long chain of change in order for this to happen.

[ns-jhu]

opened ticket FasterXML/jackson-dataformats-text#410

[ns-jhu]

opened request FasterXML/jackson-dataformats-text#410. once that is done, the new version of jackson-dataformat-yaml need to be used in the strimzi kafka operator and other strimzi repo to remove the dependency.

[ns-jhu]

@scholzj , can you help to make this change in the main branch ? 2.15.0-rc2 is the newest version of jackson release right now
and it has switched to snakeyaml 2.0 support. no 1.x version of snakeyaml can pass this security scan. and upgrade jackson to 2.15 is the only way to solve this problem.

I tried this in the pom.xml file and run
"mvn install" and it compiled successfully. Unfortunately, I don't have privilege to submit a PR.

< <fasterxml.jackson-core.version>2.14.2</fasterxml.jackson-core.version>
< <fasterxml.jackson-databind.version>2.14.2</fasterxml.jackson-databind.version>
< <fasterxml.jackson-dataformat.version>2.14.2</fasterxml.jackson-dataformat.version>
< <fasterxml.jackson-annotations.version>2.14.2</fasterxml.jackson-annotations.version>

    <fasterxml.jackson-core.version>2.15.0-rc2</fasterxml.jackson-core.version>
    <fasterxml.jackson-databind.version>2.15.0-rc2</fasterxml.jackson-databind.version>
    <fasterxml.jackson-dataformat.version>2.15.0-rc2</fasterxml.jackson-dataformat.version>
    <fasterxml.jackson-annotations.version>2.15.0-rc2</fasterxml.jackson-annotations.version>

mvn dependency:tree

shows the snakeyaml is changed to 2.0:

[INFO] +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.14.2:compile
[INFO] | - org.yaml:snakeyaml:jar:2.0:compile

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Strimzi - Apache Kafka on Kubernetes and OpenShift 0.35.0-SNAPSHOT:
[INFO]
[INFO] Strimzi - Apache Kafka on Kubernetes and OpenShift . SUCCESS [ 2.768 s]
[INFO] kafka-agent ........................................ SUCCESS [ 2.407 s]
[INFO] mirror-maker-agent ................................. SUCCESS [ 1.112 s]
[INFO] tracing-agent ...................................... SUCCESS [ 1.196 s]
[INFO] test ............................................... SUCCESS [ 2.683 s]
[INFO] crd-annotations .................................... SUCCESS [ 1.358 s]
[INFO] crd-generator ...................................... SUCCESS [ 4.395 s]
[INFO] api ................................................ SUCCESS [ 24.735 s]
[INFO] mockkube ........................................... SUCCESS [ 1.699 s]
[INFO] config-model ....................................... SUCCESS [ 1.224 s]
[INFO] config-model-generator ............................. SUCCESS [ 26.514 s]
[INFO] certificate-manager ................................ SUCCESS [ 1.482 s]
[INFO] operator-common .................................... SUCCESS [ 3.991 s]
[INFO] topic-operator ..................................... SUCCESS [ 5.211 s]
[INFO] cluster-operator ................................... SUCCESS [ 7.775 s]
[INFO] user-operator ...................................... SUCCESS [ 3.592 s]
[INFO] kafka-init ......................................... SUCCESS [ 2.453 s]
[INFO] systemtest ......................................... SUCCESS [ 6.299 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:41 min
[INFO] Finished at: 2023-04-10T10:28:28-07:00

$ git commit -m "upgrade snakeyaml to 2.0 version to solve CVE-2022-1471 issue" .
[CVE-2022-1471 abb670169] upgrade snakeyaml to 2.0 version to solve CVE-2022-1471 issue
1 file changed, 4 insertions(+), 4 deletions(-)

$ git push origin CVE-2022-1471
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
ERROR: Permission to strimzi/strimzi-kafka-operator.git denied to ns-jhu.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[scholzj]

No, sorry:

• We are not using any release candidate releases. And you should check if it is actually needed because I doubt it.
• You also need to make sure the Jackson version is in sync with what all the dependencies use and not just randomly update it. As I said right at the start -> this is way more complicated than just seeing some scanner produce some report and then randomly bumping some dependencies.
• You cannot push into someone else GitHub repositories just like that. You need to create a fork and use your own fork for any of your changes. I'm sure there will be some tutorial in the GitHub docs.