Build Docker image with class collision

[guwei0kanyun]

Info

Strimzi Version: 0.45.x

Question

When we build operator docker image with command:

DOCKER_ORG=xx DOCKER_PLATFORM="xx" DOCKER_REGISTRY=xx DOCKER_TAG=xx make MVN_ARGS='-DskipTests' all

It gives errors below
../find-classpath-collision.sh strimzi/kafka:build-kafka-3.8.0 /opt/kafka/libs "tmp/3.8.0.ignorelist"
Successfully copied 152MB to /var/folders/yw/yt4c5wj12hdc5kvyz0d4lkzm0000gn/T/tmp.rbzuHCX5ib
ERROR: Different class files with same name from different jars found!
kafka-server-common-3.8.0.jar org/apache/kafka/timeline/SnapshotRegistry.class
kafka-server-common-3.9.0.jar org/apache/kafka/timeline/SnapshotRegistry.class

It seems there are class collisions.

How can we find out which lib jar cause this problem?

Any Tips are appreciated
Thanks

[im-konge]

Hi, did you try to run make clean before the make all?
This can happen when you have there old binaries for the previous versions.
I had similar issue when adding new version of plugin, I had to clean it, especially the binaries folder.

[guwei0kanyun]

I tried it and end with the same error. It seems this is not the solution for my case.

But anyway thanks for your suggestion.

[im-konge]

Hmm it should clear everything really.
Did you added some plugin that would download it again?

[guwei0kanyun]

yes, we plugin cruise-control in our cluster.
For Broker depends on cruise-control metric reporter, so we update cruise-control's version which is compatible with 3.9.0 kafka.

I have tried exclude kafka-server-common below cruise-control dependency, and it didn't work.

[guwei0kanyun]

As a temporary and somewhat risky solution, adding

kafka-server-common-3.9.0.jar
kafka-storage-3.9.0.jar
kafka-raft-3.9.0.jar
kafka-server-3.9.0.jar
kafka-storage-api-3.9.0.jar
kafka-metadata-3.9.0.jar
kafka-transaction-coordinator-3.9.0.jar
kafka-group-coordinator-3.9.0.jar
kafka-group-coordinator-api-3.9.0.jar
kafka-server-common-3.8.0.jar
kafka-storage-3.8.0.jar
kafka-raft-3.8.0.jar
kafka-server-3.8.0.jar
kafka-storage-api-3.8.0.jar
kafka-metadata-3.8.0.jar
kafka-transaction-coordinator-3.8.0.jar
kafka-group-coordinator-3.8.0.jar
kafka-group-coordinator-api-3.8.0.jar
kafka-server-common-3.8.1.jar
kafka-storage-3.8.1.jar
kafka-raft-3.8.1.jar
kafka-server-3.8.1.jar
kafka-storage-api-3.8.1.jar
kafka-metadata-3.8.1.jar
kafka-transaction-coordinator-3.8.1.jar
kafka-group-coordinator-3.8.1.jar
kafka-group-coordinator-api-3.8.1.jar

in docker-images/kafka-based/kafka/tmp/3.8.0.ignorelist and docker-images/kafka-based/kafka/tmp/3.8.1.ignorelist can bypass this error and prevent the blockage of the image build.

docker-images/kafka-based/kafka/tmp/3.9.0.ignorelist remains as it was is ok. There is no class collisions in building 3.9.0 version image

[ShubhamRwt]

Looking into your ignore list file, why do you have to add a lot of these Kafka related jars? Are you sure you are just overriding the version of cruise-control-metrics-reporter to your desired version in the 3.8.0 pom file since we only include the metrics reporter jar in the kafka libs as it runs on broker side

[guwei0kanyun]

Looking into your ignore list file, why do you have to add a lot of these Kafka related jars? Are you sure you are just overriding the version of cruise-control-metrics-reporter to your desired version in the 3.8.0 pom file since we only include the metrics reporter jar in the kafka libs as it runs on broker side

1. Because the error log clearly mentions that some classes in these jars are "Different class files with same name from different jars found!"
2. Yes, I only changed the version below in docker-images/artifacts/kafka-thirdparty-libs/3.8.x/pom.xml by checking git history of the file

<dependency>
      <groupId>com.rockee.cruisecontrol</groupId>
      <artifactId>cruise-control-metrics-reporter</artifactId>
      <version>${cruise-control.version}</version>
      <exclusions>
          <exclusion>
              <groupId>org.apache.kafka</groupId>
              <artifactId>kafka_2.13</artifactId>
          </exclusion>
          <exclusion>
              <groupId>org.apache.kafka</groupId>
              <artifactId>kafka-clients</artifactId>
          </exclusion>
          <exclusion>
              <groupId>org.slf4j</groupId>
              <artifactId>slf4j-api</artifactId>
          </exclusion>
          <exclusion>
              <groupId>org.apache.logging.log4j</groupId>
              <artifactId>log4j-slf4j-impl</artifactId>
          </exclusion>
          <exclusion>
              <groupId>com.fasterxml.jackson.core</groupId>
              <artifactId>jackson-databind</artifactId>
          </exclusion>
      </exclusions>
  </dependency>

[kyguy]

Hi @guwei0kanyun building and debugging custom images can be tricky. If you push your code changes to a branch on your remote Strimzi fork I can take a closer look if you want!

[guwei0kanyun]

so sry that we are not allowed to do that, even though I really want to.

[kyguy]

It's difficult to debug the issue without a reproducer, code, or diff. If those cannot be shared, the only recommendation I can provide (other than Lukas's suggestion) is to clone a new local fork, make the custom code changes on top of one of the Strimzi release branches, then trying building there.

On a side note, we generally keep Strimzi in line with the latest Cruise Control version, we were waiting on some upstream Cruise Control work before the next upgrade but someone could bump the Cruise Control version on Strimzi main branch if that is all that is needed!