Investigate SBOM Irreproducibility in `log4j-bom`

[ppkarwasz]

Since version 2.25.0, the aggregated SBOM generated for the log4j-bom artifact is not reproducible. Specifically, two variants of the SBOM are occasionally produced, differing only in the ordering of the jspecify dependency.

To ensure full reproducibility across releases, we need to identify the root cause of this nondeterministic behavior and propose a solution to resolve it.

[ramanathan1504]

Hi, I’d like to take this up. I’ll start by reproducing the issue by running multiple builds of Log4j and comparing the generated SBOM outputs. If I find non-deterministic fields (e.g., timestamps, UUIDs, or ordering), I’ll propose a fix in the plugin configuration or build setup to make the SBOM reproducible.

[ppkarwasz]

Hi @ramanathan1504,

Thanks for taking this on!

To help guide your investigation, I’ve already summarized the known variations in jvm-repo-rebuild/reproducible-central#3119 (comment). Two of the three issues (those linked to the Gradle Module Metadata Maven Plugin) should already be resolved with the plugin’s latest version.

That leaves one remaining issue that still needs explanation:

• The aggregate SBOM file (log4j-bom-2.25.1-cyclonedx.xml) occasionally reorders the JSpecify dependency, shifting it between two positions. I noticed this also occurred in 2.25.0, as you had already flagged. I'm not yet sure what causes this nondeterminism—I plan to investigate and will report back once I understand the root cause.

While it’s technically possible the CycloneDX Maven Plugin itself is responsible, that seems unlikely: its maintainer, @hboutemy, is a leading expert in Java reproducibility. A more probable explanation is that the nondeterminism originates in Maven internals (e.g., the DependencyCollectorBuilder, which the plugin depends on).

One concrete step you could try is downgrading Maven to see if the issue persists. That should help determine whether the problem is tied to a specific Maven version.

[ramanathan1504]

Hi @ppkarwasz ,

Thanks a lot for the detailed context and for linking the reproducibility discussion.

Here’s my plan:
1. Reproduce consistently – I’ll first try multiple clean builds of log4j-bom to capture the nondeterministic JSpecify ordering.
2. Maven version check – I’ll then re-run the builds using different Maven versions to see if the issue is tied to a specific version or persists across them.
3. Narrow down root cause – If the nondeterminism only appears with certain versions, I’ll dig into the Maven internals (DependencyCollectorBuilder, as you mentioned). If it happens everywhere, I’ll look at how the CycloneDX plugin interacts with Maven dependency resolution.

I’ll report back here with reproducible steps and results once I’ve tested across versions.

[ramanathan1504]

Hi @ppkarwasz ,

Thanks for the guidance on this issue! I followed your suggestion and did a deep dive into how different Maven versions affect the build, and also looked into the history of the Log4j build to pinpoint the origin of the problem. The results are very interesting and I think they help narrow down the problem.

Part 1: Investigation of Maven Versions for Log4j 2.25.1

I started by testing the 2.25.1 build with older Maven versions. The conclusion is that the cyclonedx.xml issue is present in all supported Maven versions for the project (3.8.1 and newer), so simply downgrading does not solve the problem. In fact, downgrading makes the build worse by introducing new .module file failures.

Click to view all Maven version test logs for 2.25.1

• Maven 3.9.8 Log (ko=30):

  version=2.25.1
  ko=30
  koFiles="log4j-bom-2.25.1-cyclonedx.xml log4j-api-2.25.1-sources.jar..."
  

• Maven 3.9.5 Log (ko=59):

  version=2.25.1
  ko=59
  koFiles="log4j-bom-2.25.1-cyclonedx.xml log4j-api-2.25.1-sources.jar log4j-api-2.25.1.module..."
  

• Maven 3.8.8 Log (ko=59):

  version=2.25.1
  ko=59
  koFiles="log4j-bom-2.25.1-cyclonedx.xml log4j-api-2.25.1-sources.jar log4j-api-2.25.1.module..."
  

• Maven 3.6.3 Failure Log:

  [ERROR] Detected Maven Version: 3.6.3 is not in the allowed range [3.8.1,).
  

---

Part 2: Finding the Origin in Log4j's History

Since the Maven version wasn't the key, I went back through older Log4j releases to see when the cyclonedx.xml issue first appeared. This turned out to be very informative and has pinpointed the exact moment the behavior changed.

Here is the timeline I found:

• 2.25.1: The build has ko=30, with log4j-bom-2.25.1-cyclonedx.xml being one of the failures.
• 2.25.0: The build has ko=1, and the only failure is log4j-bom-2.25.0-cyclonedx.xml. This confirms your observation.
• 2.23.1: The log4j-bom-2.23.1-cyclonedx.xml file failed (ko=90 overall).
• 2.23.0: The log4j-bom-2.23.0-cyclonedx.xml file was reproducible (ok).
• 2.22.1 & 2.22.0: The cyclonedx.xml file was also reproducible (ok).
• 2.21.1: The cyclonedx-maven-plugin was not yet used in the build.

Click to view all historical Log4j build logs

• Log4j 2.25.1 (ko=30):

  version=2.25.1
  ko=30
  koFiles="log4j-bom-2.25.1-cyclonedx.xml log4j-api-2.25.1-sources.jar..."
  

• Log4j 2.25.0 (ko=1):

  version=2.25.0
  ok=148
  ko=1
  koFiles="log4j-bom-2.25.0-cyclonedx.xml"
  

• Log4j 2.23.1 (ko=90):

  version=2.23.1
  ko=90
  koFiles="log4j-bom-2.23.1-cyclonedx.xml log4j-api-2.23.1.jar..."
  

• Log4j 2.23.0 (ko=22):

  version=2.23.0
  ko=22
  okFiles="...log4j-bom-2.23.0-cyclonedx.xml..."
  

• Log4j 2.22.1 (ko=1):

  version=2.22.1
  ko=1
  okFiles="...log4j-bom-2.22.1-cyclonedx.xml..."
  

• Log4j 2.22.0 (ko=22):

  version=2.22.0
  ko=22
  okFiles="...log4j-bom-2.22.0-cyclonedx.xml..."
  

• Log4j 2.21.1 (ko=1):

  version=2.21.1
  ko=1
  # cyclonedx.xml file is not present in build results
  

---

Summary

This data suggests that the non-deterministic behavior is a latent issue in Maven that is being triggered by a specific change made inside the Log4j project between versions 2.23.0 and 2.23.1.

The cyclonedx-maven-plugin was introduced in version 2.22.0 and was reproducible for several releases. However, a change to the project's dependencies or build configuration in 2.23.1 began to trigger the non-deterministic ordering from Maven that persists today.

Hopefully, this helps narrow down the search for the root cause. Please let me know what you think or what I can investigate next.

[vy]

@ramanathan1504, would you mind briefly explaining (and sharing?) which parts of the CDX XML is causing reproducibility failures, please?

[ramanathan1504]

@vy

Here is the exact diffoscope output.

2.23.1

Different between the local build and the official reference file.

Result:

--- target/reference/org.apache.logging.log4j/log4j-bom-2.23.1-cyclonedx.xml
+++ target/bom.xml
│   --- target/reference/org.apache.logging.log4j/log4j-bom-2.23.1-cyclonedx.xml
├── +++ target/bom.xml
│ @@ -1,9 +1,9 @@
│  <?xml version="1.0" encoding="utf-8"?>
│ -<bom xmlns="http://cyclonedx.org/schema/bom/1.5" version="1" serialNumber="urn:uuid:79b4da93-8b1e-3748-8382-c91b2b199d8b">
│ +<bom xmlns="http://cyclonedx.org/schema/bom/1.5" version="1" serialNumber="urn:uuid:95e1b73f-02b8-3c9b-aa2e-f6c0d2869ac2">
│    <metadata>
│      <tools>
│        <tool>
│          <vendor>OWASP Foundation</vendor>
│          <name>CycloneDX Maven plugin</name>
│          <version>2.7.10</version>
│          <hashes>

2.25.1

The diffoscope output for the current version..

Result:

--- target/reference/org.apache.logging.log4j/log4j-bom-2.25.1-cyclonedx.xml
+++ target/bom.xml
...
│ -    <component type="library" bom-ref="pkg:maven/org.jspecify/jspecify@1.0.0?type=jar">
...
│      <component type="library" bom-ref="pkg:maven/org.apache.logging.log4j/log4j-api@2.25.1?type=jar">
...
│ +    <component type="library" bom-ref="pkg:maven/org.jspecify/jspecify@1.0.0?type=jar">
...

[ppkarwasz]

@ramanathan1504,

Have you tried setting the CI environment variable to true? Some of our Maven profiles are activated (or deactivated) based on this flag.

The profiles used during a release build are:

CI=true ./mvnw -Pdeploy,release

⚠️ Note: the deploy profile is not compatible with artifact:compare. Since we don’t use the Maven Deploy plugin, verification is skipped with the message:

Skipping goal because module skips install and/or deploy

To accurately reproduce the release artifacts, you should run:

CI=true ./mvnw -Prelease verify -DskipTests artifact:compare

[hboutemy]

I just tested by hand and confirm that the CI=true env variable made the job: now we have the root cause of the difference

I still need to see how to inject that in containerized rebuild that Reproducible Central runs through GHA, but it's now clear that the release has no issue, it's the rebuild recipe that requires additional config

great job @ppkarwasz , thank you