NullPointerException When Generating Commercial Tools Average Scorecard

[vinnimous]

Summary

BenchmarkUtils v1.3 throws a NullPointerException when attempting to generate the Commercial Tools Average scorecard if 2+ commercial tools are included in the results.

 

Environment

• BenchmarkUtils Version: 1.3 (also present in current main branch as of 2026-02-09)

• Test Suite: BenchmarkJava v1.2

• Build Tool: Maven 3.x

• JDK: 17

 

Steps to Reproduce

1. Run BenchmarkUtils scorecard generation with results from 2 or more commercial security tools

2. Ensure all individual tool result files parse successfully

3. Execute: mvn org.owasp:benchmarkutils-maven-plugin:create-scorecard -DconfigFile=<config.yml>

4. Observe error after "Tool scorecards computed." message

 

Expected Behavior

BenchmarkUtils should successfully generate:

• Individual tool scorecards

• Vulnerability comparison scorecards

• Commercial Tools Average scorecard (comparing commercial tool performance)

• Overall summary scorecards

 

Actual Behavior

Process fails with:

Tool scorecards computed.

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

Error: Failed to execute goal org.owasp:benchmarkutils-maven-plugin:1.3:create-scorecard

Caused by: java.lang.NullPointerException

    at java.io.Reader.<init> (Reader.java:168)

    at java.io.InputStreamReader.<init> (InputStreamReader.java:112)

    at org.apache.commons.io.IOUtils.copy (IOUtils.java:1430)

    at org.apache.commons.io.IOUtils.toString (IOUtils.java:3406)

    at org.owasp.benchmarkutils.score.BenchmarkScore.generateVulnerabilityScorecards (BenchmarkScore.java:966)

 

Root Cause Analysis

 

Location: plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java at line 966

 

Problematic Code:

// Line 962-966 (current implementation)

// Resources in a jar file have to be loaded as streams. Not directly as Files.

InputStream vulnTemplateStream =

        CL.getResourceAsStream(scoreCardDir + "/commercialAveTemplate.html");

String html = IOUtils.toString(vulnTemplateStream, StandardCharsets.UTF_8);

 

Issue: The code concatenates a File object (scoreCardDir) with a string path, resulting in an invalid classpath resource path like:

• Generated path: results/scorecard/commercialAveTemplate.html (invalid classpath resource)

• Expected path: scorecard/commercialAveTemplate.html (valid classpath resource)

 

When getResourceAsStream() receives an invalid classpath, it returns null. This causes IOUtils.toString() to throw a NullPointerException when attempting to read from a null InputStream.

 

Comparison with Working Code: The vulnerability template loading at line 914 correctly uses a string literal:

// Line 914 (CORRECT implementation)

final String VULNTEMPLATERESOURCE = "scorecard/vulntemplate.html";

InputStream vulnTemplateStream = CL.getResourceAsStream(VULNTEMPLATERESOURCE);

 

Proposed Fix

 

File: plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java

 

Location: In method generateVulnerabilityScorecards(), approximately line 965 (line number may vary by version)

 

Change:

// BEFORE (incorrect):

InputStream vulnTemplateStream =

        CL.getResourceAsStream(scoreCardDir + "/commercialAveTemplate.html");

 

// AFTER (correct):

InputStream vulnTemplateStream =

        CL.getResourceAsStream("scorecard/commercialAveTemplate.html");

 

Patch File (for v1.3)

Note: Line numbers may vary in different versions. Verify line numbers before applying.

 

--- a/plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java

+++ b/plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java

@@ -963,7 +963,7 @@ private static void generateVulnerabilityScorecards(

                

                 // Resources in a jar file have to be loaded as streams. Not directly as Files.

                 InputStream vulnTemplateStream =

-                        CL.getResourceAsStream(scoreCardDir + "/commercialAveTemplate.html");

+                        CL.getResourceAsStream("scorecard/commercialAveTemplate.html");

                 String html = IOUtils.toString(vulnTemplateStream, StandardCharsets.UTF_8);

                 html = html.replace("${testsuite}", BenchmarkScore.TESTSUITENAME.fullName());

                 html = html.replace("${version}", TESTSUITEVERSION);

 

Alternative: Direct sed Command

If the patch fails due to line number mismatches:

sed -i 's|CL\.getResourceAsStream(scoreCardDir + "/commercialAveTemplate\.html")|CL.getResourceAsStream("scorecard/commercialAveTemplate.html")|g' \

  plugin/src/main/java/org/owasp/benchmarkutils/score/BenchmarkScore.java

 

Impact

 

Severity: High

 

Affected Users: Anyone running BenchmarkUtils scorecards with:

• 2 or more commercial security tools in their results

• BenchmarkJava, BenchmarkPython, or BenchmarkNodeJS test suites

 

Workaround:

1. Apply the patch manually to BenchmarkUtils source before building

2. OR exclude all but one commercial tool from results directory

3. OR set showAveOnlyMode: false in scoring config (if commercial averages aren't needed)

 

Tested Solution

Patch has been validated with:

• BenchmarkJava v1.2 (2740 test cases)

• 14 security tool result files including 4 commercial tools

• Successfully generates all scorecards including Commercial Tools Average

 

Additional Notes

• Bug exists in both v1.3 release and current main branch

• Similar pattern used correctly for other templates (vulntemplate.html, template.html)

• File location: plugin/src/main/resources/scorecard/commercialAveTemplate.html is correct

• Bug only manifests when commercialAveragesTable.hasEntries() returns true (2+ commercial tools)

 

References

• Resource loading pattern at line 914: CL.getResourceAsStream(VULNTEMPLATERESOURCE)

• Resource loading pattern at line 67 (ToolReport.java): getResourceAsStream(BenchmarkScore.SCORECARDDIRNAME + "/template.html")

[davewichers]

I am unable to replicate this bug. I dropped in two commercial tool results in my test and it generated the Commercial Average scorecard no problem. @darkspirit510, are you able to replicate this?

[darkspirit510]

Can't reproduce it either. It may be a wrong custom configuration file. Or (since this issue is surprisingly long) AI generated stuff. 🤷‍♂️

@vinnimous please provide the used result files and configuration file.

[vinnimous]

I did have ai compose the message so it would be complete and thorough. Commercial tools that I was adding included Snyk. The patch I included did fix the issue. I can send additional logs of the issue in action if needed. I just didn't want to do it because it included a commercial tool.

…

On Sun, Feb 15, 2026, 11:09 AM Sascha Knoop ***@***.***> wrote: *darkspirit510* left a comment (OWASP-Benchmark/BenchmarkUtils#268) <#268 (comment)> Can't reproduce it either. It may be a wrong custom configuration file. Or (since this issue is surprisingly long) AI generated stuff. 🤷‍♂️ @vinnimous <https://github.com/vinnimous> please provide the used result files and configuration file. — Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4CDQYZQ6LBTFBJ4AXEPYD4MCK4LAVCNFSM6AAAAACURNYHW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMBUG42DIMBRGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[darkspirit510]

You (or your AI) mentioned -DconfigFile=<config.yml> in third step to reproduce. Please provide your config to me (github@darkspirit510.de) or Dave (dave.wichers@owasp.org) so we investigate further, since the NPE does not occur in default mode (= not providing any config file).

[darkspirit510]

Hi Ron,
thank you for providing the files. You probably found a bug. If I see it correctly, your patch would only fix a symptom, but not a cause. I guess the root cause is that you provided a nested path as resultsfileordir (some/nested/path) in your yaml file, which seems to be broken in Benchmark.
Can you try to not use any sub-paths and see if it is working? I'll try to make Benchmark accept nested paths, but it might take one or two weeks 🙈
Sascha

[vinnimous]

Yes no path works great.

…

On Mon, Feb 16, 2026, 3:45 PM Sascha Knoop ***@***.***> wrote: *darkspirit510* left a comment (OWASP-Benchmark/BenchmarkUtils#268) <#268 (comment)> Hi Ron, thank you for providing the files. You probably found a bug. If I see it correctly, your patch would only fix a symptom, but not a cause. I guess the root cause is that you provided a nested path as resultsfileordir (some/nested/path) in your yaml file, which seems to be broken in Benchmark. Can you try to not use any sub-paths and see if it is working? I'll try to make Benchmark accept nested paths, but it might take one or two weeks 🙈 Sascha — Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4CDQZULWR5L3EK72KUIJT4MIT5BAVCNFSM6AAAAACURNYHW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMJQGQ2DGNJWGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>