Code Coverage is falsely calculated in deploy command reports

[xL3o]

I first wanted to thank you for providing this plugin! It does exactly what it should.

However, once I import the code coverage report to SonarQube's generic test coverage report path (sonar.coverageReportPaths), it shows the uncovered lines but the code coverage stays at 0%.

Example:

The apex unit tests cover all lines of the SP_Access...cls file except 70-75. This is also shown in SonarQube:

Nevertheless, the Code Coverage for that class is 0%.

Is it a SonarQube configuration issue, or does the code coverage report need to contain all covered lines also?

[mcarvin8]

Thanks @xL3o for reporting this issue! Appreciate seeing feedback.

When I initially developed this plugin and added all lines (covered and uncovered) to the XML, I noticed that the Salesforce CLI output (JSON) contained extra line numbers as "covered" which weren't in the file. For example, it would report Line 100 as "covered" when there was only 98 lines of code in the class. I haven't tested this in a bit, so I'm unclear if it's fixed or reproducible by other users. Before I raise a Salesforce CLI bug on Salesforce's repo, I would like to do the following:

1. Create a Pre-Release version which adds covered and uncovered lines to the XML.
2. Have you download the pre-release version on your end and see if you can get SonarQube to accept the file. It will not accept it if the CLI is reporting Out-of-Range lines as "Covered".

If you can replicate that error, I'll raise a bug with the Salesforce CLI team.

If you can't replicate that error, I can release a new version of this plugin with those updates.

[mcarvin8]

In terms of why SQ is not updating the Code Coverage metric in the 1 file explorer view, let me check on my end if I can replicate that.

I know I have confirmed coverage when looking at the file directly. I can't recall at the moment if the other view displayed coverage %.

[mcarvin8]

@xL3o - The pre-release/beta build which will add all lines (covered and uncovered) to the XML has been published.

sf plugins install apex-code-coverage-transformer@1.4.1-beta.1

Please test and let me know how that works out for you when you get a moment.

[xL3o]

@mcarvin8 that did not quite work. Now for files that contain uncovered lines, the report always starts with the first uncovered line and adds covered lines below it:

Those lines are also out-of-range now:

[xL3o]

@mcarvin8 that did not quite work. Now for files that contain uncovered lines, the report always starts with the first uncovered line and adds covered lines below it: Those lines are also out-of-range now:

But the covered:uncovered ratio is right. 5 Uncovered and 30 Covered. Just the Covered line numbers are incorrect.

[xL3o]

I believe you can open a bug with the salesforce CLI team:

The class has only 80 lines but the .json reports >80 line numbers...

[mcarvin8]

forcedotcom/cli#1568

@mshanemc - Seems like the above issue on the CLI repository explains the current issue we are seeing while troubleshooting this plugin. Out-Of-Range lines are being generated in the JSON file by the latest Salesforce CLI in most scenarios, which causes SonarQube to fail to parse the code coverage XML created by this plugin (this plugin just converts the JSON to an XML SonarQube will accept).

I don't know if you would like me to create a new bug in the CLI, but you closed the above since it's working as "intended" but you had a longer-term plan to correct this behavior.

In the latest release of this plugin, I only add "uncovered" lines to the XML file. But I'm trying to troubleshoot this issue @xL3o is seeing in SonarQube, so I decided to release a "beta" build which adds covered and uncovered lines to the XML.

Side-Note: I'm suspecting this issue could stem from something in the SQ configuration if we keep the current logic intact on the main branch (only add uncovered lines to the XML which is accurate).

[mcarvin8]

Tests Done So Far:

SonarQube Version - Enterprise EditionVersion 9.9.3 (build 79811)

Test 1 - apex-code-coverage-transformer@1.4.1-beta.1 -Beta build which has covered and uncovered lines.

Salesforce CLI command - sf project deploy validate -l RunSpecifiedTests -t PrepareMySandboxTest --coverage-formatters json --results-dir coverage -x manifest/package.xml -w 240 --verbose

Plugin command - sf apex-code-coverage transformer transform -j "coverage/coverage/coverage.json"

Sonar CLI command - sonar-scanner -Dsonar.qualitygate.wait=$SONAR_GATE -Dsonar.projectKey=$CI_PROJECT_PATH_SLUG -Dsonar.gitlab.ref_name=$SONAR_REF -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHORT_SHA -Dsonar.coverageReportPaths=coverage.xml

The XML created by the plugin, which has less lines of code than the actual file does. But the Uncovered Lines are correct.

<?xml version="1.0"?>
<coverage version="1">
	<file path="force-app/main/default/classes/PrepareMySandbox.cls">
		<lineToCover lineNumber="7" covered="false"/>
		<lineToCover lineNumber="8" covered="true"/>
		<lineToCover lineNumber="9" covered="true"/>
		<lineToCover lineNumber="10" covered="true"/>
		<lineToCover lineNumber="11" covered="true"/>
		<lineToCover lineNumber="12" covered="false"/>
		<lineToCover lineNumber="13" covered="true"/>
		<lineToCover lineNumber="14" covered="true"/>
		<lineToCover lineNumber="15" covered="true"/>
		<lineToCover lineNumber="16" covered="true"/>
		<lineToCover lineNumber="17" covered="true"/>
		<lineToCover lineNumber="18" covered="true"/>
		<lineToCover lineNumber="19" covered="true"/>
		<lineToCover lineNumber="20" covered="true"/>
		<lineToCover lineNumber="21" covered="true"/>
		<lineToCover lineNumber="22" covered="true"/>
		<lineToCover lineNumber="23" covered="true"/>
		<lineToCover lineNumber="24" covered="true"/>
		<lineToCover lineNumber="25" covered="true"/>
		<lineToCover lineNumber="26" covered="true"/>
		<lineToCover lineNumber="27" covered="true"/>
		<lineToCover lineNumber="28" covered="true"/>
		<lineToCover lineNumber="29" covered="true"/>
		<lineToCover lineNumber="30" covered="true"/>
		<lineToCover lineNumber="31" covered="true"/>
		<lineToCover lineNumber="32" covered="true"/>
		<lineToCover lineNumber="33" covered="true"/>
		<lineToCover lineNumber="34" covered="true"/>
		<lineToCover lineNumber="35" covered="true"/>
		<lineToCover lineNumber="36" covered="true"/>
		<lineToCover lineNumber="37" covered="true"/>
	</file>
</coverage>

In SQ, the coverage indicators stop at the Line 37

But, I do get the Coverage % to show in the "Code" menu

[mcarvin8]

Test 2 - apex-code-coverage-transformer@1.4.0 - Latest Build which just prints Uncovered Lines
Salesforce CLI command - sf project deploy validate -l RunSpecifiedTests -t PrepareMySandboxTest --coverage-formatters json --results-dir coverage -x manifest/package.xml -w 240 --verbose

Plugin command - sf apex-code-coverage transformer transform -j "coverage/coverage/coverage.json"

Sonar CLI command - sonar-scanner -Dsonar.qualitygate.wait=$SONAR_GATE -Dsonar.projectKey=$CI_PROJECT_PATH_SLUG -Dsonar.gitlab.ref_name=$SONAR_REF -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHORT_SHA -Dsonar.coverageReportPaths=coverage.xml

The XML created by the plugin:

<?xml version="1.0"?>
<coverage version="1">
	<file path="force-app/main/default/classes/PrepareMySandbox.cls">
		<lineToCover lineNumber="7" covered="false"/>
		<lineToCover lineNumber="12" covered="false"/>
	</file>
</coverage>

@xL3o - I am able to replicate your issue. Uncovered lines show when looking in the file directly, but the % is not updated in the "Code" menu.

So basically @xL3o @mshanemc ,

• With the latest version of this plugin which only prints uncovered lines, the file view in SonarQube directly shows uncovered lines, but SonarQube will not update the Coverage % in other views.
• With the beta version of this plugin that prints covered and uncovered lines, the file view in SonarQube will show both and the Coverage % will populate in other menus. However, the Salesforce CLI will print either Out-of-Range lines or not enough lines.

So until the CLI is fixed to print the accurate covered lines, I might have to look into updating this plugin to count lines in the file itself maybe??

[mcarvin8]

@xL3o - I have published a new beta build which should read all of the files to get the total # of lines. If the line is not listed as "Uncovered" in the JSON file, the plugin will assume it's "Covered". I think this is a safe workaround until the Salesforce CLI is resolved to accurately report "Covered" lines.

v1.4.1-beta.1...v1.4.1-beta.2

Please download this version, test it out, and let me know if that does the trick when you get a moment. Once you confirm this is working on your end, I'll merge the pull request and release a new official build.

sf plugins install apex-code-coverage-transformer@1.4.1-beta.2

When I tested 1.4.1-beta.2 on my end, this is working as expected:

[xL3o]

Hi @mcarvin8 ,
thanks for trying to build a workaround for now. However, we have to keep in mind, that Salesforce ignores comments and certain other lines. The new coverage report version now get accepted but is still calculating wrong coverage:
Official SF Coverage:

Report Result:
<file path="force-app/main/default/classes/SP_Access....cls"> <lineToCover lineNumber="1" covered="true"/> <lineToCover lineNumber="2" covered="true"/> <lineToCover lineNumber="3" covered="true"/> <lineToCover lineNumber="4" covered="true"/> <lineToCover lineNumber="5" covered="true"/> <lineToCover lineNumber="6" covered="true"/> <lineToCover lineNumber="7" covered="true"/> <lineToCover lineNumber="8" covered="true"/> <lineToCover lineNumber="9" covered="true"/> <lineToCover lineNumber="10" covered="true"/> <lineToCover lineNumber="11" covered="true"/> <lineToCover lineNumber="12" covered="true"/> <lineToCover lineNumber="13" covered="true"/> <lineToCover lineNumber="14" covered="true"/> <lineToCover lineNumber="15" covered="true"/> <lineToCover lineNumber="16" covered="true"/> <lineToCover lineNumber="17" covered="true"/> <lineToCover lineNumber="18" covered="true"/> <lineToCover lineNumber="19" covered="true"/> <lineToCover lineNumber="20" covered="true"/> <lineToCover lineNumber="21" covered="true"/> <lineToCover lineNumber="22" covered="true"/> <lineToCover lineNumber="23" covered="true"/> <lineToCover lineNumber="24" covered="true"/> <lineToCover lineNumber="25" covered="true"/> <lineToCover lineNumber="26" covered="true"/> <lineToCover lineNumber="27" covered="true"/> <lineToCover lineNumber="28" covered="true"/> <lineToCover lineNumber="29" covered="true"/> <lineToCover lineNumber="30" covered="true"/> <lineToCover lineNumber="31" covered="true"/> <lineToCover lineNumber="32" covered="true"/> <lineToCover lineNumber="33" covered="true"/> <lineToCover lineNumber="34" covered="true"/> <lineToCover lineNumber="35" covered="true"/> <lineToCover lineNumber="36" covered="true"/> <lineToCover lineNumber="37" covered="true"/> <lineToCover lineNumber="38" covered="true"/> <lineToCover lineNumber="39" covered="true"/> <lineToCover lineNumber="40" covered="true"/> <lineToCover lineNumber="41" covered="true"/> <lineToCover lineNumber="42" covered="true"/> <lineToCover lineNumber="43" covered="true"/> <lineToCover lineNumber="44" covered="true"/> <lineToCover lineNumber="45" covered="true"/> <lineToCover lineNumber="46" covered="true"/> <lineToCover lineNumber="47" covered="true"/> <lineToCover lineNumber="48" covered="true"/> <lineToCover lineNumber="49" covered="true"/> <lineToCover lineNumber="50" covered="true"/> <lineToCover lineNumber="51" covered="true"/> <lineToCover lineNumber="52" covered="true"/> <lineToCover lineNumber="53" covered="true"/> <lineToCover lineNumber="54" covered="true"/> <lineToCover lineNumber="55" covered="true"/> <lineToCover lineNumber="56" covered="true"/> <lineToCover lineNumber="57" covered="true"/> <lineToCover lineNumber="58" covered="true"/> <lineToCover lineNumber="59" covered="true"/> <lineToCover lineNumber="60" covered="true"/> <lineToCover lineNumber="61" covered="true"/> <lineToCover lineNumber="62" covered="true"/> <lineToCover lineNumber="63" covered="true"/> <lineToCover lineNumber="64" covered="true"/> <lineToCover lineNumber="65" covered="true"/> <lineToCover lineNumber="66" covered="true"/> <lineToCover lineNumber="67" covered="true"/> <lineToCover lineNumber="68" covered="true"/> <lineToCover lineNumber="69" covered="true"/> <lineToCover lineNumber="70" covered="false"/> <lineToCover lineNumber="71" covered="false"/> <lineToCover lineNumber="72" covered="true"/> <lineToCover lineNumber="73" covered="false"/> <lineToCover lineNumber="74" covered="false"/> <lineToCover lineNumber="75" covered="false"/> <lineToCover lineNumber="76" covered="true"/> <lineToCover lineNumber="77" covered="true"/> <lineToCover lineNumber="78" covered="true"/> <lineToCover lineNumber="79" covered="true"/> <lineToCover lineNumber="80" covered="true"/> </file>
SQ reported Coverage:

Solution Proposal:
In the json-summary file that can be generated by appending "--coverage-formatters json-summary" additionally, the ration of uncovered and covered lines is contained:

*Notice how SF themself disagree on the coverage percentage. In the deployment verbose output it is 83% and in the summary json it is 85.71%. ;)

If you could adjust the beta version in a way to just add the amount of covered lines specified in the summary and skip the other lines, we should get pretty close to the "official" code coverage percentage.

[xL3o]

Hi @mcarvin8 , thanks for trying to build a workaround for now. However, we have to keep in mind, that Salesforce ignores comments and certain other lines. The new coverage report version now get accepted but is still calculating wrong coverage: Official SF Coverage: Report Result: <file path="force-app/main/default/classes/SP_Access....cls"> <lineToCover lineNumber="1" covered="true"/> <lineToCover lineNumber="2" covered="true"/> <lineToCover lineNumber="3" covered="true"/> <lineToCover lineNumber="4" covered="true"/> <lineToCover lineNumber="5" covered="true"/> <lineToCover lineNumber="6" covered="true"/> <lineToCover lineNumber="7" covered="true"/> <lineToCover lineNumber="8" covered="true"/> <lineToCover lineNumber="9" covered="true"/> <lineToCover lineNumber="10" covered="true"/> <lineToCover lineNumber="11" covered="true"/> <lineToCover lineNumber="12" covered="true"/> <lineToCover lineNumber="13" covered="true"/> <lineToCover lineNumber="14" covered="true"/> <lineToCover lineNumber="15" covered="true"/> <lineToCover lineNumber="16" covered="true"/> <lineToCover lineNumber="17" covered="true"/> <lineToCover lineNumber="18" covered="true"/> <lineToCover lineNumber="19" covered="true"/> <lineToCover lineNumber="20" covered="true"/> <lineToCover lineNumber="21" covered="true"/> <lineToCover lineNumber="22" covered="true"/> <lineToCover lineNumber="23" covered="true"/> <lineToCover lineNumber="24" covered="true"/> <lineToCover lineNumber="25" covered="true"/> <lineToCover lineNumber="26" covered="true"/> <lineToCover lineNumber="27" covered="true"/> <lineToCover lineNumber="28" covered="true"/> <lineToCover lineNumber="29" covered="true"/> <lineToCover lineNumber="30" covered="true"/> <lineToCover lineNumber="31" covered="true"/> <lineToCover lineNumber="32" covered="true"/> <lineToCover lineNumber="33" covered="true"/> <lineToCover lineNumber="34" covered="true"/> <lineToCover lineNumber="35" covered="true"/> <lineToCover lineNumber="36" covered="true"/> <lineToCover lineNumber="37" covered="true"/> <lineToCover lineNumber="38" covered="true"/> <lineToCover lineNumber="39" covered="true"/> <lineToCover lineNumber="40" covered="true"/> <lineToCover lineNumber="41" covered="true"/> <lineToCover lineNumber="42" covered="true"/> <lineToCover lineNumber="43" covered="true"/> <lineToCover lineNumber="44" covered="true"/> <lineToCover lineNumber="45" covered="true"/> <lineToCover lineNumber="46" covered="true"/> <lineToCover lineNumber="47" covered="true"/> <lineToCover lineNumber="48" covered="true"/> <lineToCover lineNumber="49" covered="true"/> <lineToCover lineNumber="50" covered="true"/> <lineToCover lineNumber="51" covered="true"/> <lineToCover lineNumber="52" covered="true"/> <lineToCover lineNumber="53" covered="true"/> <lineToCover lineNumber="54" covered="true"/> <lineToCover lineNumber="55" covered="true"/> <lineToCover lineNumber="56" covered="true"/> <lineToCover lineNumber="57" covered="true"/> <lineToCover lineNumber="58" covered="true"/> <lineToCover lineNumber="59" covered="true"/> <lineToCover lineNumber="60" covered="true"/> <lineToCover lineNumber="61" covered="true"/> <lineToCover lineNumber="62" covered="true"/> <lineToCover lineNumber="63" covered="true"/> <lineToCover lineNumber="64" covered="true"/> <lineToCover lineNumber="65" covered="true"/> <lineToCover lineNumber="66" covered="true"/> <lineToCover lineNumber="67" covered="true"/> <lineToCover lineNumber="68" covered="true"/> <lineToCover lineNumber="69" covered="true"/> <lineToCover lineNumber="70" covered="false"/> <lineToCover lineNumber="71" covered="false"/> <lineToCover lineNumber="72" covered="true"/> <lineToCover lineNumber="73" covered="false"/> <lineToCover lineNumber="74" covered="false"/> <lineToCover lineNumber="75" covered="false"/> <lineToCover lineNumber="76" covered="true"/> <lineToCover lineNumber="77" covered="true"/> <lineToCover lineNumber="78" covered="true"/> <lineToCover lineNumber="79" covered="true"/> <lineToCover lineNumber="80" covered="true"/> </file> SQ reported Coverage:

Solution Proposal: In the json-summary file that can be generated by appending "--coverage-formatters json-summary" additionally, the ration of uncovered and covered lines is contained:

*Notice how SF themself disagree on the coverage percentage. In the deployment verbose output it is 83% and in the summary json it is 85.71%. ;)

If you could adjust the beta version in a way to just add the amount of covered lines specified in the summary and skip the other lines, we should get pretty close to the "official" code coverage percentage.

@mcarvin8 Thought about it once again, we don't need to use the json-summary even. In the normal json the covered lines are already specified at the correct amount, meaning you could just count how many "1" lines there are depicting the covered lines amount and add covered lines accordingly.

[mcarvin8]

@xL3o - Let's try this beta build out then.

sf plugins install apex-code-coverage-transformer@1.4.1-beta.3

The update I've made here is:

1. Print "Uncovered" lines in the XML first exactly as they are in the normal JSON
2. Print "Covered" lines in the XML as they are given in the normal JSON
   1. If the "Covered" line is greater than the total number of lines in the file, generate a random number that's in the file count but not already included in the "covered", "uncovered", or "random" numbers (if multiple "covered" lines are out-of-range)

I know this isn't an ideal scenario, but this should maintain the total number of Covered and Uncovered Lines in the file to ensure the Coverage % is more accurate. It just might not have accurate "covered" lines, but I think in most cases, knowing the "uncovered" lines and overall code coverage % is more important.

I'll make an update in the README (assuming this works out for you and makes sense) stating that the "uncovered" lines and the code coverage % will always be accurate, but the "covered" lines may be randomly generated due to the existing Salesforce CLI bug with how it generates "covered" lines.