Inconsistent/wrong line coverage

[AndrzejKurek]

Hi there, I'm investigating line coverage for a new feature that I'm introducing to Mbed TLS, and I'm seeing inconsistencies when it comes to line coverage (branch information seems fine):

line 1857 is hit, even though nothing executed it.

Reproduction steps:
Starting from here:

make CFLAGS='--coverage -g3 -O0' LDFLAGS='--coverage' -C tests test_suite_x509parse
(cd tests && ./test_suite_x509parse)
make lcov #runs [scripts/lcov.sh](https://github.com/Mbed-TLS/mbedtls/blob/development/scripts/lcov.sh)
firefox Coverage/index.html

Check library/x509_crt.c coverage, line 1856 & 1857. The insides of the condition should not be hit.
I reduced the number of tests run to one that highlights the problem.
If you're not able to run this test or you'd like me to produce a possibly minimal test case - let me know, I'll try harder :)

Tool versions:
lcov 1.14 and 1.16 tested;
gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

[henry2cox]

lcov is merely presenting coverage data generated by gcov.
Could you take a look at the gcov output you get from your testcase, and check whether gcov claims that your line 1857 is executed or not?
If gcov says 'not' and lcov says "yes" - then there is an lcov bug. Otherwise, gcc/gcov bug.

Another way that this can happen is if your source code version is not aligned with your build - so the instrumented executable refers to different line numbers than correspond to your current source code.

The other thing you probably want to do is to try the current lcov TOT, as there have been some extensive changes and quite a few fixes since 1.16.

Hope this helps
Henry

[henry2cox]

The other thing to try is to remove the "-O0" flag and see if that fixes your coverage report.
This would not be the first time that optimization caused a compiler to lose track of some information.

[xaizek]

-O0 doesn't enable any optimizations, it disables them:

Most optimizations are completely disabled at -O0 or if an -O level is not set on the command line, even if
individual optimization flags are specified.
[...]
-O0 Reduce compilation time and make debugging produce the expected results.  This is the default.

[henry2cox]

Bummer.
That flag change likely won't help, then.

Could also try with LLVM and see if that gives a different result. Different gcc version might also be interesting.
(As an aside: it can be quite interesting to compile and run your tests with two different compilers and/or versions - then look at a differential coverage report. I might suggest that here, too.)

[AndrzejKurek]

-Og doesn't change anything either. Here's the gcov output with -a -c flags:

        3: 1856:            if (ret != 0) {
       1*: 1857:                return ret;
    %%%%%: 1857-block  0
        1: 1857-block  1
        -: 1858:            }

[henry2cox]

the gcov man page says:

Executed basic blocks
having a statement with zero execution_count end with * character
and are colored with magenta color with the -k option.

.. which appears to be what we see here.
I think we see the statement which has a zero execution count - but I don't think I understand why it is instrumented this way, or what I could do in 'geninfo' to better handle the case.
Maybe the answer is to look for following "zero count' blocks - and then mark the corresponding statement as not executed?

Could you run the same thing with "gcov -i" (json format output) - and show me what that looks like as well?

Thanks
Henry

[AndrzejKurek]

Could you run the same thing with "gcov -i" (json format output) - and show me what that looks like as well?

Sure, I'll upload the whole file though, since I can see that relevant data might be scattered.
The function this issue concerns is mbedtls_x509_parse_subject_alt_name, lines 1856 - 1858.
Grepping for 1857: {"branches": [], "count": 1, "line_number": 1857, "unexecuted_block": true, "function_name": "mbedtls_x509_parse_subject_alt_name"},
x509_crt_gcov.json.txt

[henry2cox]

Could you take a quick look at some of the other lines which are marked 'hit' but which have unexecuted blocks.
If I blindly tell lcov that all of them should be marked 'not hit' - then I am not quite sure whether the result will be better than today, or worse. We might need a better fix than the simplistic workaround that first came to mind.

I replaced '{' with '\n{' in the JSON file, then ran

grep '"unexecuted_block": true' | grep -v '"count": 0'

to find a bunch of lines in your file - say, lline 995, 1001, 1331, 1372, ... and so on (23 lines in total)

[AndrzejKurek]

If I blindly tell lcov that all of them should be marked 'not hit' - then I am not quite sure whether the result will be better than today, or worse.

It seems that for complex statements it's probably correct (there's a part of the condition on a given line that's not executed, but it's hard to parse this .json data manually), for example:

• while (crt->version != 0 && crt->next != NULL) { <- crt->version equals 0, so the second part is not checked.

There are however also instances that are a bit puzzling to me:

• line 995:
  if (cb != NULL) { <- this if is called but evaluates to false. I would expect that this doesn't show up as an unexecuted_block in the if line, but maybe you can correct me;
• line 1457:
  if (buflen != 0 && buf[buflen - 1] == '\0' && <- buflen is not equal to 0, so the second part of the if is evaluated, but found false. I would also expect it to not show up as an unexecuted_block.

I'm not sure how this intersects with other sources of data, so I'll leave the interpretation up to you.

[henry2cox]

Sorry for all the requests...but your testcase seems to easily generate some conditions that my simple cases do not.
(I have not checked results from any of the actual applications.)

Could you run gcov again, this time adding flags "-c -a -i" (turn on branch data).
Might also be good to try with "-c -i" (without "all blocks").
I'm most interested to see if we find lines which are marked with "unexecuted block" which do not contain "branches".
If we did not ask for branches...then we appear to see a lot of them :-) If we ask...then that will be interesting.

Ultimately, once we have resolved this issue, it will be interesting to look at a differential coverage report comparing 'before' and 'after' - to see what the differences are. This could help us figure out if there are more issues we have to fix.

Thanks

[AndrzejKurek]

Each of these calls resulted in the same file (same checksum and size). Commands ran:

make CFLAGS='--coverage -g3 -O0' LDFLAGS='--coverage' -C tests test_suite_x509parse
(cd tests && ./test_suite_x509parse)
cd ./library/
gcov -c -a -i ./x509_crt.c
mv ./x509_crt.gcov.json.gz ./x509_crt.gcov-cai.json.gz
gcov -c -i ./x509_crt.c
mv ./x509_crt.gcov.json.gz ./x509_crt.gcov-ci.json.gz 
gcov -i ./x509_crt.c
mv ./x509_crt.gcov.json.gz ./x509_crt.gcov-i.json.gz 

Let me know if I should run them in any other way.
x509_crt_gcov-cai.json.txt

[henry2cox]

Yeah...can you try yet again, with "-b -x -i".
(Misread the code...)

[AndrzejKurek]

As an update in the meantime - running this with clang yielded better results:

 make CC=clang-14 CFLAGS='--coverage -g3 -O0' LDFLAGS='--coverage' -C tests test_suite_x509parse
(cd tests && ./test_suite_x509parse)
make lcov GCOV_TOOL="llvm-cov-14 gcov"

Local patch added:

diff --git a/scripts/gcov_tool.sh b/scripts/gcov_tool.sh
new file mode 100755
index 000000000..7c663d8da
--- /dev/null
+++ b/scripts/gcov_tool.sh
@@ -0,0 +1,2 @@
+#!/bin/sh
+exec ${GCOV_TOOL:-gcov} "$@"
diff --git a/scripts/lcov.sh b/scripts/lcov.sh
index 8d141eedf..02e9c25ec 100755
--- a/scripts/lcov.sh
+++ b/scripts/lcov.sh
@@ -46,10 +46,10 @@ set -eu
 lcov_library_report () {
     rm -rf Coverage
     mkdir Coverage Coverage/tmp
-    lcov --capture --initial --directory library -o Coverage/tmp/files.info
-    lcov --rc lcov_branch_coverage=1 --capture --directory library -o Coverage/tmp/tests.info
-    lcov --rc lcov_branch_coverage=1 --add-tracefile Coverage/tmp/files.info --add-tracefile Coverage/tmp/tests.info -o Coverage/tmp/all.info
-    lcov --rc lcov_branch_coverage=1 --remove Coverage/tmp/all.info -o Coverage/tmp/final.info '*.h'
+    lcov --gcov-tool "$PWD/scripts/gcov_tool.sh" --capture --initial --directory library -o Coverage/tmp/files.info
+    lcov --gcov-tool "$PWD/scripts/gcov_tool.sh" --rc lcov_branch_coverage=1 --capture --directory library -o Coverage/tmp/tests.info
+    lcov --gcov-tool "$PWD/scripts/gcov_tool.sh" --rc lcov_branch_coverage=1 --add-tracefile Coverage/tmp/files.info --add-tracefile Coverage/tmp/tests.info -o Coverage/tmp/all.info
+    lcov --gcov-tool "$PWD/scripts/gcov_tool.sh" --rc lcov_branch_coverage=1 --remove Coverage/tmp/all.info -o Coverage/tmp/final.info '*.h'
     gendesc tests/Descriptions.txt -o Coverage/tmp/descriptions
     genhtml --title "mbed TLS" --description-file Coverage/tmp/descriptions --keep-descriptions --legend --branch-coverage -o Coverage Coverage/tmp/final.info
     rm -f Coverage/tmp/*.info Coverage/tmp/descriptions

[AndrzejKurek]

make CFLAGS='--coverage -g3 -O0' LDFLAGS='--coverage' -C tests test_suite_x509parse
(cd tests && ./test_suite_x509parse)
cd library
gcov -b -x -i ./x509_crt.c

x509_crt##78325959f5aaa72897d25968fdf44ccd.gcov.json.txt