contest: vm: capture code coverage

[matttbe]

Code coverage is a valuable info to get to know how much we can trust a test suite, and easily find out what needs to be improved.

Here is what it looks like for MPTCP: https://ci-results.mptcp.dev/lcov/export/index.html
(This is also visible on Coveralls.)

It is quite easy to get such info with the kernel, but it needs extra steps on the host (maybe we can switch this PR to a Draft):

• The kernel needs to be compiled with GCOV_KERNEL=y: this is handled by the modification suggested here, using vng -b --configitem. It requires virtme-ng >=1.26.

• GCOV profile also needs to be enabled, either with GCOV_PROFILE_ALL=y kconfig, but it is not recommended, or GCOV_PROFILE := y set in the Makefile's. I recommend adding a new commit to the branches that are created by NIPA, with a content that is similar to this (GCOV_PROFILE added in the header section to avoid future conflicts):

From c6d5f01a49ecd545d25ee638cb9a976045145e5a Mon Sep 17 00:00:00 2001
From: "Matthieu Baerts (NGI0)" <[email protected]>
Date: Fri, 27 Sep 2024 17:59:50 +0200
Subject: [PATCH] NIPA: enable GCOV support for net and drivers/net

This limit the GCOV support to the Networking subsystem.

Signed-off-by: Matthieu Baerts (NGI0) <[email protected]>
---
 drivers/net/Makefile | 3 +++
 net/Makefile         | 4 ++++
 2 files changed, 7 insertions(+)

diff --git a/drivers/net/Makefile b/drivers/net/Makefile
index 13743d0e83b5..a99351433b2f 100644
--- a/drivers/net/Makefile
+++ b/drivers/net/Makefile
@@ -3,6 +3,9 @@
 # Makefile for the Linux network device drivers.
 #
 
+# enable GCOV support for NIPA
+GCOV_PROFILE := y
+
 #
 # Networking Core Drivers
 #
diff --git a/net/Makefile b/net/Makefile
index 65bb8c72a35e..6c05a02b3e30 100644
--- a/net/Makefile
+++ b/net/Makefile
@@ -2,6 +2,10 @@
 #
 # Makefile for the linux networking.
 #
+
+# enable GCOV support for NIPA
+GCOV_PROFILE := y
+
 # 2 Sep 2000, Christoph Hellwig <[email protected]>
 # Rewritten to use lists instead of if-statements.
 #
-- 
2.45.2

• Before stopping the VM, the LCOV file is now captured using the lcov tool, version >= 2.0 is recommended. This tool needs to be added on the machines running the tests, the scripts are now going to use them from the VM.

• The last step is to use genhtml from the LCOV project to generate an HTML version using all the .lcov files. (It could be done per LCOV file, but that will then only show the coverage per VM, not the global one.). I guess this needs to be done in a separated script, after the end of all tests, using all the collected .lcov file, but it is not clear to me how to do that. Here is the command that needs to be executed:

BRANCH= # TODO: branch name
OUT=    # TODO: output dir
genhtml -j "$(nproc)" -t "${BRANCH}" --dark-mode \
	--include '/net/' --legend \
	--function-coverage --branch-coverage --keep-going \
	-o "${OUT}" */kernel-thr*.lcov

This GCOV support is disabled by default. It can be enabled via vm.gcov=on. I suggest keeping it off by default, and switch it to on later when everything is in place.

Note: on MPTCP side, I use a simple Docker image. If it is difficult to use LCOV >=2.0 on your side, a workaround is to add a file called lcov somewhere in your $PATH with this content (+ a symlink for genhtml):

docker run --rm --workdir "${PWD}" -v "${PWD}:${PWD}" \
       mptcp/docker-lcov-alpine:latest \
       "$(basename "${0}")" "${@}"

One last thing: it might be good to also add this support when running the KUnit tests, but it is not clear to me how to integrate with tools/testing/kunit/kunit.py.

[kuba-moo]

Hi! Very sorry for the delay, the GitHub PRs were last on my catch up list after PTO :)
The PR LGTM. We can merge it as is if you'd like. Obviously more work would be needed to actually generate the coverage. Presumably the code coverage is something we'd need to compute periodically? Or is there much value to do it series by series for the project our size?

[matttbe]

There was no hurry at all, especially because I cannot provide a full implementation :)

The PR LGTM. We can merge it as is if you'd like.

Thanks! Yes, I think it can be merged as is. The feature is disabled by default anyway. There are not a lot of modifications, hopefully I didn't break anything :)

Obviously more work would be needed to actually generate the coverage.

Generating the .lcov files should be easy:

• make sure virtme-ng >= 1.26 is used
• add the kernel patch on NIPA to add it to branches that are created by the CI
• install lcov >= 2.0 (or use a script named lcov doing the docker run I gave)
• enable GCOV support in one runner config, e.g. the one for MPTCP: vm.gcov=on

Then it's less clear how the rest would be done, but the idea is to run genhtml with the generated .lcov files: ideally from all the different jobs to see the global coverage. The generated HTML files can be published somewhere, it's all static.

Presumably the code coverage is something we'd need to compute periodically? Or is there much value to do it series by series for the project our size?

Good point: I think a good starting point is to do it either per batch of tests: once every 3 hours, or once a day. I don't have a clear view on how to collect of the different generated .lcov files. For example, they could all be moved to a dedicated directory, and consumed periodically: when starting a new batch, or once a day. It's just a matter of calling genhtml with .lcov files, and publishing the HTML files somewhere :)

[kuba-moo]

Does the GCOV generation slow things down significantly?

Is there a way to generate deltas between two reports? I think it may be more interesting to see than raw results?

We use multiple VMs per TARGET and reuse VM for multiple tests. Do you generate the coverage for all tests together or for each one individually?

[matttbe]

Does the GCOV generation slow things down significantly?

On MPTCP side, I didn't notice any big impact, but we only enable GCOV for code from net/mptcp. I didn't see any regressions cause by that in the tests, even when using the debug kernel config.

Is there a way to generate deltas between two reports? I think it may be more interesting to see than raw results?

I found lcov-diff project, but I didn't use it with MPTCP, because it is harder to get access to previous .lcov file, and at the end, it was enough to send all .lcov files to coveralls.io :)

It looks like it will show the diff between two lcov files.

On MPTCP side, the generated files are not used on a daily basis, but more to see where we are missing some code coverage. Implementing the diff is a bit harder just to retrieve files generated from a previous build, maybe not an issue for NIPA.
Also, please note that on MPTCP side, there are always small differences (see here), because the tests don't reproduce exactly the same thing each time: I guess more or less data are generated, more or less retransmissions, etc.

We can compare the general coverage, e.g. each build on MPTCP side prints this on the summary page (the last 4 lines produced by genhtml). It's a good indicator to see if there are regressions or not:

Overall coverage rate:
  lines......: 88.1% (7236 of 8214 lines)
  functions......: 95.5% (487 of 510 functions)
  branches......: 64.9% (3928 of 6050 branches)

If the HTML files generated by genhtml are kept, we can also compare data per sub-subsystems by looking at the coverage in a certain directory from two different builds/days.

We use multiple VMs per TARGET and reuse VM for multiple tests. Do you generate the coverage for all tests together or for each one individually?

I think what is more interesting is to get the global coverage: after having executed all tests. So I would say: one file per VM (what's currently done in this PR), then we combine everything by using genhtml with all .lcov files.

[kuba-moo]

Thanks for all the info! I added the coverage support to the TODO list: https://docs.google.com/spreadsheets/d/1mFnt91SKtA9ENIUfY0v2UmZSXJzmvsa4oqha4t6JVBs/edit?pli=1&gid=1985743919#gid=1985743919
:)