Info: differences between tarpaulin and cargo-llvm-cov specifically with --engine llvm

[helio-frota]

Caution

Apparently is not feasible to use one tool instead of the other llvm based coverage

Note

We are not using --engine llvm see here
I enabled this option to share the difference in this post/comment 👍

Commands:

cargo tarpaulin --timeout 300 --out xml --skip-clean --features _test-s3 --engine llvm
cargo llvm-cov --features _test-s3 --no-clean --cobertura --output-path cobertura.xml

[i386x]

@helio-frota @ctron I like the setup used in https://github.com/oknozor/speculoos. They are using cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info and have almost 100% coverage with very few false negatives. I am curious whether @jcrossley3 false negative (https://github.com/trustification/trustify/pull/1749/files#diff-b274c70a92550f8d3cd8280bd39b2b5b4a0907454a3af724e7b04a7f1f9a3a9eR140) will also appear under their configuration (not tried yet). If not, maybe we can switch from tarpaulin to llvm-cov?

[helio-frota]

maybe we can switch from tarpaulin to llvm-cov ?

🤷

I'm not the best person to give an opinion...until recently, I believed that, in terms of coverage, the results were approximately the same. Later, I realized that's not exactly the case. If I had to vote, I select llvm-cov, mainly because I find this part a bit concerning:

https://github.com/xd009642/tarpaulin?tab=readme-ov-file#tarpaulin

On Linux, Tarpaulin's default tracing backend is still Ptrace and will only work on x86_64 processors. 
This can be changed to the llvm coverage instrumentation with --engine llvm. 
For Mac and Windows, this is the default collection method.

So then I enabled --engine llvm and then I shared here.

And.. then my brain reads this ^ paragraph like: if linux uses Ptrace then running coverage on windows and mac we end up with different % coverage ? 🤔

[helio-frota]

Running on CI we can see these differences (screenshots):
#1833 (comment)

[helio-frota]

We have a PR #1860 👍