Accept *.profdata files as input

[RenjiSann]

Context

We are using grcov to compute coverage in CI in the uutils/coreutils and we have a scale problem. Our testsuite produces 1 .profraw file (5 MB) for each invocation of the tested binary, which is thousands of times among the whole testsuite.
This creates over 80 GB of trace files, which exceeds the storage capacity of our CI system.

Therefore, I was thinking of splitting the testsuite execution into batches (running the testsuite per-util, there is ~100) and to call grcov multiple to create several lcov files.
The issue here is that grcov invocations are quite long, and are "duplicating" work, so that's not a viable option either.

Ideal workflow

Firstly, I'd like to aggregate the batches of .profraw files manually into .profdata using llvm-profdata, mainly because at this point I only need to aggregate the trace files I can delay the creation report after, and I will spare a lot of time.
Then, ideally, I would use grcov and aggregate the .profdata using grcov and create an lcov report from them.

Would it be possible for grcov to accept profdata files as inputs ? Is there a particular concern that would prevent a trivial implementation of this feature ?

I'm willing to make a contribution if necessary

[Lavina-korani]

To address the storage and performance challenges you're facing with processing numerous .profraw files using grcov, transitioning to a workflow that utilizes llvm-profdata to merge these files into .profdata format before generating coverage reports can be highly effective. This approach reduces the number of files grcov needs to process, thereby optimizing storage usage and processing time.

Merge .profraw Files into a Single .profdata File:

Use llvm-profdata to combine multiple .profraw files into one .profdata file. This step is crucial for efficient coverage report generation.

CODE

llvm-profdata merge -sparse *.profraw -o merged.profdata
The -sparse option is recommended for Rust projects, as it helps in handling sparse profile data efficiently.
CLANG.LLVM.ORG
Generate the Coverage Report with grcov:

After merging the .profraw files into a .profdata file, you can use grcov to generate the desired coverage report. However, as of the latest updates, grcov does not natively support .profdata files directly. It primarily works with .profraw files.

To work around this limitation, you can instruct grcov to process the .profdata file by specifying the --llvm-path option to point to your LLVM tools, ensuring that grcov utilizes the appropriate version of llvm-profdata.
GITHUB.COM

CODE

grcov . --binary-path ./target/debug/deps/ -s . --llvm-path /path/to/llvm/bin/ -t lcov --branch --ignore-not-existing -o ./target/debug/coverage/
Replace /path/to/llvm/bin/ with the actual path to your LLVM binaries.
The --llvm-path option directs grcov to use the specified LLVM tools, which is essential for processing .profdata files.

[Lavina-korani]

CAN I WORK ON THIS ISSUE

[RenjiSann]

CAN I WORK ON THIS ISSUE

Considering the fact that you seem to have no prior experience with the project, and that you just copy pasted the answer of chatGPT which does not answers anything, I'd rather do it myself, thanks

[marco-c]

Context

We are using grcov to compute coverage in CI in the uutils/coreutils and we have a scale problem. Our testsuite produces 1 .profraw file (5 MB) for each invocation of the tested binary, which is thousands of times among the whole testsuite. This creates over 80 GB of trace files, which exceeds the storage capacity of our CI system.

Therefore, I was thinking of splitting the testsuite execution into batches (running the testsuite per-util, there is ~100) and to call grcov multiple to create several lcov files. The issue here is that grcov invocations are quite long, and are "duplicating" work, so that's not a viable option either.

I'm curious why grcov invocations are slow, they shouldn't be. Any chance you could take a profile?
Could it be #1013? You could test it by doing #1014.

Ideal workflow

Firstly, I'd like to aggregate the batches of .profraw files manually into .profdata using llvm-profdata, mainly because at this point I only need to aggregate the trace files I can delay the creation report after, and I will spare a lot of time. Then, ideally, I would use grcov and aggregate the .profdata using grcov and create an lcov report from them.

Would it be possible for grcov to accept profdata files as inputs ? Is there a particular concern that would prevent a trivial implementation of this feature ?

I'm willing to make a contribution if necessary

I don't have any concerns, and I'd be happy to take a patch!

[RenjiSann]

The exact command I do:

grcov  \                                          
        coverage-traces \
        --binary-path "./target/debug/coreutils" \ # I give the path to the binary directly. maybe that's not right
        --output-types lcov \
        --output-path "coverage-report/ls.lcov.info"

The strace -f -c output

% time     seconds  usecs/call     calls    errors syscall                                                                                                                                                                                                                                  
------ ----------- ----------- --------- --------- ------------------                                                                                                                                                                                                                       
 50.86  109.106974         910    119786     31863 futex                                                                                                                                                                                                                                    
 14.14   30.344407        9500      3194        87 poll                                                                                                                                                                                                                                     
  8.75   18.766647       31277       600           wait4                                                                                                                                                                                                                                    
  6.98   14.980099       16552       905           clone3                                                                                                                                                                                                                                   
  5.30   11.367350          71    157902           mprotect                                                                                                                                                                                                                                 
  4.75   10.193271         140     72636      4979 read                                                                                                                                                                                                                                     
  3.78    8.110466       36207       224       137 restart_syscall                                                                                                                                                                                                                          
  1.00    2.137800          94     22634     20693 readlink                                                                                                                                                                                                                                 
  0.73    1.573044          52     29825           close                                                                                                                                                                                                                                    
  0.58    1.238516         294      4212           prlimit64                                                                                                                                                                                                                                
  0.50    1.069256         144      7383           munmap                                                                                                                                                                                                                                   
  0.49    1.061210          28     37058      9641 openat                                                                                                                                                                                                                                   
  0.41    0.869198         474      1830        33 ioctl                                                                                                                                                                                                                                    
  0.34    0.732884          23     31189           mmap                                                                                                                                                                                                                                     
  0.27    0.584104         484      1206           pipe2                                                                                                                                                                                                                                    
  0.21    0.443245          30     14320           rt_sigprocmask                                                                                                                                                                                                                           
  0.18    0.393585         536       733           madvise                                                                                                                                                                                                                                  
  0.12    0.264582           4     55723           write                                                                                                                                                                                                                                    
  0.09    0.186043          65      2856           sched_yield                                                                                                                                                                                                                              
  0.08    0.162507          15     10816           fstat                                                                                                                                                                                                                                    
  0.08    0.161374           9     16232        10 statx                                                                                                                                                                                                                                    
  0.05    0.110109           7     14256           fcntl                                                                                                                                                                                                                                    
  0.04    0.085293           7     10700     10104 access                                                                                                                                                                                                                                   
  0.04    0.076881           0     82299           rt_sigaction                                                                                                                                                                                                                             
  0.03    0.066158          13      4825           getdents64                                                                                                                                                                                                                               
  0.03    0.062561           8      7527           brk                                                                                                                                                                                                                                      
  0.03    0.059181          44      1338           symlink                                                                                                                                                                                                                                  
  0.03    0.056239          18      3123           stat                                                                                                                                                                                                                                     
  0.03    0.055261          40      1351           unlinkat                                                                                                                                                                                                                                 
  0.02    0.050479          82       613         8 execve                                                                                                                                                                                                                                   
  0.02    0.041389           5      7247      3004 newfstatat                                                                                                                                                                                                                               
  0.01    0.026726         197       135           mremap                                                                                                                                                                                                                                   
  0.01    0.019633          15      1291           sigaltstack                                                                                                                                                                                                                              
  0.01    0.010999          15       710       700 lseek                                                                                                                                                                                                                                    
  0.01    0.010928          74       147           sched_getaffinity                                                                                                                                                                                                                        
  0.00    0.010637          11       910           set_robust_list                                                                                                                                                                                                                          
  0.00    0.009653          33       292           prctl                                                                                                                                                                                                                                    
  0.00    0.007517           4      1573           pread64                                                                                                                                                                                                                                  
  0.00    0.006882           7       910           rseq                                                                                                                                                                                                                                     
  0.00    0.006369          10       600           readlinkat                                                                                                                                                                                                                               
  0.00    0.005196           8       610           getrandom                                                                                                                                                                                                                                
  0.00    0.002217         184        12           mkdir
  0.00    0.002088           1      1799           dup2
  0.00    0.002019           3       605           arch_prctl
  0.00    0.001777           2       605           set_tid_address
  0.00    0.000329          82         4           faccessat2
  0.00    0.000308          10        29           getcwd
  0.00    0.000171          21         8           open
  0.00    0.000140          11        12           uname
  0.00    0.000062          15         4           sysinfo
  0.00    0.000016           8         2           getpid
------ ----------- ----------- --------- --------- ------------------
100.00  214.533780         291    734801     81259 total

I can't manage to create a meaningful flamegraph however :/

[marco-c]

@RenjiSann what files do you have under coverage-traces?

Could you do a perf report too? Might be more insightful.

[RenjiSann]

what files do you have under coverage-traces?

Only .profrawfiles at the time of the test , around 1500.

Could you do a perf report too? Might be more insightful.

I am no expert with perf, but I couldn't produce a useful perf report, I don't really know why :/
Will try again when I have the time

[sylvestre]

https://github.com/mstange/samply is probably better

[RenjiSann]

https://github.com/mstange/samply is probably better

Still nothing, I wonder if I am doing something wrong.

../grcov/target/debug/grcov \
    "coverage/traces" \
    --binary-path "./target/debug/" \
    --output-types lcov \
    --output-path "coverage/report/ALL.lcov.info" \
    --keep-only `pwd`'/src/*'

This gives me coherent results, but when I prefix it with samply record, it just won't load the profile and load forever.
I will keep trying to provide a perf file to understand what takes so much time here.

Anyway #1287 is ready for review. The patch itself is almost trivial, and I have added some tests which account for most of the diff.
It would be nice to merge and release this quickly, as we'll depend on on it on uutils/coreutils.

Thanks you !