testthat + parallel + Valgrind best practices?

[krivit]

CRAN wants maintainers to run Valgrind on their packages, but when parallel testing is enabled, R CMD check --use-valgrind runs testthat.R and the examples under Valgrind, but not the actual tests.

What are the best practices for checking a package using testthat under Valgrind?

As far as I can tell, the least intrusive way to do so is to set the TESTTHAT_PARALLEL=FALSE environment variable, so all the tests are run in the same process and Valgrind output is produced, though I think the test that generated the output is not identified. (This variable is not documented anywhere, but perhaps it ought to be?)

However this foregoes the benefits of parallelisation, which would be particularly helpful given the speed hit of Valgrind. Is there a better way?

[hadley]

I don't know of any better solution than temporarily suppressing parallelism sorry 😞

[krivit]

Thanks @hadley for looking into it!

I think I have a solution, and I am leaving this here in case anyone else has a similar problem. The following runs the worker processes under Valgrind as well (via --trace-children=yes) and saves the Valgrind output into a bunch of files identified by their PID and sequence number:

env VALGRIND_OPTS="--trace-children=yes --log-file=/tmp/$PKG.valgrind.%p.%n.log"  R CMD check --use-valgrind $PKG

A log file produced for every subprocess is spawned or forked (including to execute shell commands), but you can zero in on the relevant files via:

grep -l -E '^==[0-9]+== Command: /usr/lib/R/bin/exec/R' $PKG.valgrind.*.*.log

(assuming your R executable installation path is as above). Unfortunately, I don't think there is a good way to map the messages back to the tests.

P.S.: May I suggest documenting TESTTHAT_PARALLEL? I only found it looking through the source code.