non-deterministic interleaved output to stdout with multiple threads

[ntm]

Dear devs,

writing interleaved output to stdout with multiple threads is non-deterministic again.
For me this is identical to issue #552 , but that issue had been resolved with release 0.24.3, which fixed it (and indeed 0.24.3 doesn't have this problem).

I tested every release between 0.24.3 and 1.0.1, using commands such as (example below for 0.26.0):

fastp.0.26.0 --stdout --in1 test_1.fq.gz --in2 test_2.fq.gz --detect_adapter_for_pe --json /dev/null --html /dev/null --thread 1 > test_T1_v026.fq
fastp.0.26.0 --stdout --in1 test_1.fq.gz --in2 test_2.fq.gz --detect_adapter_for_pe --json /dev/null --html /dev/null --thread 24 > test_T24_v026.fq

I then checked the MD5 sums of the produced .fq files.
Results are identical for 1 and 24 threads with version 0.24.3, but they differ for every subsequent version starting at 0.25.0.
So, the bug must come from one of the 4 commits between 0.24.3 and 0.25.0. Maybe commit 6aa8f09 ?

Kind regards,
Nicolas

[ntm]

Hello,

I can confirm that commit 6aa8f09 is responsible for the regression:
I built fastp from the tip of master branch (455a859), and built the same after git revert 6aa8f09 (required resolving a small conflict in src/peprocessor.cpp, I kept the lines from the tip of master).
I then ran each of the obtained binaries, same as above, with --thread 1 or 24.
The resulting FASTQs are identical for latest single thread, latest_without_6aa8f09 single thread, and latest_without_6aa8f09 with 24 threads; but latest with 24 threads differs.

[ntm]

PR #655 fixes the issue in my tests

[sfchen]

Thanks, merged.