Multithreading is not working

[mr-eyes]

In manysearch and probably the rest of the commands, multi-threading is not working. Even when I hard-code the number of cores, it only works on 2. I noticed that with htop

[mr-eyes]

Maybe we can define it globally in each function

rayon::ThreadPoolBuilder::new().num_threads(threads).build_global().unwrap();

[ctb]

The benchmarks suggest otherwise? Unless it's broken in the last few days, which is certainly possible... but seems unlikely!

• benchmark - searching against Genbank cover at 100k #7
• benchmarks against GTDB 65k #10
• benchmarks - SRR606249 x GTDB RS214 #12
• fastmultigather benchmarks: swine x gtdb reps #22

e.g. in the most recent (#22) we see:

        Percent of CPU this job got: 3189%

with 32 threads.

So! mystery! What system are you running on?

[mr-eyes]

I am running on farm. How did you set the 32 cores while num_threads does not seem to be passed to rust as per #53 ? Am I getting something wrong?

[ctb]

can you try this:

export RAYON_THREADS=32

[ctb]

ref #31

[mr-eyes]

It still does not work.
Maybe it behaves like that because I have a small number of target sigs? I have 1K query vs. 15 targets.

[ctb]

yes, the 15 targets is likely to be a problem!

here's something that runs on farm with the manysearch code, and definitely does do multithreading...

background repo: https://github.com/dib-lab/2022-branchwater-benchmarking
integrating with pyo3_branchwater: dib-lab/2022-branchwater-benchmarking#8

[ctb]

(how does #57 fix this issue? curious what you figured out ;)

[ctb]

FYI large scale benchmarks confirm that something is fishy with manysearch and threading -

dib-lab/2022-branchwater-benchmarking#9

[mr-eyes]

(how does #57 fix this issue? curious what you figured out ;)

Here're some details

• I started an srun job with 2 cpus, but squeue showed allocation of 34 cpus due to mem<->cpus proportionality.
• I made an initial code modification to set the number of threads, but it was after current_num_threads so it showed working in parallel while it's really working on only 2 cpus not 34. That's why I was confused.
• The PR does not directly fix this issue but ensures I get the info about the actual number of cores the program is running on. So, eventually fixes this.

[mr-eyes]

yes, the 15 targets is likely to be a problem!

The number of cores significantly impacted the run time, even when the number of targets was small.

[ctb]

I think we can close this now due to #57 and follow-on benchmarking and tests!