large chromosome simulation,multithread?

[zpcohen]

Hello, I'm attempting to generate a neutral vcf for 40 individuals on a 50Mb chromosome using the command line as msp ancestry and msp mutation and was wondering if I can multi-thread this process on a slurm cluster or a high performance node with 96 cores. As of now this is only using 1 CPU and 0.5% memory, thank you.

[molpopgen]

Hi @cistarsa --

Welcome, and thanks for the question. These simulation algorithms do not support parallel computing. Would you mind sharing your command line for msp ancestry? The way that recombination rate interacts with some other parameters can have a big impact on performance.

[zpcohen]

Ah sure, yeah so even on a high performance node with many cpus it only uses 1 core at a time? Is there a way to have it take full advantage of the system it's on? Thanks and here's what I've been using

msp ancestry 25 -N 40071 -L 5774900 --recombination-rate 2.95e-6 --ploidy 2 > LI_25.trees

[molpopgen]

Ah sure, yeah so even on a high performance node with many cpus it only uses 1 core at a time? Is there a way to have it take full advantage of the system it's on? Thanks and here's what I've been using

msp ancestry 25 -N 40071 -L 5774900 --recombination-rate 2.95e-6 --ploidy 2 > LI_25.trees

No, parallel methods can only be used if the algorithms can be broken into approximately independent bits. For coalescent models, it is not clear that doing so is possible.

[molpopgen]

One question: where does that recombination rate come from?:. It is an order or magnitude or two higher than typical values used for, e.g., humans.

[zpcohen]

Gotcha, thanks for the quick reply and yeah, I see the same single-threading with Slim##, too...true typical rate for some model systems but this was estimated using pyrho for insect, which is on par.

…

On Sat, Apr 2, 2022 at 6:45 PM Kevin R. Thornton ***@***.***> wrote: One question: where does that recombination rate come from?:. It is an order or magnitude or two higher than typical values used for, e.g., humans. — Reply to this email directly, view it on GitHub <#2024 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE7FXO4LRWJSX2YJADGE65DVDDL2BANCNFSM5SKAHXLQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jeromekelleher]

Hi @cistarsa 👋 As @molpopgen says it's not trivial to break up these simulation algorithms into parallel chunks. The main reason that we haven't put any effort into this is that we're nearly always running many replicates when doing such simulations, and these parallelise naturally.