Question on running princess on single machine

[RenzoTale88]

Hello,
I'm trying to run princess on a single node using a total of 24 cores. When I run the task using the command:

princess all -f reference.fa -j 24 -a ngmlr -e -u -r ccs -d ./ -g princess.log -s all.reads.fq.gz

The workflow runs fine, however I noticed that NGMLR is actually using only 5 cores at the time, making it running inefficiently. Is there a way to specify the number of cores for each single substage?
Otherwise, is it possible to break down the workflow into substages to optimize the number of resources by running them one after another? For example, would something like this lead to use 24 cores for the alignment, the variant calling and the phasing?

princess align -j 24 [OPTIONS HERE]
princess variant -j 24 [OPTIONS HERE]
princess phase -j 24 [OPTIONS HERE]

Thank you in advance,
Andrea

[MeHelmy]

Thank you @RenzoTale88 ,

Sure, you can control the number of threads; for aligning I refer you to this section:

# Aligner
#########
aligner: "minimap" # minimap or ngmlr
aligner_threads: 5
#########

In config.yaml, please change from 5 threads to what suits you.

While you can control most of the parameters from the command line, there is advanced parameters control from config.yaml file https://github.com/MeHelmy/princess/blob/master/config.yaml. But, regarding your question changing aligner_threads: 5 is enough.

Note: -j 24 refers to the number of jobs running on a cluster, not threads.

Best,
Medhat

[RenzoTale88]

Great thanks @MeHelmy. One more thing: if I limit -j 1, will I run one task at the time locally?
Thank you again!

[RenzoTale88]

@MeHelmy one minor note: I've tried to run the workflow on a single node of our cluster, submitted within a nextflow workflow.
The process starts without issues, however it always fails in running minimap due to a \t not being recognized in the read group flag. Chaing line 45 from:

minimap2  {params.x} -Y -R '@RG\tID:hg2' "{params.reference}"  "{input.datain}" {params.h} "{params.md}" -t "{threads}" | samtools sort -@ {threads} - > "{output.dataout}"   

to

minimap2  {params.x} -Y -R '@RG\\tID:hg2' "{params.reference}"  "{input.datain}" {params.h} "{params.md}" -t "{threads}" | samtools sort -@ {threads} - > "{output.dataout}"   

Fixed the issue. Hope this is of any help,
Andrea

[MeHelmy]

Great thanks @MeHelmy. One more thing: if I limit -j 1, will I run one task at the time locally? Thank you again!

Yes, the J option will make sure that no more than n tasks run at the same time, in your case you are running the jobs consecutively, and you are not making use of the cluster, so it will be okay.

[MeHelmy]

@MeHelmy one minor note: I've tried to run the workflow on a single node of our cluster, submitted within a nextflow workflow. The process starts without issues, however it always fails in running minimap due to a \t not being recognized in the read group flag. Chaing line 45 from:

minimap2  {params.x} -Y -R '@RG\tID:hg2' "{params.reference}"  "{input.datain}" {params.h} "{params.md}" -t "{threads}" | samtools sort -@ {threads} - > "{output.dataout}"   

to

minimap2  {params.x} -Y -R '@RG\\tID:hg2' "{params.reference}"  "{input.datain}" {params.h} "{params.md}" -t "{threads}" | samtools sort -@ {threads} - > "{output.dataout}"   

Fixed the issue. Hope this is of any help, Andrea

Unfortunately, I did not try to submit it using nextflow, there are some subtle changes that need to be changed to make it compatible with both. But, make sure that the read-group (RG) has been added correctly in your file.

Thank you for pointing me to that,
Medhat

[RenzoTale88]

@MeHelmy thank you for clarifying about the -j flag.
Just checked and the read group is correctly assigned. It is quite interesting, because the nextflow job simply call princess inside a bash script, and run every dependency within a debian container to make everything standardised. So I'm quite unsure why it shouldn't like the tab-spacing in the script. Nevertheless, escaping the \t made the trick, previously said. Thanks again for the help!

[MeHelmy]

Sure! anytime,
" It is quite interesting, because the nextflow job simply calls princess inside a bash script" -- Good point, I will remove it and test it on my machine, and see if it works too.