Panaroo killed due to memory usage

[mikhailova-es]

HI,
I am trying to build orthologous gene clusters for ~4000 E. coli genomes. I ran the following command on an HPC cluster node with 126 GB RAM and 6 threads:
panaroo -i input_panaroo_file.txt --threads 6 -o results --clean-mode strict --remove-invalid-genes

The job was killed by the system after 2.5 days of runtime. According to sacct, the memory usage reached the node limit:

JobID            MaxRSS     ReqMem    Elapsed
34665.batch   127856448K   126000M   2-12:17:49

The last lines in the output and log file:

running cmd: cd-hit -T 6 -i results/combined_protein_CDS.fasta -o results/combined_protein_cdhit_out.txt -c 0.98 -s 0.98 -aL 0.0 -AL 99999999 -aS 0.0 -AS 99 999999 -M 0 -d 999 -g 1 -n 2 
generating initial network... 
Processing paralogs... 
collapse mistranslations... 
Processing depth: 1 ... 
Processing depth: 2 ... 
Processing depth: 3 ... 
collapse gene families... 
Processing depth: 1 ... 
Processing depth: 2 ... 
Processing depth: 3 ... 
trimming contig ends... 
refinding genes... 
Number of searches to perform: 174718261 
Searching... 
1290it [5:35:30,14.43s/it]/var/spool/slurm/d/job34665/slurm_script: line 12: 271468 Killed 

I am trying to figure out is it possible to complete this analysis with 126 GB. Maybe reducing number of threads or using a different --clean-mode will help? If it possible, how long do you think this analysis could take? Would appreciate any advice.
Thank you!

[nzmacalasdair]

Hello!

Panaroo is trying to perform an enormous number of searches for refound genes here: 174718261, which will take way longer than the runtime limit on most HPCs, and require quite a bit of memory. Are there potentially some very low-quality/highly fragmented assemblies in this run? Removing those might help reduce that, panaroo-qc is an easy way to quickly check all the input if you don't have those statistics readily available. If not, it may just be that the size and diversity of E. coli genomes means that ~4000 is too many (especially if they are sampled from across the diversity of E. coli, and not one ST or lineage).

I would definitely keep the --clean-mode as strict unless you have some reason to think that there are some very rare genes which you know are not erroneous annotations and are of interest. Less stringent clean modes result in a more structurally complex pangenome graph, leading to longer runtimes and higher memory usage.

If the ~4000 genomes are very diverse, the best way to run them would probably be to split them up into groups of 500-1000, ideally based on relatedness, eg by clustering using BAPS or poppunk, and run those separately with panaroo using something like 8-12 threads and 60-80gb of RAM (you might be able to get away with less, maybe as little as 40 if you have severe RAM restrictions). You can then combine the panaroo output using panaroo-merge to get orthologous gene clusters for the entire collection. If you can run the 4-8 panaroo runs in parallel, you should be able to get results within a few days.

Let us know if you keep having problems!