MCL step hangs without log output for 24 hours on 32 cores with 200+ genomes

[qianhg0706]

Hi，I am currently running pandagma to generate pan-gene sets for over 200 genomes. However, I encountered an issue during the MCL step. The job has been running for 24 hours without any log output, and I am unsure whether the process is still running or if it has become stuck.
Here are the details of my run:
• Command: mcl 05_filtered_pairs.tsv -I 1.6 -te 32 --abc -o tmp.syn_pan.clust.tsv
• System: [ HPC cluster with 32 cores]
• Input size: Over 200 genomes
• Last log output:

[mcl] new tab created
[mcl] pid 3461449
 ite   chaos  time hom(avg,lo,hi) m-ie m-ex i-ex fmv
  ...
 40     0.00 500.37 1.00/1.00/1.00 1.00 1.00 0.54   0
[mclProcess] ====== Inflation bumped to 1.6016 ======

I would like to know if this is expected behavior for such a large dataset, or if there might be an issue with the process. If additional computing resources are needed, please let me know the recommended specifications.
Thank you for your help!
Best regards,
Qian

[StevenCannon-USDA]

Hi Qian,
I have not tested pandagma with that many genomes, so I don't know how the run-time will be affected for particular steps. In my experience with smaller numbers of genomes, the mcl step has typically been relatively fast -- generally minutes rather than hours.

The programs are designed to build clusters from a designated set of "main" annotation, and then to add "extra" annotation sets onto those clusters. From the manuscript:
"The ability to add extra annotations by homology allows inclusion of annotation sets that may come from fragmented assemblies, or from more distant taxa. This option also allows for arbitrary numbers of annotations to be added to an initial set with relatively little computational cost. Specifically, the computational complexity of calculating primary clusters with n “main” annotations is O(n2), while the cost of adding “extra” annotations is linear."

In the configuration file, three types of files can be designated:
"Main" annotations, from which the pan-gene or family clusters are constructed. The computational cost for "n" main annotations is n^2:

annotation_files
cds_files
protein_files

Optional "extra" annotations, to be added to those constructed from "main" above. The computational cost of adding "m" extra annotations is linear with "m":

annotation_files_extra_constr
cds_files_extra_constr
protein_files_extra_constr

Optional "extra" annotations, NOT constrained by chromosome matches. Again, The computational cost of adding "m" extra annotations is linear with "m":

annotation_files_extra_free
cds_files_extra_free
protein_files_extra_free

To select annotations for the "main" set, try to pick those that are (1) high quality and (2) probably (together) capture most of the diversity for your species. I suggest starting with not more than about 20 annotations in the "main" set. You might expand that to 40 or 50; but I would not be surprised if 100 or 200 fails (or takes a VERY long time). Adding more to the "main" set will not necessarily produce a better pangene set, because the program will keep finding the same clusters, and will slowly add more and more "peripheral" pangenes -- some of which are likely to just be bad gene calls.

[qianhg0706]

Hi, Steven
Firstly, thank you very much for your previous advice on selecting the "main" annotations and setting the parameters. I really appreciate your guidance.I followed your suggestion and started with 20 annotations as the "main" set. I set iden=0.95 and cover=0.8 to construct the pangenes. However, during the mmseqs process, I encountered an error message: "there must be an error: 69995 deleted from 34177 that now is empty, but not assigned to a cluster". The results showed that there were 78,258 genes (in 09_genes_not_in_clusters.fna) out of 776,824 genes (in 02_all_main_cds.fna) that were unclustered. Additionally, the gene count in the 18_syn_pan_aug_extra.clust.tsv file was less than the total gene count of the main annotations (776,824). I'm wondering if this result is correct. If not, what adjustments should I make to my parameters?For reference, my conf header is as follows:

clust_iden='0.95'
clust_cov="0.80"
extra_iden='0.80'
TE_match_iden='0.40'
mcl_inflation='1.6'
strict_synt="1"
pctl_low="25"
pctl_med="50"
pctl_hi="75"

If I want every gene to appear in the final result, how should I adjust my parameters if needed?

Thank you again for your time and help.
Best regards,
qian

[StevenCannon-USDA]

Hi Qian,

I apologize: I accidentally deleted the email notification of your response.

Regarding the mmseqs error: I don't believe those indicate a genuine problem. Apparently it has been fixed in a soedinglab/MMseqs2#961, which I will incorporate in pandagma; but I don't think they are the cause of the problem you are seeing with mcl.

Regarding your result: those look reasonable to me. It is not unusual to see many genes in 09_genes_not_in_clusters.fna. Those get added back to the clusters established based on homology+synteny (07_pan_fasta_cds.fna).

Most of the starting genes should be present in 18_syn_pan_aug_extra.clust.tsv and 18_syn_pan_aug_extra.hsh.tsv, but not all will be. The remaining ones, which didn't cluster for some reason, are reported in 18_syn_pan_aug_extra_complement.fna.

For interpreting your results, also look at the out_pandagma/stats.txt file. It is difficult to know when a result is sufficient. You can certainly try different parameters and compare the differences between runs.