IQ-TREE job terminated without explicit error message

[hungweichen0327]

Dear IQ-TREE developers,

I am running IQ-TREE (iqtree3) on a high-memory server and encountered an unexpected job termination without an explicit error message. I would like to ask for clarification regarding the memory usage warning and the subsequent job termination.

My command was:

iqtree3 -s 20251102_outgroup_filter.min4.fasta -alninfo -t PARS -m GTR+F+R6 -st DNA -bb 1000 --bnni -nt 40 -mem 800G -keep-ident --prefix 20251102_outgroup_filter.min4_GTR_v2

The server specifications are:

Total RAM: ~1 TB
CPU cores available: 128
I explicitly limited IQ-TREE to use 40 threads and 800 GB RAM.
During the run, IQ-TREE printed the following messages:

NOTE: 804234 MB RAM (785 GB) is required!
WARNING: Memory required per CPU-core (49.0866 GB) is higher than your computer RAM per CPU-core (7 GB), thus multiple runs may exceed RAM!

Attached log file is below
20251102_outgroup_filter.min4_GTR_v2.log

Shortly after this message (during model parameter estimation), the job was terminated ("killed") by the system, but no explicit error message was written in the IQ-TREE log.

My questions are:

Is the "memory required per CPU-core" warning purely informational, or does IQ-TREE internally attempt to allocate memory per thread in a way that could exceed the total memory limit (-mem)?

Given that I explicitly specified -nt 40 and -mem 800G, is manual tuning of -nt required to avoid such termination?

Is there a recommended strategy for selecting the number of threads for large SNP-based WGS alignments (millions of sites, ~100+ taxa) to avoid silent job termination due to memory pressure?

Any clarification on how IQ-TREE estimates and enforces memory usage across threads would be greatly appreciated.

Thank you very much!

[bqminh]

This is a massive dataset. I suspect it runs out of memory. The -mem option only enforces the RAM storage for the likelihoods, but there may be other RAM requirements, that may exceed this and I suspect it potentially is over the 1TB, causing iqtree to be killed by the system. The WARNING is supposedly informational, indeed. You can try to lower -mem, say, -mem 500GB, and see how it goes.

If you want to know the max RAM required without -mem option: you can use this formula:

#taxa * #site_patterns * #states (4 for DNA) * #rate_categories (6 for +R6 in this case) * 8, in bytes

So in your case, it is 144*101798779*4*6*8 bytes =2.815×10¹² bytes or 2.8 TB.

[roblanf]

You should also be able to see why the job was killed in the system log, as well as how much memory it was using at the time.

Alternatively (or additionally) you could use Valgrind to incrementally track the memory of your job and see if it climbs towards a peak before it's killed. Something like this might work:

valgrind --tool=massif iqtree -s example.phy -m GTR+G