skani dist memory issues for large number of genomes.

[MjelleLab]

Hi,
I am running skani dist v2 (due to skani database made by using v2) and comparing 14000 unknown genomes against 1.4mill sketches, using the commands skani dist "--ql" and "--rl". Skani dist crashed tue to insufficient memory. Is this the recommended approach to identify the closest species to my unknown 14000 genomes?

[bluenote-1577]

You can do skani search even in v0.2.0. This will take less memory and should work okay (assuming your 1.4 m genomes are > ~100kb with default params).

[jolespin]

@bluenote-1577 How much memory do you think would be required to run skani triangle with GlobDB:r226 with a single thread?

[bluenote-1577]

@jolespin hmmm... I would estimate this from doing skani triangle on a subset of ~1000 genomes fro GlobDB:r226, and then multiply by the total # of genomes.

The memory cost should be approximately linear in the # of genomes, but the time will not be. If you actually want to do skani triangle on globdb, and you only care about comparing similar genomes, I'd recommend using -s 85 or something higher to screen out more genomes, and maybe use --fast...

[jolespin]

What does --fast do? Does it increase the memory requirements or decrease the accuracy?

[jolespin]

@bluenote-1577 for GlobDB:r226 it's 226GB of memory using 30 threads. Crazy that it's the same number as the version lol

[bluenote-1577]

@jolespin sorry I didn't see your comment about --fast : the skani samples 1/120 k-mers by default. --fast samples 1/200 k-mers. This reduces memory and speed by a factor of 200/120. It makes it slightly less accurate for fragmented/lower ANI genomes, though. I usually use --fast if i only care about comparing > 95% ANI genomes and don't care about aligned fraction too much.