Memory usage of each nextflow instance

[powelleric]

I'm finding each nextflow process requires around ~300Mb of RAM while running. I was surprised this was so high. I wanted to run 100 workflows simultaneously but don't have a machine big enough. Does this memory usage sound correct? How do others work around this?

[pditommaso]

Memory can change a lot depending on the complexity and the number of tasks the workflows execute. Also take into consideration nextflow runs on top of the JVM, therefore Java memory allocation rules applies. You can use classic Xms and -Xmx flags to control it. Read more here and you can find it more on ChatGPT/Stackoverflow.

Said that, I'd say that 300MB is fair amount of mem for a nextflow workflow.

[powelleric]

Thank you for your response, @pditommaso. I have tried to use the -Xms and -Xmx flags. I can tell that they are being correctly recognized when running _JAVA_OPTIONS="-Xmx68m" nextflow ... because it shows Picked up _JAVA_OPTIONS: -Xmx68m when NF starts.

Xmx seems to have reduced the memory footprint a bit. It is not quite clear what is happening because I put 68Mb and it uses ~250Mb, down from ~350Mb normally. But it seems to start OK with that. I will have to watch it to make sure it completes with the low allocation.

[bentsherman]

Also new blog post about this hot off the presses: https://nextflow.io/blog/2024/optimizing-nextflow-for-hpc-and-cloud-at-scale.html