Environment variables detected with 1 task/cpu but fails with >1 cpu

[tomouellette]

Hi there,

Setup:
I am running nextflow (on a sge cluster with Ubuntu linux) for a very simple workflow consisting of a:

• Single nextflow process .nf
• Singe nextflow .config

Issue:
I am using a specific C program that has certain path requirements which means it has to be exported every run/task. Notably, I have no issue with the pipeline when only using a single cpu. However, with more than 1 cpu, the job fails everytime. When it fails, it returns the message saying "[dependency] cannot be found". The [dependency] being the code related to the specific C program.

Attempted failed solutions:

• I've explicitly placed the export statement in the script block
• I've explicitly placed the export statement in the beforeScript statement
• I've placed the export statement in .bashrc and sourced the .bashrc within and before the workflow
• I've placed the export statement in the bash script used for job submission on the target cluster (sge)
• I've placed the path that is exported within my PATH

Other diagnostics:

• Looking at the runs, the jobs tend to run a quarter to halfway before failing; usually when a second joblib Parallel function is called
• I have run the workflow locally on a Mac and because there is no path issues, I can use an arbitrary number of cpus without an issue

Likely problem and questions I need help with

• So it seems the problem has something to do with the export being dropped, and path information being lost, when a parallel task is executed
• Potentially related, the python code that I am using makes calls to the Parallel function in the joblib library. Does executing parallel tasks repeatedly under the hood cause problems in nextflow?
• Are there alternative solutions for handling the export?

Thanks for the help. Please let me know if you need any specific outputs and I can add them in the follow-up.

Updates

May 25, 2023: Adding an update to this - I have found a workaround where exporting my path to both (i) LD_LIBRARY_PATH in beforeScript and (ii) LD_LIBRARY_PATH in os.environ['LD_LIBRARY_PATH'] in python resolves the issue. Ideally, it seems like incorrect functionality to have to call os.environ across all classes/functions that execute a Parallel joblib call to make the workflow run - so I will leave this issue open.

[bentsherman]

It sounds like you are using a C program that you built yourself. In this case, you should install it to your host environment so that you can use it like any other program. If it expects some shared libraries, then you should install those libraries to the system path or add them to LD_LIBRARY_PATH in your .bashrc. Or if your SGE cluster allows you to create your own environment modules, then you should create a module for your program that handles all of that stuff.

This is the best practice when using Nextflow with locally-built programs. While you could have Nextflow take care of these environment variables, it should really be provided by the launching environment. That way you can use the pipeline regardless of whether the program is built from source or installed via package manager.

By the way, Nextflow now supports Spack, so if your software can be installed via Spack then that's probably the easiest way to do it.

If you can give me a more concrete example, I might be able to give some better advice.

[tomouellette]

So concrete example would be:

1. Downloaded and compiled https://openslide.org/ C on a HPC with a linux OS (Ubuntu 18.04.6)
2. Compiled openslide
3. Added the openslide library to LD_LIBRARY_PATH in .bashrc
4. Installed the openslide-python wrapper for openslide C
5. Built a basic nextflow pipeline to run a script using openslide
6. Ran pipeline successfully with 1 cpu
7. Pipeline then fails with more than >1 cpu

The hack that seems to allow for successfully running on multiple cpus involves:

1. Adding export LD_LIBRARY_PATH=[openslide_lib_path] in beforeScript
2. Adding import os; os.environ['LD_LIBRARY_PATH'] = [openslide_lib_path] directly in python script

So it seems that the .bashrc file isn't sourced by nextflow when running on multiple cpus.

I haven't heard of Spack but it seems quite useful for standardizing HPC workflows - I will check that out too.

[bentsherman]

Thanks, that's more or less the scenario I was imagining. My guess is the error happens with cpus > 1 because it has different code paths for cpus == 1 and cpus > 1, and the parallel version probably uses additional libraries.

For what it's worth, I have run custom binaries with Nextflow before, and I would just put these environment variables in my .bashrc. But maybe your SGE scheduler isn't running .bashrc for each job, maybe it's running a different script? I'm not sure.

I did a quick search and found spack packages for openslide:

• https://packages.spack.io/package.html?name=openslide
• https://packages.spack.io/package.html?name=py-openslide-python

I think if you install the python wrapper it will also install the C binary for you.