run multiple slurm jobs using the same singularity container that is run via nextflow

[luciazifcakova]

Hi guys,
have some solved this issue already?

I want to run multiple slurm jobs using the same singularity container that is run via nextflow.
I have create separate temp dir for each input file from an slurm array to be run in and the output dir is also in created within temp dir.
The issue is that when I am within temp dir, nextflow cannot find the singularity container to run because it is outside of temp dir.
I have tried to provide sym links to nextflow command (did not work) but I don't know how to change the paths to singularity container within main.nf and nextflow.config that all point to the container outside of temp dir.

Here is my slurm job:

#!/bin/bash
#SBATCH --job-name=nf_repeat2_all_chr1
#SBATCH --partition=compute
#SBATCH --time=1-0
#SBATCH --mem=100G
#SBATCH --cpus-per-task=20
#SBATCH [email protected]
#SBATCH --mail-type=BEGIN,FAIL,END
#SBATCH --array=0-5

file_array=(/flash/MillerU/S-les_scaf2/Scaffold_*.fa)
file=${file_array[${SLURM_ARRAY_TASK_ID}]}
out=$(basename -s .fa $file) 

ml bioinfo-ugrp-modules
ml DebianMed
ml singularity
ml Nextflow
 
# create a temporary directory for this job and save the name
tempdir=$(mktemp -d nf.XXXXXX)
cd $tempdir
 
awk -v cuv2=$out '{sub(/XXXoutXXX/,cuv2); print;}' /flash/MillerU/main.nf | awk -v cuv1=$file '{sub(/XXXfileXXX/,cuv1); print;}' > main.nf

ls -s $tempdir/nf_repeat2.sif /flash/MillerU/nf_repeat2.sif
ls -s $tempdir/sanger2.sif /flash/MillerU/sanger2.sif

nextflow run main.nf -c /flash/MillerU/nextflow.config -with-singularity $tempdir/nf_repeat2.sif -w $tempdir/${out}

Here is how the path to singularity container is defined within main.nf:

process recentLTRs {
  container 'file://./nf_repeat2.sif'
  cache 'deep'

and full path for the container is added in nextflow.config.

[bentsherman]

Hi @luciazifcakova , it may be that your temp dir does not reside in shared storage. In that case only the nextflow job would be able to access it, the work jobs would not. Make sure you place your singularity image in a shared directory such as your home directory or a shared scratch directory.

Also, you don't need to specify the container image with both -with-singularity and container. I would just use -with-singularity without the image and use container with the absolute path. No need to create symlinks.