Issues passing directory and tuple value as input

[ashdederich]

I have a nextflow process file. As a channel input, I am trying to pass on a whole set of directories as input. Generally, the directories are named as such:

IGVF_DIR1_1
IGVF_DIR2_1
IGVF_DIR3_1
IGVF_DIR4_1

Everything before the _1 at the end can be considered the ‘plateid’.

Here is the channel I create and the first process:

params.batch='20220914'
image_dir_ch=Channel.fromPath('/project/shared/gcrb_igvf/ashley/workspace/images/${params.batch}/*_1',type: 'dir')

process createLoadDataCsvs{
    publishDir '/project/shared/gcrb_igvf/ashley/workspace/load_data_csv/${params.batch}/${plateid}/', mode: 'move'
    label params.process
    label 'pycytoandr'
    tag "${params.batch}"
    tag "${plateid}"

    input:
    tuple val(plateid), path(imagedir) from image_dir_ch
    
    output:
    tuple val(plateid), file("${plateid}_load_data.csv") into load_data_csv_ch

    script:
    """
   ./generate_load_data.sh $imagedir
   mv ${plateid}/load_data.csv ${plateid}_load_data.csv
   """
}

However, it fails and says:

ERROR ~ No such variable: plateid

I’m unsure how to fix this. Any tips?

Thank you!

[jemunro]

Hi,

I see a couple of issues:

1. publishDir should use double quotes instead of single quotes to allow variable interpolation, e.g.:

   publishDir "/project/shared/gcrb_igvf/ashley/workspace/load_data_csv/${params.batch}/${plateid}/", mode: 'move'

2. Repeating tag will cause it to be overridden (i.e. the first tag "${params.batch}" will be ignored)
3. The image_dir_ch will only contain the path to the directory, not a tuple of plateid and path to directory. I think you want something like:

   image_dir_ch = Channel
        .fromPath('/project/shared/gcrb_igvf/ashley/workspace/images/${params.batch}/*_1', type: 'dir')
        .map { [it.name, it ] }

Hopefully that helps.

[ashdederich]

Hi @jemunro,

Thank you for your quick response! The map operator definitely helped process the channel into a tuple. I was able to test it by this:

image_dir_ch=Channel.fromPath("/project/shared/gcrb_igvf/ashley/workspace/images/${params.batch}/*_1/",type: 'dir').map { [ it.name, it ] }.subscribe onNext: {println it }, onComplete: { println 'Done' }

printing each directory.

Now that it was emitting a tuple, I wanted to grab the tuple key as the plateid var. I simplified the process by this:

#!/usr/bin/env nextflow

/*this is the directory that leads to all of the plates*/
params.process='high_cpu_mem'
params.batch='20220914'
image_dir_ch=Channel.fromPath("/project/shared/gcrb_igvf/ashley/workspace/images/${params.batch}/*_1/",type: 'dir').map { [ it.name, it ] }.subscribe onNext: {println it }, onComplete: { println 'Done' }

process testtuple {    
    input:
    tuple val(plate), each('imagedir') from image_dir_ch

    script:
    """
    echo this is plate $plate
    """
}

And received this as the stdout:

N E X T F L O W  ~  version 0.31.0
Launching `test.nf` [prickly_shaw] - revision: 33428bd70f
[IGVF_DIR1_1, /project/shared/gcrb_igvf/ashley/workspace/images/20220914/IGVF_DIR1_1]
[IGVF_DIR3_1, /project/shared/gcrb_igvf/ashley/workspace/images/20220914/IGVF_DIR3_1]
[IGVF_DIR4_1, /project/shared/gcrb_igvf/ashley/workspace/images/20220914/IGVF_DIR4_1]
[IGVF_DIR2_1, /project/shared/gcrb_igvf/ashley/workspace/images/20220914/IGVF_DIR2_1]
Done
ERROR ~ No such variable: plate

 -- Check script 'test.nf' at line: 10 or see '.nextflow.log' file for more details

So, even though I am emitting a tuple, it will not recognize the tuple key value. Any suggestions?

**Oh, as a note, I also tried as input:

tuple val(plate), path('imagedir') from image_dir_ch

However, it asked if I meant to use 'each' instead.

Thank you!

[jemunro]

Hi,

It looks like you are using a fairly old version of Nextflow. I'd recommend updating if possible.

I think your version of Nextflow predates DSL2, where tuple and path where introduced. Try instead:

set val(plate), file(imagedir) from image_dir_ch

[ashdederich]

Hi, yes, you are completely correct. It turns out I was using a very old version of Nextflow on my HPC by accident.

I really appreciate your help! It looks like it is working now.

[jemunro]

No problem! Please mark as answered if it's resolved.