Difficulty returning env vars from process

[stkgo]

I've been attempting to adapt this example: https://www.nextflow.io/docs/latest/process.html#output-type-env to create a process that counts the number of variants in a VCF and am running into a couple of issues. Below is my demo pipeline:

process count_variants {
    input:
    tuple val(sampleID), path(vcf)

    output:
    env FOO

    script:
    '''
    FOO=$(/home/ubuntu/gatk-4.2.3.0/gatk CountVariants -V ${vcf})
    FOO=$(echo $FOO | awk -F ": " '{print $2}')
    '''
}

workflow {
    input_vcf = tuple "valid", file("/home/ubuntu/valid.vcf")
    println input_vcf
    count_variants(input_vcf) | view { "$it" }
}

When I hardcode the VCF in the process this works, however when I try to pass in a tuple containing a VCF as I do above I am getting

.command.sh: line 2: vcf: unbound variable

Here are the contents of .command.sh:

#!/bin/bash -ue
FOO=$(/home/ubuntu/gatk-4.2.3.0/gatk CountVariants -V ${vcf})
FOO=$(echo $FOO | awk -F ": " '{print $2}')

# capture process environment
set +u
echo FOO=${FOO[@]} > .command.env

I'm quite confused because I've written pipelines in the past that pass in files in the exact same way and they work without issues.

The second issue I'm having is that I can't seem to get the output from count_variants into a Nextflow variable. I went back to hardcoding the VCF and thought something along the lines of

count = count_variants()
println count[0]

would work but it simply prints

DataflowVariable(value=null)

Any help on either of these issues would be greatly appreciated.

[bentsherman]

This happens because you used single quotes ''' for your script block. Dynamic strings in Groovy must use double quotes.

[bentsherman]

I see, you're just using the branch operator incorrectly. The processes should be executed outside of the branch operator. But since there is only one decision branch, the filter operator is still the easier option so I'll give you an example with that instead:

variant_count = count_variants(vcf).filter { it.toInteger() > 0 }
annotate_ch = annotate_variants(vcf)
filter_variants(annotate_ch.vcf)

If the count is 0, the filter operator will return nothing and the downstream processes won't be executed. You can also use pipes to make this code simpler but I'll leave that as an exercise

[stkgo]

I tried this out on a very simple example:

process count_variants {
    input:
    tuple val(sampleID), path(vcf)

    output:
    env FOO

    script:
    """
    FOO=\$(/home/ubuntu/gatk-4.2.3.0/gatk CountVariants -V $vcf)
    FOO=\$(echo \$FOO | awk -F ": " '{print \$2}')
    """
}

workflow {
    input_vcf = tuple "invalid", file("/home/ubuntu/invalid.vcf")

    count = count_variants(input_vcf).filter { it.toInteger() > 0 }
    println "greater than 0"
}

and it always prints "greater than 0" immediately while process > count_variants still says 0 of 1 so I know it hasn't actually run the process yet. Does the count variable need to be an input to the downstream process in order for the filter to take effect?

[bentsherman]

Does the count variable need to be an input to the downstream process in order for the filter to take effect?

yes exactly. the print statement is executed before any processes are executed, so it cannot reference process outputs. the only way to use process outputs is to send them to downstream processes / operators.

[stkgo]

Does anything look wrong about this to you?

variant_count = count_variants(vcf).filter { it.toInteger() > 0 }
annotate_ch = vep_annotation(vcf, variant_count)
filter_variants(annotate_ch.vcf, variant_count)

For some reason it is still trying to annotate and filter a vcf that has 0 variants. I've double checked and both vep_annotation and filter_variants have an input for the variant count, so I would expect that Nextflow should respect the filter. I can run that same exact vcf through

process count_variants {
    input:
    tuple val(sampleID), path(vcf)

    output:
    env FOO

    script:
    """
    /home/ubuntu/gatk-4.2.3.0/gatk IndexFeatureFile --input $vcf
    FOO=\$(/home/ubuntu/gatk-4.2.3.0/gatk CountVariants -V $vcf)
    FOO=\$(echo \$FOO | awk -F ": " '{print \$2}')
    """
}

process dummy_process {
    input:
    val count

    script:
    """
    echo 'valid vcf' > /home/ubuntu/is_valid.txt
    """
}

workflow {
    input_vcf = tuple "invalid", file("/home/ubuntu/invalid.vcf.gz")

    count = count_variants(input_vcf).filter { it.toInteger() > 0 }
    dummy_process(count)
}

and see that dummy_process does not execute, as expected.

[bentsherman]

I'm not sure, your code looks right. Maybe use the view operator to inspect different channels and see what they are emitting