Bash Parameter expansion

[Jbrenton191]

Bug report

I found a work around using either rename or basename functions but just wanted to highlight.

Expected behavior and actual behavior

Expected behaviour: Typical bash parameter expansion: ie:

~/nextflow_pd$ gtf_file=Homo_sapiens.GRCh38.97.gtf
~/nextflow_pd$ echo ${gtf_file%.gtf}
Homo_sapiens.GRCh38.97

Actual behaviour:
but when placed in a nextflow process it leads to this error:

Script compilation error

• file : /home/jbrenton/nextflow_pd/test_bug.nf
• cause: Unexpected input: '{' @ line 4, column 20.
  process gtf_to_bed {
  ^

1 error

Steps to reproduce the problem

-ran the following process with command: nextflow run test_bug.nf

nextflow.enable.dsl=2


process gtf_to_bed {

echo true

    input:
    path(gtf_file)

    output:
    path("*.bed")

    script:
    """
    echo $gtf_file > ${gtf_file%.gtf}.bed
        """
}
workflow {
gtf_file=file("$projectDir/Homo_sapiens.GRCh38.97.gtf")
gtf_to_bed(gtf_file)
}

Program output

`Script compilation error

• file : /home/jbrenton/nextflow_pd/test_bug.nf
• cause: Unexpected input: '{' @ line 4, column 20.
  process gtf_to_bed {
  ^

1 error

`

Environment

• Nextflow version: 21.04.0 build 5552
• Java version: 11.0.9.1-internal
• Operating system: Linux: Ubuntu 16.04.7 LTS (Xenial Xerus)
• Bash version: GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

Additional context

Running on a linux server

[abhi18av]

Hi @Jbrenton191 ,

The usage of $ in the script section is the issue.

script:
"""
echo $gtf_file > ${gtf_file%.gtf}.bed
 """

Please either

1. Escape the character for e.g. echo \$gtf_file ...
2. Use the shell directive which doesn't need these characters to be escaped https://www.nextflow.io/docs/latest/process.html#shell

[Jbrenton191]

Thanks very much for replying @abhi18av, and the page.

I tried doing 1. and 2. but still haven't got it to work.

I keep getting .command.sh: line 2: gtf_file: unbound variable

When you talk about escaping characters are you talking about the first or second $ sign/ $gtf_file in the example? Because to my understanding you can't escape the first since it won't interpret is as a nextflow variable being passed by path right? And if the second then it will think it s a missing bash variable because it hasn't been defined before? If I change script to shell I get the same problem of which to call as a nextflow variable vs a bash one. Maybe I'm missing something/getting confused? But I think it is the interpreting of bash vs nextflow variables and I haven't encountered it before, or maybe the % sign is the problem.

Again if I use a bash command like rename it works fine and can be used on the $gtf_file in the script section. But again wanted to flag.

Did you get it to work or something along the same lines?

[abhi18av]

Yes, you are right the % is the problematic expansion.

However, you don't need to rely upon that expansion to obtain the name of the file since the Nextflow variable gtf_file already captures the complete name of the file.

Also, in the snippet below it's worth noting the use of ! to indicate what's a Nextflow variable as with shell directive, all $var are assumed to be bash variables.

nextflow.enable.dsl=2

process gtf_to_bed {

echo true

input:
path(gtf_file)

output:
path("*.bed")

shell:
'''
echo !{gtf_file} > !{gtf_file}.bed

'''

}

workflow {
gtf_file=file("$projectDir/Homo_sapiens.GRCh38.97.gtf")
gtf_to_bed(gtf_file)
}

Here's the invocation behavior and the generated output of the script.

$ nextflow run test_bug.nf 
N E X T F L O W  ~  version 21.04.3
Launching `test_bug.nf` [nostalgic_descartes] - revision: 2e9cd1295e
executor >  local (1)
[18/59296b] process > gtf_to_bed [100%] 1 of 1 ✔


$ ls work/18/59296b9e7ec11ee3009c16b9c1ec0d/
total 32
drwxr-xr-x 11 eklavya staff  352 Sep 15 16:55 .
drwxr-xr-x  3 eklavya staff   96 Sep 15 16:55 ..
lrwxr-xr-x  1 eklavya staff   84 Sep 15 16:55 Homo_sapiens.GRCh38.97.gtf -> /projects/code/nextflow_playground/Homo_sapiens.GRCh38.97.gtf
-rw-r--r--  1 eklavya staff   27 Sep 15 16:55 Homo_sapiens.GRCh38.97.gtf.bed

[abhi18av]

If you wish to have finer control over the name of the file then perhaps file attributes might be worth taking a look https://www.nextflow.io/docs/latest/script.html?highlight=basename#check-file-attributes

[Jbrenton191]

Thanks for the help @abhi18av, really appreciate the time and you looking into this. Your way works but it keeps adding extension, but like you say you can trim it with the groovy commands in the workflow. Or in using a rename command for example.

Just wanted to highlight so others might not get stuck on it briefly like I did.

[abhi18av]

Happy to be of help @Jbrenton191 .

Do you mind making the initial comment as solution?