Failed to create symbolic link...File Exists

[tjdurant]

Hello NextFlow Community,

I am attempting to integrate a python script into my NF pipeline.

In order for the python script to run, it must be passed a path to a .csv file that is stored in some location.

When I run the pipeline for the first time, I see that it creates an 'Alias' to that path (running on a mac) and the script runs successfully.

However, if I then try to run it again, it fails with an error saying:

Command wrapper:
  ln: failed to create symbolic link 'path/to/identity/identity': File exists

This also happens on a linux server. I ran it once, worked fine. But if I run a second time, that symlink is there, and NF gets cranky that the symlink it was going to create is already there.

Also, of interest, the path that I am passing is '/path/to/identity' - but NF removes the leading / and adds a second identity to the end of the path. Weird to me, and wondering if anyone can shed light on why that happens.

But mainly, interested in how I can fix the former issue. I can solve it currently by deleting the Alias and re-running, but that is not a sustainable solution in production.

Thanks,
Tom

[tjdurant]

Okay, this was solved by not including the path as an input parameter. For those that might encounter a similar problem.

In other words, i had this:

  input:
  tuple val(sample_id), path(consensus_gisaid_fa) from consensus_ch_gisaid.collect()
  val (params.pipeline_version)
  path (params.path_to_file)

Changed to this:

  input:
  tuple val(sample_id), path(consensus_gisaid_fa) from consensus_ch_gisaid.collect()
  val (params.pipeline_version)

This results in a symlink not being created at all. So...problem avoided!

[mribeirodantas]

You can prevent ln from throwing that error by passing the -f parameter. From the manpage: If the target file already exists, then unlink it so that the link may occur. (The -f option overrides any previous -i and -w options.)

[kingarjun93]

This worked for me. Thanks!

[antortjim]

@mribeirodantas how can I achieve the effect of -f while still using NextFlow process inputs and file()? I have the same problem as @tjdurant when -resumeing the pipeline, but I cannot simply remove the input file, since it is indeed needed.

Thanks!
A

[mribeirodantas]

I was thinking out loud, I think. Can't remember exactly why I said that, as it's indeed Nextflow internals and one shouldn't be able to easily change that 😅

To be honest, it's not really clear to me why this is happening. Could you provide a minimal reproducible example so that I can test it? I believe some manual handling of files is being done.

[antortjim]

I have a workaround fow now, which is to look for cached folders where the exitcode is not 0 (i.e. failed executions) and remove them, which allows me to clean up while keeping successful previous executions that nextflow run -resume can simply skip. This is the shell script:

#! /bin/bash

WORK_FOLDER = foo

find $WORK_FOLDER -name .exitcode -exec grep -H 0 {} \; | cut -f 1 -d: | rev | cut -f 2- -d/ | rev | sort > success.txt
find $WORK_FOLDER -name .exitcode -exec grep -H . {} \; | cut -f 1 -d: | rev | cut -f 2- -d/ | rev | sort > all.txt
diff all.txt  success.txt  | grep "^<" | cut -f 2 -d' ' > failed.txt
cat failed.txt  | xargs -I% rm -rf %
find $WORK_FOLDER -name .exitcode -exec grep -H . {} \; | sort > clean.txt
# the folders in work (listed in clean.txt) should all have now .exitcode of 0

[Chutlhu]

Just experienced this 'Failed to create symbolic link... File exists" error as well with nextflow and python script.
In my case, I recognized that the some relative paths were broken.
Don't really know why, but I solved by using declaring the path to my python script outside from the process definition:
That is, from this

process run_python {
    input:
        path DATA
    script:
        pyfile = file("path/to/python/script.py")  # with this here, I get this error
        """
        python $pyfile --path-data $DATA
        """
DATA_PATH = Channel.fromPath("path/to/data.h5")
workflow {
    main:
        run_python(DATA_PATH)
}

To this:

pyfile = file("path/to/python/script.py")  # moved up here
process run_python {
    input:
        path DATA
    script:
        """
        python $pyfile --path-data $DATA
        """
DATA_PATH = Channel.fromPath("path/to/data.h5")
workflow {
    main:
        run_python(DATA_PATH)
}

[mribeirodantas]

Hello, @Chutlhu. Things handled inside processes should be provided through Nextflow channels. This Python script should ideally be within the ./bin folder of your project, this way Nextflow will automatically add it to the path and you'll be able to use it just fine. Or, if it's not a program you want to call but something else, another file, in an input channel to this process.