parabricks/fq2bam needs the fasta in the same folder as the BWAIndex

[famosab]

This leads to troubles when running on a cluster because a simple symlinking logic does not work there. However we want to move away from copying files which is not effective.

@gburnett-nvidia can you bring this up again with your team? Because we need fq2bam to be able to handle the reference differently. Maybe two command-line flags? One for fasta and one for the index?

We can update the module as soon as that works :)

See #9229 for the reverting to copying.

[famosab]

If I use any symlinking written in the script part of the process, I run into this error:

ln: failed to create symbolic link 'Homo_sapiens_assembly38.fasta': Permission denied

This is the cluster I am using: https://wiki.m3c.uni-tuebingen.de/ but from a quick search it seems to me that symlinking is something that might not easily work with the way certain filesystems are setup.

These are the commands for symlinking that I tried out:

    fasta_basename=\$(basename ${fasta})
    cd ${index} && \
        ln -s ../\$fasta_basename \$fasta_basename && \
        cd -

and

cd ${index} && ln -sf ../${fasta} ${fasta} && cd ..

[gburnett-nvidia]

Thank you for the extra information! I will talk to the wider team about other solutions.

[gburnett-nvidia]

@famosab I did some more looking into this and it seems like you're getting this error because my original solution involved creating a symlink in the index folder, which you might not have write permissions for. I propose a new solution (see PR) that instead creates a new ref folder with symlinks for all the reference and index files. I'm hoping that this new folder will not have the same permissions issues.

[famosab]

I tried something similar to this and that did not work :/ I think its just not possible to do this symlinking in this way on (our) cluster. It would be best if the input structure of fq2bam would not need this shuffling :)

[gburnett-nvidia]

I have made some edits to the PR.

It removes the requirement for a reference file when writing BAMs (by creating a dummy reference file in the correct location so PB doesn't error out and removing it after). I added an extra nf-test that shows this process working without a reference file given.

Writing CRAMs, however, does require a reference file in the index directory (which is also a requirement of bwa, so this is just the standard). Note that the bwa mem module does not test for CRAM, so we have nothing to compare against there. There are only two options to achieve this if the reference is not already in the correct location: copying the file or symlinking it.