Publishing to remote server via publishdir()

[mebbert]

Hello,

We have a cluster and storage configuration that I don't think works with the Nextflow publishdir(). Allegedly this configuration is becoming more common out in the wild (according to our cluster administrators), so I'm wondering if there's already a solution we're missing, or if I should submit a feature request.

In short, we purchased private storage that is hosted by our campus computing center, but the storage is NOT directly mounted to the cluster. i.e., we can't access it using a standard Unix path. We have to transfer between the cluster and our private storage as though it's a remote server (e.g,. scp, rclone, etc.).

We wondered whether publishdir() might accept the following format (allowing ssh keys to handle authentication), but Nextflow rejects the @ symbol: <username>@<remote_server>:/destination/path/.

Error

Illegal character in scheme name at index 7: <username>@<remote_server:/destination/path/` # The @ symbol is at index 7

Ultimate questions:

1. Publishing to remote server using publishdir() via compute node: Any suggestions on how to specify a remote server as the publishdir()?
2. (Advanced) Publishing to remote server using publishdir() via the cluster's data-transfer node: Our cluster admins have emphasized that we'll have much faster transfer speeds if we can transfer between the cluster and our private storage using the cluster's data-transfer node. This would obviously require some advanced options for publishdir().

If neither of these are possible, there are obviously other ways to do this via command line as part of our jobs, but would be nice and tidy if we could do it with publishdir().

Really appreciate any suggestions.

[pditommaso]

This is an interesting use case. The publishDir relies on Java virtual file system, therefore the use of SSH (or other tools such as rclone) does not help here. e

However, our plan to tackle use cases similar to yours is to improve the support for S3-compatible object storage on the premise.

This would require:

• make the remote storage accessible via a S3 like interface e.g. Minio
• the nextflow pipeline use it via the Fusion (virtual) file system
• the pipeline requires the use of container runtime such as Singularity or Sarus

Could this work for you?