Operation 'toFile' is not supported by AzPath

[FabianDK]

Dear Nextflow devs,

I am trying to connect the azure storage in my config with a very simple script that is just printing the head of a file and the folder contents.

I run it with: NXF_VER=22.10.3 nextflow run scripts/nextflow/test_ls.nf -c scripts/nextflow/nextflow_azure_mount.config

When I run this, I get:

N E X T F L O W  ~  version 22.10.3
Launching `scripts/nextflow/test_ls.nf` [naughty_linnaeus] DSL2 - revision: a1d6b99a4b
WARN: Local executor only supports default file system (unless Fusion is enabled) -- Check work directory: az://nf-scratch/work
[a3/d24ea5] process > FILE_CONTENT (1) [100%] 1 of 1, failed: 1
[-        ] process > DIR_CONTENT      [  0%] 0 of 1
Error executing process > 'FILE_CONTENT (1)'

Caused by:
  Operation 'toFile' is not supported by AzPath

The run folder is generated in the workdir in the az folder, but there is only the .command.sh, .command.run, and .azure_blob_dir.

This is the nf workflow file (test_ls.nf):

#!/usr/bin/env nextflow
nextflow.enable.dsl=2 

process DIR_CONTENT {

  // publishDir params.outdir, mode:'copy', overwrite: true

  input: 
    val dir
  output:
    path "*.txt"
  script:
    """
    #!/bin/bash
    ${params.cmd} ${dir} > dir_contents.txt
    touch test_touch.txt
    """
}

process FILE_CONTENT {

  // publishDir params.outdir, mode:'copy', overwrite: true

  input: 
    val file
  output:
    path "*.txt"
  script:
    """
    #!/bin/bash
    head ${file} > file_head.txt
    echo "test2" > test2.txt
    """
}

workflow {
    Channel
        .fromPath(params.folder, checkIfExists: true)
        .set { dir_input_ch }

    Channel
        .fromPath(params.file, checkIfExists: true)
        .set { file_input_ch }

    file_ch = FILE_CONTENT(file_input_ch)
    folder_ch = DIR_CONTENT(dir_input_ch)

}

I tried with and without the publishDir.

This is the config I am using (nextflow_azure_mount.config):

workDir = 'az://nf-scratch/work/'

params.folder = "az://nf-scratch/test/wkbooks"
params.file = "az://nf-scratch/test/wkbooks/report.Rmd"
params.outdir = "az://nf-scratch"
params.cmd = "ls"

plugins {
  id 'nf-azure'
}

azure {
        storage {
                accountName = "storage_account"
                sasToken = "sasToken_without_first_?"
        }
}

Do you know what could be causing this error?

[pditommaso]

This

WARN: Local executor only supports default file system (unless Fusion is enabled) -- Check work directory: az://nf-scratch/work

[FabianDK]

When I enable fusion, I get: Fusion feature requires enabling Wave service.

So I also enabled wave and got: Error executing process > 'FILE_CONTENT (1)' Caused by: Missing task container -- Fusion requires the task to be executed by a container process

I did this by adding these lines to the config:

fusion {
  enabled = true
}

wave {
  enabled = true
}

I was trying to follow the instructions from https://www.nextflow.io/docs/edge/azure.html but there is no mention of fusion or wave as a requirement.

[bentsherman]

Hi Fabian, Fusion support for Azure storage was added very recently, so it might not be in the docs yet. Wave and Fusion must be used with containers, so you would have to specify a container image for each process.

If you just want to print some files from an Azure bucket, but run everything locally instead of in Azure Batch, why don't you just use a local work dir? Nextflow will download the files from Azure into a local task directory and do everything else from there. That way you don't have to bother with containers.

[FabianDK]

Thanks for the reply. When I change the workDir (workDir = 'az://nf-scratch/work/') to a local one ('/home/user/test/') then dir_contents.txt and file_head.txt are generated but are empty.

It seems that nextflow is not able to pull the files from azure with the current setup. I also cannot see the azure files in the local workDir.

What exactly has to be changed in the .nf and .config above to run this locally?