How to gracefully pass a nextflow.confg file when submitting task to Azure batch Pool?

[andy-canexia]

Hello, nextflow community,

I am wondering if anyone can generously share a cli example to create and submit tasks to Azure batch?

The one I am using is

az batch task create --job-id myjob --task-id andytask5 --command-line "nextflow run nextflow-io/rnaseq-nf -w az://my-testdata-output/work"

The pool with a custom image from shared image gallary and azure batch job have been created successfully. The error msg I got is about Missing AZURE_STORAGE_ACCOUNT_NAME.

Then I add one more extra args

az batch task create --job-id myjob --task-id andytask6 --command-line "nextflow -C /home/azureuser/nextflow.config run nextflow-io/rnaseq-nf -w az://my-testdata-output/work"

Then, nextflow complained about The specified configuration file does not exist: /home/azureuser/nextflow.config -- check the name or choose another file

I'm wondering if anyone can shed some light? Many thanks!!

[abhi18av]

@andy-canexia , would it be possible for you to SSH into the node and then try to debug using your custom values in /home/azureuser/nextflow.config file?

Also, the absence of these env variables, which should be automatically available makes me wonder whether you've connected your Azure Batch account to the Storage Account, refer here.

Out of curiosity, you have an interesting use-case of running a pipeline on Azure Batch on a task node, any particular reason for this?

[manuelesimi]

Hello,

I'm not sure why you are trying to do this. I mean, Nextflow takes care of interfacing AZ Batch, creating the pool and submitting tasks starting from your workflow definition. You only need to follow the documentation to use the azurebatch executor.

Do you have specific requirements for submitting nextflow itself as a task?

Thanks,
manuele