AZ Batch fails to load container from private azure container registry

[alliemclean]

Bug report

AZ Batch fails to load container from private azure container registry

Expected behavior and actual behavior

Expected behavior

To work following these instructions here.

This pipeline works with a local config as opposed to az-batch

Actual behavior

After AzBatch allocates the pool and the nodes, the task is launched in a node but produces no output.

[ba/7e4a36] process > somalier (1) [100%] 1 of 1, failed: 1 ✘          
Error executing process > 'somalier (1)'                                                                                                                 
                                                                                                                                                         
Caused by:                                                                                                                                               
  The task exited with an exit code representing a failure                                                                                               
                                                                                                                                                         
Command executed:                                                                                                                                        
                                                                                                                                                         
  echo "input-chr1.vcf.gz"                                                                                                                                                                                                                                                                         
  /root/somalier --help  

Command exit status:
  -

Command output:
  (empty)

I've tried to log into the node and it doesn't appear that the docker container ever ran. In the azure container registry the container appears to have been successfully pulled. Public containers work and this private container works as expected using a local config.

If I add containerOptions to the process I get an error:

Error executing process > 'somalier (1)'                                                                                                                 
                                                                                                                                                         
Caused by:                                                                                                                                               
  At least one value of specified task container settings is invalid  

This could be a second issue, but ideally I'd assume it should work.

I also tried to add a sleep command to beforeScript so I had time to log onto the node but that doesn't appear to run either. It appears like connecting the the registry is successful in the log, azcopy appears to install and work, but something is happening before the script runs to cause an error.

Steps to reproduce the problem

nextflow run test_private_reg.nf \
 --name "test_reg" \
 --in az://pipeline/input/input-chr1.vcf.gz \
 --outdir az://pipeline/output/ \
 -w az://pipeline/working/ \
 -c azure.config \
 -resume

test_private_reg.nf

#!/usr/bin/env nextflow

params.mount = "/mnt/shared/references/"

process somalier {
    debug true
    container '<server>.azurecr.io/<group>/<container>:latest'
    //containerOptions "--mount type=bind,source=${params.mount},target=/references"

    input:
    path(base)

    output:
    stdout

    shell:
    """
    echo "${base}"
    sleep 500
    /root/somalier --help
    """
}
workflow {
    vcf = Channel.fromPath(params.in)
    somalier(vcf)
}

azure.config

plugins {
  id 'nf-azure'
}
process {
  executor = 'azurebatch'
}

azure {
  batch {
    location = <loc>
    accountName = <batchAccountName>
    accountKey = <batchAccountKey>
    autoPoolMode = true
    allowPoolCreation = true
    deletePoolsOnCompletion = true
    deleteJobsOnCompletion = true
    copyToolInstallMode = 'node'
        pools {
       auto {
       autoScale = true
       vmCount = 5
       maxVmCount = 10
       }
    }
  }
  storage {
    accountName = <storageAccountName>
    accountKey = <storageAccountKey>
    fileShares {
      <fsname> {
            mountPath = "/mnt/shared/"
      }
    }
  }
  registry {
        server =  <server>.azurecr.io
        userName =  <containerUser>
        password =  <containerPassword>
    }
}

Program output

nextflow.log

Environment

• Nextflow version: 22.10.4.5836
• Java version: java 19.0.1 2022-10-18
• Operating system: macOS & Linux
• Bash version: 3.2.57

Additional context

• [email protected] - I tried 0.9.8 as well but it failed to parse the workflow

[abhi18av]

Hi @alliemclean ,

Interesting problem, could you please try to reproduce this by removing the fileshare configs and usage from the code?

For me, this is working fine with private containers 🤔

[alliemclean]

I finally had time to log onto the node and try to run the image and I got this error:

standard_init_linux.go:228: exec user process caused: exec format error

Upon googling I found that this is because I built the image on my mac. Even using the --platform parameter did not resolve this. I rebuilt the image on a linux VM, pushed it, and the issue was resolved.

The containerOptions also work now, but for some reason I don't have access to the root directory in the image. It's likely unrelated to nextflow but I'll keep debugging.

[abhi18av]

Ah, yeah - in that case I guess you're using an M1/M2 chip for building the image.

... but for some reason I don't have access to the root directory in the image.

In this case, I can suggest that it's worth a try with the priviledged or the runAs option for the pool

Reference https://www.nextflow.io/docs/latest/azure.html#advanced-settings

[alliemclean]

Thanks. I do have an M1/M2 processor. I also just discovered that I'm now unable to load the image that I built on a linux VM on my mac for a local run now. I guess switching back and forth between running locally and on batch will be non-trivial. Do you think it's necessary to host 2 images for the different environments and use a parameter to decide which to use? It seems that this is exactly what using docker should prevent.

I tried with privileged and it didn't work. I changed my Dockerfile to host my code in a different directory and changed the permissions there to access it.

It might be helpful if the error message related to failing to load the container propagated.

[abhi18av]

Do you think it's necessary to host 2 images for the different environments and use a parameter to decide which to use? It seems that this is exactly what using docker should prevent.

I think that if you use the docker buildx build functionality as documented here https://docs.docker.com/engine/reference/commandline/buildx_build/#platform then you can build your desired image for multiple platforms together.

$ docker buildx build --platform=linux/amd64,linux/arm64,darwin/arm64,darwin/amd64 .

Once the image is published on dockerhub, when you pull and run that particular docker container then depending on the host machine platform (M1/arm64 on your machine and x86 on the Azure Batch node), the image for the right platform should be pulled and used without requiring any specific options.

That being said, apologies if I can't confirm this as I don't have a M1/M2 machine available.

I know that @mribeirodantas has looked into the docker architecture issue in the past and perhaps he might have some advice to offer (please) 😊

[pditommaso]

I think linux/amd64,linux/arm64 is enough, Docker on Mac runs via a Linux VM