Slow performance of nextflow with cloud life sciences and cloud batch executors

[AMCalejandro]

Bug report

Expected behavior and actual behavior

Exepcted behaviour
Nextflow pulls the image from the docker Hub and uses it in multiple processes

Actual behaviour
My understanding of what is happening is that the executors are either building a docker image from the Hub every time or the executors are building a new VM for every process of the pipeline

Steps to reproduce the problem

Running nextflow with my gcb or gls profile making use of gloud life sciences and cloud batch executor takes longer than running locally

Command
nextflow run workflows/main.nf -profile gcb -params-file params.yml

GitHub Remote with the main workflow and the config file I am using
Config file
main.nf

Environment

• Nextflow version: [version 22.10.7 build 5853]
• Java version: [openjdk version "11.0.19" 2023-04-]
• Operating system: [Running Linux on WSL2]
• Bash version: [GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)]

Additional context

I have been browsing the documentation to see the arguments we can use to manipulate the deployment of our pipelines.
There is no configuration that seems to allow me modify this

[bentsherman]

Hi @AMCalejandro , my understanding is that both GLS and GCB map each task to its own VM, which means there is little opportunity to pack tasks onto a VM and/or cache and re-use the Docker images, as you might with AWS or Kubernetes. Not much we can do about that.

I have heard that you can specify both a machine type and cpus / memory in such a way that multiple tasks are packed onto the same VM, which means that the Docker images could also be re-used (maybe). But I haven't been able to make it work myself.

@hnawar can you speak to whether it's possible to pack multiple tasks onto the same VM? Or if there is anything else we can do to reduce / hide latency?

We are working on a PR #3892 which will add support for array jobs, including Google Batch. This might allow you to amortize the latency of spinning up a VM (it uses a TaskGroup under the hood).

[AMCalejandro]

Hello,

I am just coming back to this
@hnawar I was wondering if you could advice whether packing multiple tasks onto a VM in GCP is possible or not.

Thanks!!