Skip to content

Implement image streaming for Google Batch#6836

Open
thalassemia wants to merge 2 commits intonextflow-io:masterfrom
thalassemia:gcp-image-streaming
Open

Implement image streaming for Google Batch#6836
thalassemia wants to merge 2 commits intonextflow-io:masterfrom
thalassemia:gcp-image-streaming

Conversation

@thalassemia
Copy link

Closes #4719. This improves startup time for jobs in my workflow from ~2.5 minutes to ~1 minute (image about 600MB compressed). It does, however, come with some pretty important limitations, most notably ignoring the process.containerOptions setting.

@thalassemia thalassemia requested a review from a team as a code owner February 14, 2026 21:06
@netlify
Copy link

netlify bot commented Feb 14, 2026

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit bec85f2
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69970fb95955dd000846cce5
😎 Deploy Preview https://deploy-preview-6836--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@pditommaso
Copy link
Member

Well limitation are not just about containerOptions. For Google docs

  • Batch only supports Image streaming for container images that are stored in Artifact Registry. If you currently use Container Registry to manage your container images, you can transition to Artifact Registry.
  • You must run your Batch job's VMs in the same location as where you store your container image in Artifact Registry.
  • Containers that use the Docker image manifest version 2, schema 1 aren't supported.
  • When you use Image streaming, containers runnables only support the following fields:
    • imageUri
    • commands
    • entrypoint
    • volumes
    • enableImageStreaming
  • Container images with empty layers or duplicate layers aren't supported.

https://docs.cloud.google.com/batch/docs/use-image-streaming

@thalassemia
Copy link
Author

Yes, I link to that exact page in the docs as part of my PR.

Signed-off-by: Sean Cheah <cheah_sean@yahoo.com>
Signed-off-by: Sean Cheah <cheah_sean@yahoo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support container streaming for Google Batch

3 participants