Update nf-azure: Add Azure Compute Gallery image and verification options by LennyBEL · Pull Request #7338 · nextflow-io/nextflow

LennyBEL · 2026-07-15T13:58:45Z

Split from #7321

Overview

Azure Compute Gallery
Last year we had a major outage when an update to the Microsoft Marketplace Ubuntu 22.04 HPC image flipped its verification state to unverified. Because Nextflow only targets verified images, new pools could no longer be created. Our pipeline tooling did not support Ubuntu 24.04 yet, so we had to scramble to restore service.

In order to avoid such an event again, Microsoft suggested that we switch to Azure Compute Gallery images. Within Azure Compute Gallery, we have full control over the images. However, Nextflow did not support the usage of them yet.

A new option virtualMachineImageId has been introduced to support usage of the Azure Compute Gallery.

Image verification state
For non-production and testing purposes we occasionally need to use unverified Marketplace images, but Nextflow always forced verified.

This PR adds azure.batch.pools.<name>.verification with values verified (default), unverified, or any - where any ignores the verification state entirely. This setting is ignored when virtualMachineImageId is set.

Note: Windows support was left out compared to the original PR. OsType is no longer configurable and remains fixed on Linux.

Expected Impact

None, as the default behavior does not change at all.

We have been running these changes as a custom nf-azure fork (based on the upstream plugin) in production for a while, so we are confident they behave as intended.

Tests

Unit tests added/updated in nf-azure:

AzPoolOptsTest
- Compute Gallery image configuration (virtualMachineImageId + sku).
- Parsing/validation of verification (verified/unverified/any, case-insensitive), with an error on invalid values.
- Cache-key (funnel) sensitivity: the pool hash changes when the image config changes.
AzBatchServiceTest
- Updated the auto-pool id hash to reflect the new pool options now included in the cache key.

Docs updated: docs/azure.mdx and docs/reference/config.mdx.

Add support for provisioning Azure Batch pool nodes from a custom VM image in an Azure Compute Gallery via the new 'azure.batch.pools.<name>.virtualMachineImageId' option, and add the 'azure.batch.pools.<name>.verification' option to choose the image verification type ('verified', 'unverified' or 'any'). Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

netlify · 2026-07-15T13:58:50Z

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	`890ca8e`
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a57946f108481000802dd0b

Signed-off-by: Lenny <vdwlenny@outlook.be>

adamrtalbot

This would also help when Azure deprecates a Batch image without a replacement image lined up. It would improve the supply chain security of our Azure users. So worth it overall.

adamrtalbot · 2026-07-15T14:13:28Z

+
+    @ConfigOption
+    @Description("""
+        The image verification type to match when resolving the VM image from the Batch supported-images list. Can be `verified`, `unverified`, or `any` (default: `verified`). Ignored when `virtualMachineImageId` is set.


I feel like verified should be included in unverified? It seems weird to reject verified images when you're allowing unverified images 🤔 this would remove the need for "any".

In fact, unverified could be a boolean:

unverified = false // Only verified images unverified = true // allow unverified images, here be monsters

adamrtalbot · 2026-07-15T14:23:01Z

+**Image verification**
+
+By default, Nextflow only selects images that Azure Batch has formally verified. Set `verification` to `unverified` to also allow images that Azure Batch lists but has not verified, or to `any` to accept both. This setting is ignored when `virtualMachineImageId` is set.
+
+```groovy
+azure {
+    batch {
+        pools {
+            <pool-name> {
+                verification = 'unverified'  // 'verified' (default), 'unverified' or 'any'
+            }
+        }
+    }
+}
+```


This might be bordering on too much information for this doc and should go to the advanced config options.

adamrtalbot · 2026-07-15T14:25:45Z

@@ -175,6 +188,20 @@
        this.password = opts.password
        this.virtualNetwork = opts.virtualNetwork
        this.lowPriority = opts.lowPriority as boolean
+        this.virtualMachineImageId = opts.virtualMachineImageId ?: null
+        this.verification = parseVerification(opts.verification)
+    }


A bit of code tidying, let's put them with their logical groupings.

Suggested change

}

AzPoolOpts(Map opts) {

this.runAs = opts.runAs ?: ''

this.privileged = opts.privileged ?: false

this.publisher = opts.publisher ?: DEFAULT_PUBLISHER

this.offer = opts.offer ?: DEFAULT_OFFER

this.virtualMachineImageId = opts.virtualMachineImageId ?: null

this.verification = parseVerification(opts.verification)

this.sku = opts.sku ?: DEFAULT_SKU

this.vmType = opts.vmType ?: DEFAULT_VM_TYPE

this.fileShareRootPath = opts.fileShareRootPath ?: buildFileShareRootPath()

this.vmCount = opts.vmCount as Integer ?: 1

this.autoScale = opts.autoScale as boolean

this.scaleFormula = opts.scaleFormula

this.schedulePolicy = opts.schedulePolicy

this.scaleInterval = opts.scaleInterval as Duration ?: DEFAULT_SCALE_INTERVAL

this.maxVmCount = opts.maxVmCount as Integer ?: vmCount *3

this.startTask = new AzStartTaskOpts( opts.startTask ? opts.startTask as Map : Map.of() )

this.registry = opts.registry

this.userName = opts.userName

this.password = opts.password

this.virtualNetwork = opts.virtualNetwork

this.lowPriority = opts.lowPriority as boolean

}

adamrtalbot · 2026-07-15T14:27:02Z

+        this.virtualMachineImageId = opts.virtualMachineImageId ?: null
+        this.verification = parseVerification(opts.verification)


Yeah, I think I like unverified = true/false more:

Suggested change

this.virtualMachineImageId = opts.virtualMachineImageId ?: null

this.verification = parseVerification(opts.verification)

this.virtualMachineImageId = opts.virtualMachineImageId ?: null

this.unverified = opts.verification as Boolean

adamrtalbot · 2026-07-15T14:28:16Z

        this.privileged = opts.privileged ?: false
        this.publisher = opts.publisher ?: DEFAULT_PUBLISHER
        this.offer = opts.offer ?: DEFAULT_OFFER
        this.sku = opts.sku ?: DEFAULT_SKU


There are enough fields here that it may require it's own config scope.

adamrtalbot · 2026-07-15T14:37:57Z

            if( it.osType != opts.osType )
                continue
-            if( it.verificationType != opts.verification )
+            if( opts.verification != null && it.verificationType != opts.verification )


If unverified was just a boolean, we might use falsey here to catch null and false together:

Or something, not sure, just spitballing 😆

Suggested change

if( opts.verification != null && it.verificationType != opts.verification )

if( opts.unverified && it.verificationType != opts.verification )

adamrtalbot

(Claude review here, but raised some good points)

Focused, sensible change — the verification == null = "any" handling in getImage, the gallery/marketplace branch in poolVmConfig, and the verified default all look correct. A few points below, most-important first.

Cache-key note (whole-PR): the pool funnel now hashes verification, which was a field but was previously not hashed. On upgrade every existing auto-pool id changes (hence the test hash change 42f3635f…→7483c5b1…), so existing pools won't be reused and will idle out. Worth calling out in the description — "Expected Impact: None" isn't quite accurate for users relying on pool reuse across the upgrade.

adamrtalbot · 2026-07-15T14:41:07Z

+    }
+
+    protected ImageReference customImageReference(AzPoolOpts opts) {
+        if( !opts.sku )


Medium — this guard is unreachable. In the AzPoolOpts constructor sku is opts.sku ?: DEFAULT_SKU ("batch.node.ubuntu 24.04"), so opts.sku is never null/empty and if( !opts.sku ) never fires.

The docs promise that for a gallery image "sku must be set to the Batch node agent SKU id that matches the image OS", but in practice a user who omits sku silently gets batch.node.ubuntu 24.04. If their custom image is a different OS the pool fails to provision at Azure with a much more confusing error than the intended one.

Either drop the dead guard, or detect "sku not explicitly provided" (check before the ?: DEFAULT_SKU fallback in the constructor) and enforce it for the gallery path.

adamrtalbot · 2026-07-15T14:41:07Z

 import com.azure.compute.batch.models.ContainerType
 import com.azure.compute.batch.models.ElevationLevel
+import com.azure.compute.batch.models.ImageReference
+import com.azure.compute.batch.models.MetadataItem


Unused import — only BatchMetadataItem (below) is used; MetadataItem isn't referenced. Drop this line.

adamrtalbot · 2026-07-15T14:41:07Z

        hasher.putUnencodedChars(virtualNetwork ?: '')
        hasher.putBoolean(lowPriority)
+        hasher.putUnencodedChars(virtualMachineImageId ?: '')
+        hasher.putUnencodedChars(verification?.toString() ?: 'any')


Minor: verification is hashed even when virtualMachineImageId is set, where it's documented/coded as ignored — two gallery configs differing only in a no-op verification get different pool ids. Cosmetic.

adamrtalbot · 2026-07-15T14:41:07Z

+        if( str == 'unverified' ) return ImageVerificationType.UNVERIFIED
+        if( str == 'any' ) return null
+        throw new IllegalArgumentException("Invalid azure.batch.pools.<name>.verification value: '$value' - expected 'verified', 'unverified' or 'any'")
    }


Optional: null here encodes a third state ("any") on an enum-typed field — a one-line comment would help future readers, since the type alone doesn't hint that null is meaningful.

LennyBEL requested a review from a team as a code owner July 15, 2026 13:58

LennyBEL mentioned this pull request Jul 15, 2026

Update nf-azure: node communication mode setting, allow Azure Compute Gallery and allow configuration of verification state #7321

Closed

LennyBEL and others added 2 commits July 15, 2026 16:04

Merge branch 'master' into nf-azure-compute-gallery

c284277

Signed-off-by: Lenny <vdwlenny@outlook.be>

Merge branch 'master' into nf-azure-compute-gallery

890ca8e

adamrtalbot approved these changes Jul 15, 2026

View reviewed changes

adamrtalbot reviewed Jul 15, 2026

View reviewed changes

-    }
+AzPoolOpts(Map opts) {
+this.runAs = opts.runAs ?: ''
+this.privileged = opts.privileged ?: false
+this.publisher = opts.publisher ?: DEFAULT_PUBLISHER
+this.offer = opts.offer ?: DEFAULT_OFFER
+this.virtualMachineImageId = opts.virtualMachineImageId ?: null
+this.verification = parseVerification(opts.verification)
+this.sku = opts.sku ?: DEFAULT_SKU
+this.vmType = opts.vmType ?: DEFAULT_VM_TYPE
+this.fileShareRootPath = opts.fileShareRootPath ?: buildFileShareRootPath()
+this.vmCount = opts.vmCount as Integer ?: 1
+this.autoScale = opts.autoScale as boolean
+this.scaleFormula = opts.scaleFormula
+this.schedulePolicy = opts.schedulePolicy
+this.scaleInterval = opts.scaleInterval as Duration ?: DEFAULT_SCALE_INTERVAL
+this.maxVmCount = opts.maxVmCount as Integer ?: vmCount *3
+this.startTask = new AzStartTaskOpts( opts.startTask ? opts.startTask as Map : Map.of() )
+this.registry = opts.registry
+this.userName = opts.userName
+this.password = opts.password
+this.virtualNetwork = opts.virtualNetwork
+this.lowPriority = opts.lowPriority as boolean
+    }

		this.virtualMachineImageId = opts.virtualMachineImageId ?: null
		this.verification = parseVerification(opts.verification)

	if( opts.verification != null && it.verificationType != opts.verification )
	if( opts.unverified && it.verificationType != opts.verification )

Uh oh!

Conversation

LennyBEL commented Jul 15, 2026

Overview

Expected Impact

Tests

Uh oh!

netlify Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for nextflow-docs canceled.

Uh oh!

adamrtalbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamrtalbot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

netlify Bot commented Jul 15, 2026 •

edited

Loading

adamrtalbot left a comment •

edited

Loading