Add parameter `deferrable` to `AzureBatchOperator`

[joern-m-h]

Description

The AzureBatchOperator currently has no option to set it to deferrable, which would allow the task to be run by an async low-resource worker ("triggerer"). Therefore Azure Batch tasks block a regular worker, even though they only ping the Azure platform from time to time.

I propose to add this parameter and changing the operator to use asynchronous calls.

As a reference, the AWS Batch operator has a parameter deferrable.

Use case/motivation

Adding this parameter and changing the operator to use asynchronous calls would reduce resource consumption when using Azure Batch.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Ankurdeewan]

Looking at the current implementation, AzureBatchOperator submits the job and then spends the remaining of its execution synchronously polling Azure Batch. Both pool readiness and task completion are handled via repeated SDK GET/list calls wrapped in sleep loops (wait_for_all_node_state() and wait_for_job_tasks_to_complete()), with no callbacks, streams, or long-lived connections involved.

That control flow is inherently deferrable and effectively matches the AWS Batch operator’s model prior to deferral. Refactoring this to submit once, defer to a trigger for async polling, and resume on a terminal state would remove unnecessary worker blocking while keeping behavior unchanged, with deferrable=False remaining the default for compatibility.

I’m happy to take this on and implement it following the AWSBatchOperator/BatchJobTrigger pattern, unless there’s a different direction that would be preferred.

[potiuk]

I’m happy to take this on and implement it following the AWSBatchOperator/BatchJobTrigger pattern, unless there’s a different direction that would be preferred.

Feel free. I think when PR is raised, the implementation can be assessed, but "consistency with other, similar things" is important - you should look for other Azure operators if there are other similar patterns, but if not - borrowing the patterns from other providers is as good as it can get.