ECS executor for running jobs on external compute

[saif-el]

Hello.

My team has been running a Nextflow pipeline to crunch a lot of data. In the past we have had good experience with the tooling that your Nextflow provides, especially the integration with AWS Batch and S3. However, I think we have optimized the dollar costs per run as much as we can with AWS Batch (using EC2 spot) and S3, and it is still not good enough to meet our targets given the budget constraints.

Because of this, we are exploring the option to rent 3rd party or on-prem hardware and then orchestrate the containerized tasks using Amazon's ECS Anywhere feature. But I could not find any documentation on how to write our own executor/plugin for this use case. I am fairly confident that if we can run Nextflow on AWS Batch, we should be able to run it on ECS directly as well. At the same time, we do not have the bandwidth to run and maintain an EKS cluster to use the k8s executor (assuming its possible to do that).

Any help or pointers to relevant docs or source code would be appreciated.

Thanks for listening. We love Nextflow!

[pditommaso]

This is an interesting use-case, would not be possible to use ECS Anywhere along with Batch API? doing so you would not need to implement a custom executor for nextflow

[saif-el]

We thought of doing this at first, but AWS does not seem to have any documentation on how to use externally attached instances in Batch APIs. I also found this thread on the AWS public roadmap for containers, and is still marked as "Open". So, this is likely not possible, otherwise it would be the best of both worlds for us.

[pditommaso]

Understand. There's no specific documentation to write a nextflow executor, however there are plenty of example in the nextflow codebase. You may want to have a look at the AWS Batch executor for example

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchExecutor.groovy

Line 57 in ac90cc2

class AwsBatchExecutor extends Executor implements ExtensionPoint {