Running Nextflow on a Kubernetes pod and the pod is restarted

[pdp10]

Morning,

My Nextflow pipeline is executed within a Kubernetes pod.
Although the processes are set up with an errorStrategy, if the main pod triggering the pipeline is killed by the node (e.g. for resource optimisation), the whole pipeline crashes.

Is there a way to make the main pod robust against these run time issues?

For instance:

• is it possible that the pod terminated and restarted by the node, resumes the pipeline, automatically?
• Alternatively, is it possible to set up an error strategy for the main pod (probably not, but prefer to ask anyway)?

Thanks for your help,
Piero

[bentsherman]

There isn't much you can do if the Nextflow pod is deleted by external forces. You just have to retry the workflow.

We recently added the option to submit jobs instead of pods. The job just creates a pod in the background, but you can configure the job to automatically retry the pod if it fails for any reason. We don't actually use that feature in Nextflow because Nextflow has its own retry mechanism, but it would be interesting to use the job's retry for the nextflow job created by kuberun. That way you could configure the nextflow job to automatically resume the workflow any number of times if it fails.

On the other hand, it's wasteful if your pipeline is failing for other reasons. And if the nextflow job is failing randomly often enough, the better solution is probably to find a more stable place to launch nextflow jobs 😄