The pod may be evicted by a node shutdown event

[syousry6]

Hello Everyone,

We are using nextflow with AWS EKS cluster, We use spot instances for cost saving. Unfortunately we are getting the following pod eviction issue every time we run the pipeline but we couldn't be able to figure it out, do you have any clue?

Nextflow version: 21.10.6

 Error executing process > 'alignment (B01-00159460-VL47175-A-d:L2)'

Caused by:
  Unable to find pod nf-6d5503fb55142c0f3507a0b7bb85e582 - The pod may be evicted by a node shutdown event

Command executed [/mnt/efs/ngs_scratch/nextflow/work/210812_NB501331_0892_AH57FVBGXK/test-new-rnaaconfig3/templates/alignment.sh]:

  #!/bin/bash

  source /mnt/efs/ngs_scratch/nextflow/work/210812_NB501331_0892_AH57FVBGXK/test-new-rnaaconfig3/templates/common.sh
  script_begin alignment "B01-00159460-VL47175-A-d"

  {

        echo "runid = 210812_NB501331_0892_AH57FVBGXK"
        echo "sample_name = B01-00159460-VL47175-A-d"
        echo "sample_id = S2"
        echo "lane_id = L2"
        echo "readFilesIn = B01-00159460-VL47175-A-d_S2_L002_R1_001.fastq.gz B01-00159460-VL47175-A-d_S2_L002_R2_001.fastq.gz"
        echo "uuid = test-new-rnaaconfig3"
        echo "reference = _REFs_002"
        echo "params_reference = /mnt/efs/ngs_scratch/data/ngs_clia/newref/_REFs_002/"

        echo "starting STAR at `date`"

        echo "making temporary directory /mnt/efs/tmp/test-new-rnaaconfig3"
        mkdir -p /mnt/efs/tmp/test-new-rnaaconfig3

        echo "removing temp dir in case of a retry..."
        rm -rf /mnt/efs/tmp/test-new-rnaaconfig3/STAR-210812_NB501331_0892_AH57FVBGXK-B01-00159460-VL47175-A-d.L2

        /usr/local/pipeline/packages/STAR_2.4.1b/STAR           --genomeLoad LoadAndKeep                --genomeDir /mnt/efs/ngs_scratch/data/ngs_clia/newref/_REFs_002/                --readFilesIn B01-00159460-VL47175-A-d_S2_L002_R1_001.fastq.gz B01-00159460-VL47175-A-d_S2_L002_R2_001.fastq.gz              --readFilesCommand zcat                 --outReadsUnmapped Fastx                --outSAMunmapped  Within                --outSAMtype BAM SortedByCoordinate             --outSAMattrRGline ID:B01-00159460-VL47175-A-d.L2 LB:library PL:Illumina SM:B01-00159460-VL47175-A-d                 --limitBAMsortRAM 15000000000           --runThreadN 4          --chimSegmentMin 15             --chimJunctionOverhangMin 15            --chimScoreMin 0                --chimScoreDropMax 20           --chimScoreSeparation 10             --chimScoreJunctionNonGTAG -1           --sjdbOverhang 74               --seedSearchStartLmax 30                --outFileNamePrefix L2.                 --outTmpDir /mnt/efs/tmp/test-new-rnaaconfig3/STAR-210812_NB501331_0892_AH57FVBGXK-B01-00159460-VL47175-A-d.L2
        ev=$?

  } 2>&1

  script_exit

Command exit status:
  -

Command output:
  (empty)

Work dir:
  /mnt/efs/ngs_scratch/workdir/210812_NB501331_0892_AH57FVBGXK/test-new-rnaaconfig3/work/6d/5503fb55142c0f3507a0b7bb85e582

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

[pdp10]

This might be the same as: #2424

[bentsherman]

@syousry6 can you try again with the latest edge release? There have been several improvements to the K8s executor in recent versions. In particular, try the experimental k8s jobs feature: https://www.nextflow.io/docs/edge/config.html#scope-k8s

Add k8s.computeResourceType = 'Job' to your config and see if that helps.

[syousry6]

Thank a million @bentsherman, we are testing now and will get back to you accordingly. Thanks once again for your efforts!