AWS Batch Job stuck in runnable

[STB1132]

Hi, I'm just starting to learn how to use Nextflow with AWS batch using this useful guide from Alex Peltzer, but when I run the code it gets stuck in Job status RUNNABLE forever. This problem is quite common from what I've seen, but I couldn't solve it with the documentation I found.

The simple process I am running:

/*
 * Parse software version numbers
 */
process get_software_versions {
    publishDir "${params.outdir}/pipeline_info", mode: 'copy',
        saveAs: { filename ->
            if (filename.indexOf(".csv") > 0) filename
            else null
        }

    output:

    file "software_versions.csv"

    script:

    """
    echo $workflow.manifest.version > v_pipeline.txt
    echo $workflow.nextflow.version > v_nextflow.txt

    cat v_pipeline.txt >> software_versions.csv
    cat v_nextflow.txt >> software_versions.csv

    """
}

The command:

#! /bin/bash
// To stop previous runnable or pending jobs
for state in SUBMITTED PENDING RUNNABLE
do
    for job in $(aws batch list-jobs --job-queue NGS_prioritize_sv  --job-status $state --output text --query jobSummaryList[*].[jobId])
    do
        echo -ne "Stopping job $job in state $state\t"
        aws batch terminate-job --reason "Terminating job." --job-id $job && echo "Done." || echo "Failed."
    done
done


nextflow run .  -c SV_region  -profile awsst  \
               --vcf 's3://{my_bucket}/subsample_test_SV.vcf.gz' \
               --experiment_id 'sampl3' \
               --outdir 's3://{my_bucket}/output/' \
               --awsqueue 'NGS_prioritize_sv' \
               --awsregion 'eu-central-1'

And my config file

awsst {
    accessKey = 'xxx'
    secretKey = 'xxx'
    region = 'eu-central-1'
    workDir = 's3://{my_bucket}/work/'
    container = 'xxx'
    executor = 'awsbatch'
    queue = 'NGS_sv'
}

aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws'

Could I be missing something?

[manuelesimi]

Hi,

A job can remain in the RUNNABLE status when:

1. the queue or the Compute Environment on AWS Batch are not in an healthy status. You can check if this is the case in the CE and queue pages on the AWS console.
2. the queue does not satisfy the job requirements (e.g. it can't allocate a node with the amount of memory requested)
3. a few more reasons explained here: https://aws.amazon.com/premiumsupport/knowledge-center/batch-job-stuck-runnable-status