process complete but files are not published - terminated with an error exit status (1)

[complexgenome]

Hi developers,

I've a main.nf script that calls multiple scripts/processes. One of them is msi_sensor.nf as below:

process msi {

         maxForks 3
         debug true
         errorStrategy 'retry'
         maxRetries 2

         label 'medium_mem'
         publishDir path: "${params.outdir}/${sample_name}/msi/", mode: 'copy'

input:

        tuple val(sample_id_tumor),path(bqsrbam_tumor, stageAs: 'tumor_msi/*')
        tuple val(sample_id_tumor),path(bqsrbam_bai_tumor, stageAs: 'tumor_msi/*')
        tuple val(sample_id_normal),path(bqsrbam_normal, stageAs: 'normal_msi/*')
        tuple val(sample_id_normal),path(bqsrbam_bai_normal, stageAs: 'normal_msi/*')
output:

      tuple  val(sample_name), path("msi_output")
      tuple  val(sample_name), path("msi_output_dis")
      tuple val(sample_name),  path("msi_output_germline")
      tuple  val(sample_name), path("msi_output_somatic")

script:

sample_name=sample_id_normal.split('_N')[0]


"""

/data1/software/msisensor/binary/msisensor_Linux_x86_64  msi -d /data1/datasets/arvadosResources/msisensor/GRCh38.primary_assembly.genome.fa.microsatellites.list -n ${bqsrbam_normal} -o "msi_output" -t ${bqsrbam_tumor}

"""
}

In work directory /data1/users/sanjeev/nextflow_pipeline/scripts_pipeline/crusher_scripts/upgraded_split_csv/work/14/00b53d7f1b9dd30a30529d08300701 there are output files:

msi_output  
msi_output_dis  
msi_output_germline 
 msi_output_somatic 

I do not understand why the output files are not being pushed/published to the directory? Is it because the tool prints on the screen?

Please see attached .nextflow.log
nextflow.log

[colindaven]

Your process failed - exit status 1.

Try turning off debug - comment out
//debug true

Then find the error message. You can look in the work dir for an error msg too.

Are you really sure all the outputs are being created ?

Also the pipeline code indentation looks very messy, why not tidy this up a bit?

[complexgenome]

@colindaven Hi, thank you for looking into it.

I ran the following code which worked fine on unix shell.

/data1/software/msisensor/binary/msisensor_Linux_x86_64 msi -d /data1/datasets/arvadosResources/msisensor/GRCh38.primary_assembly.genome.fa.microsatellites.list -n normal_msi/MM-0489_N.sorted.markdup.recal.bam -o again_msi_output -t tumor_msi/MM-0489_T.sorted.markdup.recal.bam

Same output files are generated, same number of lines. The code/command works fine with the input files.

I looked into the work dir as well, checked .command.log, .command.err .command.sh .command.out .command.run files. For example:

I turned off the debug true it gives same line/errors. I created a new script to run this step.

cat .command.err

*** Summary information ***

Number of total sites: 1792837
Number of sites with enough coverage: 7461
Number of MSI sites: 27

These are information upon completion of tool, and not errors.
Does nextflow consider the log messages on screen as errors?

[colindaven]

You could for bugfixing try adding || true to the end of your command. I have to use this when reporting versions, which normally end with an exit code 1, which causes Nextflow to abort. It may be that your tool is writing to stderr and causing a similar problem.

If you distort the exit codes with this hack, then Nextflow cannot judge if your process succeded or not - which can cause problems for potential later steps in your pipeline, missing output etc.

/data1/software/msisensor/binary/msisensor_Linux_x86_64 msi -d /data1/datasets/arvadosResources/msisensor/GRCh38.primary_assembly.genome.fa.microsatellites.list -n ${bqsrbam_normal} -o "msi_output" -t ${bqsrbam_tumor} || true

[colindaven]

You could also try rerouting stderr

/data1/software/msisensor/binary/msisensor_Linux_x86_64 msi -d /data1/datasets/arvadosResources/msisensor/GRCh38.primary_assembly.genome.fa.microsatellites.list -n ${bqsrbam_normal} -o "msi_output" -t ${bqsrbam_tumor} 2> stderr.txt

[complexgenome]

@colindaven I'd used || true which worked fine. The process's complete with no errors and I can see the output files published/staged to the output directory as needed.