Workflow execution stalled

[fgualdr]

Hello everyone,

I am testing a workflow on different datasets.
I am running them on an HPC cluster with pbspro and I am using nextflow version 22.10.6 installed with conda locally.
I am experiencing the following issues. When running the code it stalled randomly at various steps.
Sometimes even re-running it on a successful datasets the workflow breaks (I mean a dataset for which the workflow worked from beginning to the end).
I am monitoring the workflow using nextflow Tower and the only thing I get is:
"Workflow execution stalled
It was not possible to determine the execution status for the workflow. This may be caused by an abrupt interruption of the workflow execution or by a protracted network connectivity problem."

I am struggling to figure out what to do.
Thanks in advance

Francesco

[pditommaso]

Can you please provide the .nextflow.log file?

[fgualdr]

Here is the nextflow.log.
This workflow appears currently still running from the cluster although I don't know if it is stalled or not.
On nextflow tower I got the "Workflow execution stalled" message associated with it.
nextflow.log
This particular dataset was run with the same workflow but stalled in a different point not the same of the workflow.
I cannot figure out whether is an issue of the HPC, the nodes, the connection of the clusters etc..

Thanks in advance
Francesco

[fgualdr]

Just to add a second example as I am running the pipe on multiple indipendent dataset here is a second workflow that appears stalled from nextflow tower (the nextflow.log attached).
nextflow.log

[pditommaso]

Looking in the log file it looks like, the multiqc hangs the execution because some input channels are not bringing the expected data. More precisely

  port 2: (value) OPEN  ; channel: software_versions
  port 3: (queue) OPEN  ; channel: workflow_summary
  port 14: (value) OPEN  ; channel: alignment/mergedLibrary/filtered/*
  port 15: (value) OPEN  ; channel: alignment/mergedLibrary/filtered/*
  port 16: (value) OPEN  ; channel: alignment/mergedLibrary/filtered/*
  port 17: (value) OPEN  ; channel: alignment/mergedLibrary/filtered/picard_metrics/*
  port 19: (value) OPEN  ; channel: deeptools/*
  port 20: (value) OPEN  ; channel: phantompeakqualtools/*
  port 21: (value) OPEN  ; channel: phantompeakqualtools/*
  port 22: (value) OPEN  ; channel: phantompeakqualtools/*
  port 23: (value) OPEN  ; channel: phantompeakqualtools/*
  port 24: (value) OPEN  ; channel: macs2/peaks/*
  port 25: (value) OPEN  ; channel: macs2/peaks/*
  port 26: (value) OPEN  ; channel: macs2/annotation/*
  port 27: (value) OPEN  ; channel: macs2/featurecounts/*

I don't know if it's a pipeline issue or a wrong configuration. Tagging @drpatelh for visibility