`egapx:gnomon_plane:chainer:run_chainer` FAILED

[jianghao0799]

When I annotating 770M genome with 51 Illumina RNA-seq and ONT DRS data, I get this.
run.yaml:

tasks:
  minimap2_wnode:
    minimap2-params: -t 20
  star_index:
    STAR: --runThreadN 30
trnascan:
  enabled: true

[fc/f99cde] process > egapx:rnaseq_long_plane:filter_est_align:align_sort                                                  [100%] 1 of 1 ✔                                         
[a8/2e2a60] process > egapx:gnomon_plane:chainer:run_align_sort (1)                                                        [100%] 1 of 1 ✔                                         
[dd/570d23] process > egapx:gnomon_plane:chainer:generate_jobs                                                             [100%] 1 of 1 ✔                                         
[64/f66eed] process > egapx:gnomon_plane:chainer:run_chainer (16)                                                          [ 74%] 43 of 58, failed: 43, retries: 42
...
[-        ] process > egapx:convert_summary_files:run_convert_summary_files                                                -                                                       
[-        ] process > egapx:busco:run_busco                                                                                -                                                       
[-        ] process > export                                                                                               -                                                       
ERROR ~ Error executing process > 'egapx:gnomon_plane:chainer:run_chainer (16)'                                                                                                    
                                                                                                                                                                                   
Caused by:                                                                                                                                                                         
  Process `egapx:gnomon_plane:chainer:run_chainer (16)` terminated with an error exit status (3)                                                                                   
                                                                                                                                                                                   
Command executed:                                                                                                                                                                  
                                                                                                                                                                                   
  echo "" > evidence_denylist.mft                                                                                                                                                  
  echo "" > gap_fill_allowlist.mft                                                                                                                                                 
  echo "list.seqids" > scaffolds.mft                                                                                                                                               
  echo "71274.trusted_proteins.gi" > trusted_genes.mft                                                                                                                             
  # HACK: derive start_job_id from job file extension                                                                                                                              
  filename=$(basename -- "job.015")                                                                                                                                                
  extension="${filename##*.}"                                                                                                                                                      
  (( start_job_id = ((10#$extension) * 13133) + 1 ))                                                                                                                               
                                                                                                                                                                                   
  mkdir -p tmp                                                                                                                                                                     
  # make the local LDS of the genomic and protein (if present) sequences                                                                                                           
  lds2_indexer -source indexed -db tmp/LDS2                                                                                                                                        
                                                                                                                                                                                   
  mkdir -p tmp/interim                                                                                                                                                             
  chainer_wnode -altfrac 80.0 -capgap 5 -cdsbonus 0.05 -composite 10000 -end-pair-support-cutoff 0.1 -endprotfrac 0.05 -filters 'remove_single_exon_est_models remove_single_exon_n
oncoding_models' -high-identity 0.98 -hmaxlen 0.25 -hthresh 0.02 -i3p 14.0 -i5p 7.0 -lenpen 0.005 -longenoughcds 900 -max-extension 20 -min-consensus-support 2 -min-edge-coverage 
5 -min-non-consensussupport 10 -min-support-fraction 0.03 -minex 10 -mininframefrac 0.95 -minlen 225 -minpolya 6 -minprotfrac 0.9 -minscor 40.0 -minsupport 3 -minsupport_mrna 1 -m
insupport_rnaseq 5 -mrnaCDS use_objmgr -oep 10 -protcdslen 450 -sharp-boundary 0.2 -tolerance 3 -trim 6 -utrclipthreshold 0.01 -fillgenomicgaps -filterest -filtermrna -filterprots
 -opposite -collapsest -start-job-id $start_job_id  -workers 32 -input-jobs job.015 -O tmp/interim -nogenbank -lds2 tmp/LDS2 -evidence-denylist-manifest evidence_denylist.mft -gap
-fill-allowlist-manifest gap_fill_allowlist.mft -param 4081.params -scaffolds-manifest scaffolds.mft -trusted-genes-manifest trusted_genes.mft                                     
  mkdir -p output
  cat tmp/interim/* > output/chainer_wnode.16.gpx-job.asnb
  rm -rf tmp

Command exit status:
  3
                                                                                                                                                                                   
Command output:                                                                                                                                                                    
  (empty)

Command error:
  00503/016/14157/R  A90001F7963B6BA1 212857/7092 2026-01-11T15:04:34.455659 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3773: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Link2 chains in 1.40667e-05                                                                                      
  00503/016/14157/R  A90001F7963B6BA1 212858/7093 2026-01-11T15:04:34.455722 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3804: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find genes in 4.79221e-05                                                                                        
  00503/016/14157/R  A90001F7963B6BA1 212859/7094 2026-01-11T15:04:34.455738 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3846: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Cap/polyA info in 9.53674e-07                                                                                    
  00503/016/14157/R  A90001F7963B6BA1 212860/7095 2026-01-11T15:04:34.455755 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         candidate_chains=
0                                           
  00503/016/14157/R  A90001F7963B6BA1 212861/7096 2026-01-11T15:04:34.455772 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         final_chains=0   
  00503/016/14157/R  A90001F7963B6BA1 212862/7097 2026-01-11T15:04:34.455983 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         prepare_time_msec
=2828.23&chain_time_msec=5.90706&filter_time_msec=0.0450611&format_time_msec=0.174999    
  00503/016/14157/RE A90001F7963B6BA1 212863/7098 2026-01-11T15:04:34.456231 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 2.835494995 2
51 119                                      
  After filtering: 0 introns, 1 alignments
  Forced orintation: 0
  After collapsing: 1 alignments 0 Flexible caps 0 Flexible polyas
  00503/018/14165/R  A90001F7963B6BA1 212864/6718 2026-01-11T15:04:34.679572 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         scaffold_start=39
64435&scaffold_stop=3964478&span_length=44&aligns_processed=1                            
  00503/018/14165/R  A90001F7963B6BA1 212865/6719 2026-01-11T15:04:34.693816 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3426: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Create pointers in 9.58443e-05                                                                                   
  00503/018/14165/R  A90001F7963B6BA1 212866/6720 2026-01-11T15:04:34.706668 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3430: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find overlapping with trusted in 6.91414e-06                                                                     
  00503/018/14165/R  A90001F7963B6BA1 212867/6721 2026-01-11T15:04:34.706727 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3434: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find contained in 1.09673e-05                                                                                    
  00503/018/14165/R  A90001F7963B6BA1 212868/6722 2026-01-11T15:04:34.706793 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3560: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find coding in 4.91142e-05                                                                                       
  Introns: 0 0 0
  00503/018/14165/R  A90001F7963B6BA1 212869/6723 2026-01-11T15:04:34.706867 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3664: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Link1 chains in 5.88894e-05                                                                                      
  00503/018/14165/R  A90001F7963B6BA1 212870/6724 2026-01-11T15:04:34.706884 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3670: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Chains for partial proteins in 2.14577e-06                                                                       
  00503/018/14165/R  A90001F7963B6BA1 212871/6725 2026-01-11T15:04:34.706915 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3773: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Link2 chains in 1.71661e-05                                                                                      
  00503/018/14165/R  A90001F7963B6BA1 212872/6726 2026-01-11T15:04:34.707010 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3804: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find genes in 7.98702e-05
  00503/018/14165/R  A90001F7963B6BA1 212873/6727 2026-01-11T15:04:34.707026 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3846: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Cap/polyA info in 9.53674e-07                                                                                    
  00503/018/14165/R  A90001F7963B6BA1 212874/6728 2026-01-11T15:04:34.707048 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         candidate_chains=
0                                           
  00503/018/14165/R  A90001F7963B6BA1 212875/6729 2026-01-11T15:04:34.707070 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         final_chains=0   
  00503/018/14165/R  A90001F7963B6BA1 212876/6730 2026-01-11T15:04:34.707322 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         prepare_time_msec
=2437.3&chain_time_msec=31.9321&filter_time_msec=0.0388622&format_time_msec=0.22912      
  00503/018/14165/RE A90001F7963B6BA1 212877/6731 2026-01-11T15:04:34.707578 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 2.469982147 2
49 119                                      
  After filtering: 0 introns, 1 alignments
  Forced orintation: 0
  After collapsing: 1 alignments 0 Flexible caps 0 Flexible polyas
  00503/009/14161/R  A90001F7963B6BA1 212878/6989 2026-01-11T15:04:34.732948 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         scaffold_start=52
70132&scaffold_stop=5270171&span_length=40&aligns_processed=1                            
  00503/009/14161/R  A90001F7963B6BA1 212879/6990 2026-01-11T15:04:34.734058 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3426: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Create pointers in 0.000102043                                                                                   
  00503/009/14161/R  A90001F7963B6BA1 212880/6991 2026-01-11T15:04:34.734079 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3430: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find overlapping with trusted in 2.14577e-06                                                                     
  00503/009/14161/R  A90001F7963B6BA1 212881/6992 2026-01-11T15:04:34.734091 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3434: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find contained in 5.96046e-06                                                                                    
  00503/009/14161/R  A90001F7963B6BA1 212882/6993 2026-01-11T15:04:34.734122 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3560: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find coding in 2.28882e-05                                                                                       
  Introns: 0 0 0
  00503/009/14161/R  A90001F7963B6BA1 212883/6994 2026-01-11T15:04:34.734153 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3664: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Link1 chains in 2.19345e-05                                                                                      
  00503/009/14161/R  A90001F7963B6BA1 212884/6995 2026-01-11T15:04:34.734176 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3670: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Chains for partial proteins in 1.19209e-06                                                                       
  00503/009/14161/R  A90001F7963B6BA1 212885/6996 2026-01-11T15:04:34.734205 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3773: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Link2 chains in 1.40667e-05                                                                                      
  00503/009/14161/R  A90001F7963B6BA1 212886/6997 2026-01-11T15:04:34.734272 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3804: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find genes in 5.60284e-05                                                                                        
  00503/009/14161/R  A90001F7963B6BA1 212887/6998 2026-01-11T15:04:34.734289 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", 
line 3846: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Cap/polyA info in 9.53674e-07                                                                                    
  00503/009/14161/R  A90001F7963B6BA1 212888/6999 2026-01-11T15:04:34.734307 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         candidate_chains=
0                                           
  00503/009/14161/R  A90001F7963B6BA1 212889/7000 2026-01-11T15:04:34.734323 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         final_chains=0   
  00503/009/14161/R  A90001F7963B6BA1 212890/7001 2026-01-11T15:04:34.734538 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         prepare_time_msec
=2644.05&chain_time_msec=5.96404&filter_time_msec=0.0309944&format_time_msec=0.191927    
  00503/009/14161/RE A90001F7963B6BA1 212891/7002 2026-01-11T15:04:34.734802 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 2.650731801 2
50 119
  00503/001/14169/RB A90001F7963B6BA1 212892/1321 2026-01-11T15:04:34.744373 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=o
utput&processing_cycle=331&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115                                                  
  00503/001/14169/R  A90001F7963B6BA1 212893/1322 2026-01-11T15:04:34.744412 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         success_jobs=21&f
ail_jobs=0                                  
  00503/001/14169/R  A90001F7963B6BA1 212894/1323 2026-01-11T15:04:34.744500 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         attempts=1       
  00503/001/14169/RE A90001F7963B6BA1 212895/1324 2026-01-11T15:04:34.744511 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000164985 0
 0                                          
  00503/000/0000/P  A90001F7963B6BA1 212896/0168 2026-01-11T15:04:35.582568 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Error: LIB(CException::eUnknown)
 "wn_app_core.cpp", line 256: ncbi::CGPX_WorkerAppCore::Run() --- 1639 jobs failed       
  00503/000/0000/P  A90001F7963B6BA1 212897/0169 2026-01-11T15:04:35.582656 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Error: CORELIB(106.16) "ncbiapp.
cpp", line 725: ncbi::CNcbiApplicationAPI::x_TryMain() --- Application's execution failed (CException::eUnknown) 1639 jobs failed                                                  
  00503/000/0000/PE A90001F7963B6BA1 212898/0170 2026-01-11T15:04:39.584999 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode stop          3 1356.811403036  

Work dir:
  /data/public/jianghao/00_egapx_v0.5.0/egapx/dm82_work/64/f66eedaee60997f1adda0b420fa877                                                                                          

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`                                                                  

 -- Check '/data/public/jianghao/00_egapx_v0.5.0/egapx/dm82_out/nextflow/nextflow.log' file for details                                                                            

None
To resume execution, run: sh /data/public/jianghao/00_egapx_v0.5.0/egapx/dm82_out/nextflow/resume.sh 

[YanXie-11-18]

Hi, I’m encountering exactly the same error under similar settings.
May I ask if you’ve managed to fix this issue? Any hint or related commit would be greatly appreciated.

[murphyte]

Noting @honghuxian on #197 is also getting the same chainer error.

@YanXie-11-18 - are you also using long reads?

Have any of you tried a run without the long reads, to help isolate the problem?

Are all of you using local long reads, or are you using long reads retrieved from SRA using egapx.py? If you're using your own long reads, what do the seq-ids look like? There may still be an issue related to long-read seq-id formatting that can lead to some problems in chainer (e.g. #188). That could also explain why we're having a hard time reproducing the issue, if we're not mimicking the problematic seq-ids.

[jianghao0799]

seq-ids look like this:

$zcat *corrected.fasta.gz| grep ">"  | head
>462b2802-b8ae-43d4-a910-de4acc1c3a5e
>75dd73e2-58a2-49d4-b90d-12b159087d24
>37dfbc12-cd7b-4d70-8d4b-d6b6a36752ec
>d66bcf8c-3f74-4b12-ad97-2a06bb0d8993
>def00bf9-a5a7-4143-8983-7f9f333faaa5
>db691fd7-1ddd-4707-998e-332b12783e18
>9014dc8a-d4a9-40e3-840a-512ebb5fdb2c
>5ab9ace4-e105-4822-b078-a7098e512460
>89f81f88-eb9c-4ee6-baf8-b84d56c34832
>027dd40d-8943-4601-adb9-6ca6f020afcc

I previously succeeded in running v0.4.1 using these reads.

[murphyte]

Are any of you working with publicly available data (either in GenBank + SRA, or files are available somewhere)? It would help with us trying to reproduce the problem.

[murphyte]

And how about post one of the chainer log files. e.g. find the path for a FAILED run_chainer task from nextflow/run.trace.txt, and post the .command.err file from that directory. That should provide some more information on where it's failing.

[jianghao0799]

I have deleted the previous log files.
And I have changed long-reads seq-ids, now seq-ids look like:

$grep "@" *corrected_rename.fq | head
@SRR4234567.1.1
@SRR4234567.2.1
@SRR4234567.3.1
@SRR4234567.4.1
@SRR4234567.5.1
@SRR4234567.6.1
@SRR4234567.7.1
@SRR4234567.8.1
@SRR4234567.9.1
@SRR4234567.10.1

I using egapx with publicly SRA(short-reads) and local long-reads.
I will give you log files and .command.err file, if I encounter the same issue when I re-run it with changed seq-ids.

[jiangxiuhan0310]

I have deleted the previous log files. And I have changed long-reads seq-ids, now seq-ids look like:

$grep "@" *corrected_rename.fq | head
@SRR4234567.1.1
@SRR4234567.2.1
@SRR4234567.3.1
@SRR4234567.4.1
@SRR4234567.5.1
@SRR4234567.6.1
@SRR4234567.7.1
@SRR4234567.8.1
@SRR4234567.9.1
@SRR4234567.10.1

I using egapx with publicly SRA(short-reads) and local long-reads. I will give you log files and .command.err file, if I encounter the same issue when I re-run it with changed seq-ids.

Hi, I’m also encountering the same error. Is the error still there after you changed the Seq-ID?

[jianghao0799]

Hi, 师兄!
This error still occurs after I changed the seq-ids.
Then I converted fq2fa, it's still running. I should know by tomorrow whether it works.

[jiangxiuhan0310]

Hi, 师兄! This error still occurs after I changed the seq-ids. Then I converted fq2fa, it's still running. I should know by tomorrow whether it works.

OK.thanks

[jianghao0799]

It's not work. But it had previously run successfully on v0.4.1. I will run these data with v0.4.1.
My error information:

Jan-24 23:32:47.983 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1552; name: egapx:gnomon_plane:chainer:run_chainer (15); status: COMPLETED; exit: 3; error: -; workDir: /data/public/jianghao/00_egapx_v0.5.0/egapx/dm82_work/e3/3982533635db62af7ef33845d55439]
Jan-24 23:32:47.983 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=egapx:gnomon_plane:chainer:run_chainer (15); work-dir=/data/public/jianghao/00_egapx_v0.5.0/egapx/dm82_work/e3/3982533635db62af7ef33845d55439
  error [nextflow.exception.ProcessFailedException]: Process `egapx:gnomon_plane:chainer:run_chainer (15)` terminated with an error exit status (3)
Jan-24 23:32:47.984 [Task monitor] INFO  nextflow.processor.TaskProcessor - [e3/398253] NOTE: Process `egapx:gnomon_plane:chainer:run_chainer (15)` terminated with an error exit status (3) -- Execution is retried (3)
Jan-24 23:32:47.986 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jan-24 23:32:47.986 [Task submitter] INFO  nextflow.Session - [ec/36f489] Re-submitted process > egapx:gnomon_plane:chainer:run_chainer (7)
Jan-24 23:34:55.660 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1556; name: egapx:gnomon_plane:chainer:run_chainer (16); status: COMPLETED; exit: 3; error: -; workDir: /data/public/jianghao/00_egapx_v0.5.0/egapx/dm82_work/2f/a7139f68745094b96558cd0678786a]
Jan-24 23:34:55.661 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=egapx:gnomon_plane:chainer:run_chainer (16); work-dir=/data/public/jianghao/00_egapx_v0.5.0/egapx/dm82_work/2f/a7139f68745094b96558cd0678786a
  error [nextflow.exception.ProcessFailedException]: Process `egapx:gnomon_plane:chainer:run_chainer (16)` terminated with an error exit status (3)
Jan-24 23:34:55.664 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jan-24 23:34:55.664 [Task submitter] INFO  nextflow.Session - [b1/20952f] Re-submitted process > egapx:gnomon_plane:chainer:run_chainer (9)
Jan-24 23:34:55.813 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'egapx:gnomon_plane:chainer:run_chainer (16)'

Caused by:
  Process `egapx:gnomon_plane:chainer:run_chainer (16)` terminated with an error exit status (3)

Command executed:

  echo "" > evidence_denylist.mft
  echo "" > gap_fill_allowlist.mft
  echo "list.seqids" > scaffolds.mft
  echo "71274.trusted_proteins.gi" > trusted_genes.mft
  # HACK: derive start_job_id from job file extension
  filename=$(basename -- "job.015")
  extension="${filename##*.}"
  (( start_job_id = ((10#$extension) * 1645) + 1 ))
  
  mkdir -p tmp
  # make the local LDS of the genomic and protein (if present) sequences
  lds2_indexer -source indexed -db tmp/LDS2
  
  mkdir -p tmp/interim
  chainer_wnode -altfrac 80.0 -capgap 5 -cdsbonus 0.05 -composite 10000 -end-pair-support-cutoff 0.1 -endprotfrac 0.05 -filters 'remove_single_exon_est_models remove_single_exon_noncoding_models' -high-identity 0.98 -hmaxlen 0.25 -hthresh 0.02 -i3p 14.0 -i5p 7.0 -lenpen 0.005 -longenoughcds 900 -max-extension 20 -min-consensus-support 2 -min-edge-coverage 5 -min-non-consensussupport 10 -min-support-fraction 0.03 -minex 10 -mininframefrac 0.95 -minlen 225 -minpolya 6 -minprotfrac 0.9 -minscor 40.0 -minsupport 3 -minsupport_mrna 1 -minsupport_rnaseq 5 -mrnaCDS use_objmgr -oep 10 -protcdslen 450 -sharp-boundary 0.2 -tolerance 3 -trim 6 -utrclipthreshold 0.01 -fillgenomicgaps -filterest -filtermrna -filterprots -opposite -collapsest -start-job-id $start_job_id  -workers 32 -input-jobs job.015 -O tmp/interim -nogenbank -lds2 tmp/LDS2 -evidence-denylist-manifest evidence_denylist.mft -gap-fill-allowlist-manifest gap_fill_allowlist.mft -param 4081.params -scaffolds-manifest scaffolds.mft -trusted-genes-manifest trusted_genes.mft
  mkdir -p output
  cat tmp/interim/* > output/chainer_wnode.16.gpx-job.asnb
  rm -rf tmp

Command exit status:
  3

Command output:
  (empty)

Command error:
  00605/034/1929/R  A900025D974E4D11 27524/0344 2026-01-24T15:34:10.354840 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         candidate_chains=0
  00605/034/1929/R  A900025D974E4D11 27525/0345 2026-01-24T15:34:10.354848 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         final_chains=0
  00605/034/1929/R  A900025D974E4D11 27526/0346 2026-01-24T15:34:10.354955 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         prepare_time_msec=1538.66&chain_time_msec=3.78084&filter_time_msec=0.0140667&format_time_msec=0.0948906
  00605/034/1929/RE A900025D974E4D11 27527/0347 2026-01-24T15:34:10.355077 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 1.543024063 257 119
  After filtering: 0 introns, 1 alignments
  Forced orintation: 0
  After collapsing: 1 alignments 0 Flexible caps 0 Flexible polyas
  00605/024/1924/R  A900025D974E4D11 27528/0865 2026-01-24T15:34:10.395940 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         scaffold_start=377233&scaffold_stop=377394&span_length=162&aligns_processed=1
  00605/024/1924/R  A900025D974E4D11 27529/0866 2026-01-24T15:34:10.396653 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3426: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Create pointers in 9.70364e-05
  00605/024/1924/R  A900025D974E4D11 27530/0867 2026-01-24T15:34:10.396667 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3430: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find overlapping with trusted in 9.53674e-07
  00605/024/1924/R  A900025D974E4D11 27531/0868 2026-01-24T15:34:10.396689 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3434: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find contained in 1.28746e-05
  00605/024/1924/R  A900025D974E4D11 27532/0869 2026-01-24T15:34:10.396775 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3560: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find coding in 7.79629e-05
  Introns: 0 0 0
  00605/024/1924/R  A900025D974E4D11 27533/0870 2026-01-24T15:34:10.396882 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3664: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Link1 chains in 9.799e-05
  00605/024/1924/R  A900025D974E4D11 27534/0871 2026-01-24T15:34:10.396891 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3670: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Chains for partial proteins in 9.53674e-07
  00605/024/1924/R  A900025D974E4D11 27535/0872 2026-01-24T15:34:10.396928 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3773: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Link2 chains in 3.00407e-05
  00605/024/1924/R  A900025D974E4D11 27536/0873 2026-01-24T15:34:10.396962 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3804: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Find genes in 2.6226e-05
  00605/024/1924/R  A900025D974E4D11 27537/0874 2026-01-24T15:34:10.396969 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Note[E]: GNOMON "chainer.cpp", line 3846: ncbi::gnomon::CChainer::CChainerImpl::MakeChains() --- Cap/polyA info in 0
  00605/024/1924/R  A900025D974E4D11 27538/0875 2026-01-24T15:34:10.396981 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         candidate_chains=0
  00605/024/1924/R  A900025D974E4D11 27539/0876 2026-01-24T15:34:10.396989 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         final_chains=0
  00605/024/1924/R  A900025D974E4D11 27540/0877 2026-01-24T15:34:10.397110 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         prepare_time_msec=1857.3&chain_time_msec=4.48179&filter_time_msec=0.0209808&format_time_msec=0.101089
  00605/024/1924/RE A900025D974E4D11 27541/0878 2026-01-24T15:34:10.397249 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 1.862298965 290 119
  00605/001/1931/RB A900025D974E4D11 27542/0275 2026-01-24T15:34:10.666036 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=88&normalized_load=105&total_ram=2162558332928&effective_ram=2160410849280&pct_ram_used=2&mem_total=55211728896&mem_peak=56532049920&mem_self=55211728896&mem_available=2105199120384&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115
  00605/001/1931/RE A900025D974E4D11 27543/0276 2026-01-24T15:34:10.666074 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000056028 0 0
  00605/001/1932/RB A900025D974E4D11 27544/0277 2026-01-24T15:34:13.666341 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=88&normalized_load=101&total_ram=2162558332928&effective_ram=2160410849280&pct_ram_used=2&mem_total=55211995136&mem_peak=56532049920&mem_self=55211995136&mem_available=2105198854144&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115
  00605/001/1932/RE A900025D974E4D11 27545/0278 2026-01-24T15:34:13.666382 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000061035 0 0
  00605/001/1933/RB A900025D974E4D11 27546/0279 2026-01-24T15:34:16.666669 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=88&normalized_load=101&total_ram=2162558332928&effective_ram=2160410849280&pct_ram_used=2&mem_total=55211995136&mem_peak=56532049920&mem_self=55211995136&mem_available=2105198854144&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115
  00605/001/1933/RE A900025D974E4D11 27547/0280 2026-01-24T15:34:16.666712 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000061035 0 0
  After filtering: 627 introns, 626960 alignments
  Forced orintation: 0
  00605/001/1934/RB A900025D974E4D11 27548/0281 2026-01-24T15:34:19.666997 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=88&normalized_load=97&total_ram=2162558332928&effective_ram=2160410849280&pct_ram_used=2&mem_total=55302492160&mem_peak=56532049920&mem_self=55302492160&mem_available=2105108357120&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115
  00605/001/1934/RE A900025D974E4D11 27549/0282 2026-01-24T15:34:19.667037 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000061035 0 0
  00605/003/1935/RB A900025D974E4D11 27550/0213 2026-01-24T15:34:20.397990 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=output&processing_cycle=54&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115
  00605/003/1935/R  A900025D974E4D11 27551/0214 2026-01-24T15:34:20.398039 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         success_jobs=29&fail_jobs=0
  00605/003/1935/R  A900025D974E4D11 27552/0215 2026-01-24T15:34:20.398148 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         attempts=1
  00605/003/1935/RE A900025D974E4D11 27553/0216 2026-01-24T15:34:20.398159 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000191926 0 0
  00605/001/1936/RB A900025D974E4D11 27554/0283 2026-01-24T15:34:22.667317 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=88&normalized_load=97&total_ram=2162558332928&effective_ram=2160410849280&pct_ram_used=2&mem_total=55279677440&mem_peak=56532049920&mem_self=55279677440&mem_available=2105131171840&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115
  00605/001/1936/RE A900025D974E4D11 27555/0284 2026-01-24T15:34:22.667358 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000060796 0 0
  00605/001/1937/RB A900025D974E4D11 27556/0285 2026-01-24T15:34:25.667646 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=88&normalized_load=93&total_ram=2162558332928&effective_ram=2160410849280&pct_ram_used=2&mem_total=55279681536&mem_peak=56532049920&mem_self=55279681536&mem_available=2105131167744&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115
  00605/001/1937/RE A900025D974E4D11 27557/0286 2026-01-24T15:34:25.667688 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000061988 0 0
  00605/005/0020/R  A900025D974E4D11 27558/0004 2026-01-24T15:34:28.277380 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Error: CORELIB(CCoreException::eNullPtr) "ncbiobj.cpp", line 1010: ncbi::CObject::ThrowNullPointerException() --- Attempt to access NULL pointer.
                    Stack trace:
                                      /img/gp/arch/x86_64/lib/libxncbi.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/corelib/ncbiobj.cpp:1012 ncbi::CObject::ThrowNullPointerException() offset=0x0 addr=0x2b9d85601f51
                                                                                                               /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/include/corelib/ncbiobj.hpp:1051 ThrowNullPointerException offset=0x0 addr=(nil)
                                                                                                                              /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/include/objmgr/tse_handle.hpp:682 GetNonNullPointer offset=0x0 addr=(nil)
                                                                                                                                      /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/objmgr/bioseq_handle.cpp:119 operator-> offset=0x0 addr=(nil)
                                                                                                                                     /img/gp/arch/x86_64/lib/libxobjmgr.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/include/corelib/ncbiobj.hpp:830 ncbi::objects::CBioseq_Handle::x_GetInfo() const offset=0x0 addr=0x2b9d80e15488
                                                                              /img/gp/arch/x86_64/lib/libxobjmgr.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/objmgr/seq_descr_ci.cpp:54 ncbi::objects::CSeq_descr_CI::CSeq_descr_CI(ncbi::objects::CBioseq_Handle const&, unsigned long) offset=0x0 addr=0x2b9d80d5972a
                                                                      /img/gp/arch/x86_64/lib/libxobjmgr.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/objmgr/seqdesc_ci.cpp:129 ncbi::objects::CSeqdesc_CI::CSeqdesc_CI(ncbi::objects::CBioseq_Handle const&, ncbi::objects::CSeqdesc_Base::E_Choice, unsigned long) offset=0x0 addr=0x2b9d80e51ee2
                                                                                                 /img/gp/arch/x86_64/lib/libxobjutil.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/objmgr/util/sequence.cpp:297 ncbi::objects::sequence::GetMolInfo(ncbi::objects::CBioseq_Handle const&) offset=0x0 addr=0x2b9d8087bd5d
                                                                     /img/gp/arch/x86_64/lib/libxalgognomon.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/algo/gnomon/aligncollapser.cpp:2431 ncbi::gnomon::CAlignCollapser::GetCollapsedAlgnments(ncbi::gnomon::CModelClusterSet<ncbi::gnomon::CModelCluster<ncbi::gnomon::CAlignModel> >&) offset=0x0 addr=0x2b9d63b6c2bb
                                                                                                                        /img/gp/arch/x86_64/bin/chainer_wnode /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/internal/gpipe/app/worker_nodes/gnomon/chainer_wnode_impl.cpp:415 ncbi::CChainerWnodeJob::Run(ncbi::objects::CScope&, std::__cxx11::list<ncbi::CRef<ncbi::objects::CSeq_align, ncbi::CObjectCounterLocker>, std::allocator<ncbi::CRef<ncbi::objects::CSeq_align, ncbi::CObjectCounterLocker> > >&) offset=0x0 addr=0x417e35
                                                                                                                                       /img/gp/arch/x86_64/bin/chainer_wnode /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/internal/gpipe/app/worker_nodes/gnomon/chainer_wnode_app.cpp:173 ncbi::CGPX_ChainerJob::Process(ncbi::objects::CScope&, std::__cxx11::list<ncbi::CRef<ncbi::objects::CSeq_align, ncbi::CObjectCounterLocker>, std::allocator<ncbi::CRef<ncbi::objects::CSeq_align, ncbi::CObjectCounterLocker> > >&) offset=0x0 addr=0x40de7c
              /img/gp/arch/x86_64/bin/chainer_wnode /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/internal/gpipe/app/worker_nodes/gnomon/chainer_wnode_base.cpp:156 ncbi::CGPX_ChainerJobBase::Process(std::__cxx11::list<ncbi::objects::CJob, std::allocator<ncbi::objects::CJob> > const&, int, ncbi::IGPX_IOContext&) offset=0x0 addr=0x423686
                                                                                           /img/gp/arch/x86_64/lib/libgpxlib_core.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/internal/gpipe/gpexec/queue/lib/wn_worker_thread.cpp:263 ncbi::CWorkerThread::x_DoJob(ncbi::CRef<ncbi::objects::CGPX_Job, ncbi::CObjectCounterLocker>, ncbi::CDiagContext&) offset=0x0 addr=0x2b9d63e28770
                                                                                                                                       /img/gp/arch/x86_64/lib/libgpxlib_core.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/internal/gpipe/gpexec/queue/lib/wn_worker_thread.cpp:462 ncbi::CParallelWorkerThread::Main() offset=0x0 addr=0x2b9d63e2a7ed
                                                                                                    /img/gp/arch/x86_64/lib/libxncbi.so /export/home/gpipe/TeamCity/Agent2/work/6d4fb99b9e4908f7/ncbi_cxx/src/corelib/ncbithr.cpp:578 ncbi::CThread::Wrapper(void*) offset=0x0 addr=0x2b9d85640610
                    /usr/lib/x86_64-linux-gnu/libc.so.6 :0  offset=0x0 addr=0x2b9d86ec2aa3
                                                                                                /usr/lib/x86_64-linux-gnu/libc.so.6 :0  offset=0x0 addr=0x2b9d86f4fa63
                                   :0  offset=0x0 addr=0xffffffffffffffff

  00605/005/0020/R  A900025D974E4D11 27559/0005 2026-01-24T15:34:28.277432 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Error: LIB "wn_worker_thread.cpp", line 267: ncbi::CWorkerThread::x_DoJob() --- error processing job: (CCoreException::eNullPtr) Attempt to access NULL pointer.
  00605/005/0020/RE A900025D974E4D11 27560/0006 2026-01-24T15:34:28.277557 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  500 428.453918933 0 20
  00605/003/1938/RB A900025D974E4D11 27561/0217 2026-01-24T15:34:28.294582 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=output&processing_cycle=55&ncbi_app_version=0.0.1940&ncbi_app_sc_version=29&ncbi_app_vcs_revision=704115&ncbi_app_revision=704115
  00605/003/1938/R  A900025D974E4D11 27562/0218 2026-01-24T15:34:28.294626 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         success_jobs=0&fail_jobs=1
  00605/003/1938/R  A900025D974E4D11 27563/0219 2026-01-24T15:34:28.294639 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode extra         attempts=1
  00605/003/1938/RE A900025D974E4D11 27564/0220 2026-01-24T15:34:28.294648 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000088214 0 0
  00605/000/0000/P  A900025D974E4D11 27565/0168 2026-01-24T15:34:42.059708 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Error: LIB(CException::eUnknown) "wn_app_core.cpp", line 256: ncbi::CGPX_WorkerAppCore::Run() --- 146 jobs failed
  00605/000/0000/P  A900025D974E4D11 27566/0169 2026-01-24T15:34:42.059805 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode Error: CORELIB(106.16) "ncbiapp.cpp", line 725: ncbi::CNcbiApplicationAPI::x_TryMain() --- Application's execution failed (CException::eUnknown) 146 jobs failed
  00605/000/0000/PE A900025D974E4D11 27567/0170 2026-01-24T15:34:45.252120 fat01           UNK_CLIENT      UNK_SESSION              chainer_wnode stop          3 452.281992912

Work dir:
  /data/public/jianghao/00_egapx_v0.5.0/egapx/dm82_work/2f/a7139f68745094b96558cd0678786a

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Jan-24 23:34:55.820 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `egapx:gnomon_plane:chainer:run_chainer (16)` terminated with an error exit status (3)
Jan-24 23:34:55.927 [Task monitor] DEBUG nextflow.Session - The following nodes are still active:
[process] egapx:gnomon_plane:chainer:run_gpx_make_outputs
  status=ACTIVE
  port 0: (value) OPEN  ; channel: files
  port 1: (value) bound ; channel: params
  port 2: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_wnode:gpx_qsubmit
  status=ACTIVE
  port 0: (value) bound ; channel: scaffolds
  port 1: (value) OPEN  ; channel: chains
  port 2: (value) OPEN  ; channel: chains_slices
  port 3: (value) bound ; channel: params
  port 4: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_wnode:annot
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: jobs
  port 1: (value) OPEN  ; channel: chains
  port 2: (value) bound ; channel: hmm_params
  port 3: (value) bound ; channel: softmask_lds2
  port 4: (value) bound ; channel: softmask
  port 5: (value) bound ; channel: genome
  port 6: (value) bound ; channel: proteins_asn
  port 7: (value) OPEN  ; channel: lines_per_file
  port 8: (value) bound ; channel: params
  port 9: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_wnode:gpx_qdump
  status=ACTIVE
  port 0: (value) OPEN  ; channel: files
  port 1: (value) bound ; channel: params
  port 2: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_evidence_summary:generate_scaffolds
  status=ACTIVE
  port 0: (value) OPEN  ; channel: evidence
  port 1: (value) OPEN  ; channel: evidence_slices
  port 2: (value) OPEN  ; channel: gn_models
  port 3: (value) OPEN  ; channel: gn_models_slices
  port 4: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_evidence_summary:gpx_qsubmit
  status=ACTIVE
  port 0: (value) OPEN  ; channel: scaffolds
  port 1: (value) OPEN  ; channel: gn_models
  port 2: (value) bound ; channel: params
  port 3: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_evidence_summary:gnomon_report
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: jobs
  port 1: (value) OPEN  ; channel: evidence
  port 2: (value) OPEN  ; channel: evidence_slices
  port 3: (value) OPEN  ; channel: gn_models
  port 4: (value) OPEN  ; channel: gn_models_slices
  port 5: (value) bound ; channel: sra_exons
  port 6: (value) bound ; channel: sra_exons_slices
  port 7: (value) bound ; channel: genome_asn
  port 8: (value) bound ; channel: protein_asn
  port 9: (value) OPEN  ; channel: lines_per_file
  port 10: (value) bound ; channel: taxid
  port 11: (value) bound ; channel: params
  port 12: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_evidence_summary:gpx_qdump
  status=ACTIVE
  port 0: (value) OPEN  ; channel: files
  port 1: (value) bound ; channel: params
  port 2: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_evidence_summary:create_reports
  status=ACTIVE
  port 0: (value) OPEN  ; channel: reports
  port 1: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_evidence_summary:gnomon_summary
  status=ACTIVE
  port 0: (value) OPEN  ; channel: gnomon_report
  port 1: (value) OPEN  ; channel: gnomon_quality_report
  port 2: (value) OPEN  ; channel: gn_models
  port 3: (value) bound ; channel: genome_asn
  port 4: (value) bound ; channel: params
  port 5: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:prot_gnomon_prepare:prot_gnomon_prepare_p
  status=ACTIVE
  port 0: (value) OPEN  ; channel: models
  port 1: (value) bound ; channel: params
  port 2: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:diamond_worker:run_diamond_egap
  status=ACTIVE
  port 0: (value) OPEN  ; channel: gnomon_prot_ids
  port 1: (value) bound ; channel: swiss_prot_ids
  port 2: (value) OPEN  ; channel: gnomon_prot_asn
  port 3: (value) bound ; channel: swiss_prot_asn
  port 4: (value) bound ; channel: params
  port 5: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:best_protein_hits:run_protein_filter_replacement
  status=ACTIVE
  port 0: (value) OPEN  ; channel: gnomon_prot_asn
  port 1: (value) bound ; channel: swiss_prot_asn
  port 2: (value) OPEN  ; channel: input_prot_alignments
  port 3: (value) bound ; channel: align_filter_params
  port 4: (value) bound ; channel: align_sort_params
  port 5: (cntrl) -     ; channel: $

[process] egapx:gnomon_plane:gnomon_add_filtering_scores:gnomon_filter_models
  status=ACTIVE
  port 0: (value) OPEN  ; channel: naming_db_secondary_support
  port 1: (value) bound ; channel: search_set_secondary_support
  port 2: (value) bound ; channel: gnomon_filtering_scores_file
  port 3: (value) OPEN  ; channel: gnomon_quality_report
  port 4: (value) OPEN  ; channel: best_naming_hits
  port 5: (value) OPEN  ; channel: models
  port 6: (value) bound ; channel: swiss_prot_asn
  port 7: (value) bound ; channel: params
  port 8: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:gnomon_biotype:run_gnomon_biotype
  status=ACTIVE
  port 0: (value) OPEN  ; channel: models_files
  port 1: (value) bound ; channel: splices_files
  port 2: (value) bound ; channel: denylist
  port 3: (value) bound ; channel: gencoll_asn
  port 4: (value) bound ; channel: swiss_prot_asn
  port 5: (value) bound ; channel: lds2_source
  port 6: (value) OPEN  ; channel: raw_blastp_hits
  port 7: (value) bound ; channel: name_cleanup_rules_file
  port 8: (value) bound ; channel: lineage_taxids
  port 9: (value) bound ; channel: parameters
  port 10: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:annot_builder:annot_builder_run
  status=ACTIVE
  port 0: (value) bound ; channel: gencoll_asn
  port 1: (value) bound ; channel: genome_asn
  port 2: (value) OPEN  ; channel: gnomon_annots
  port 3: (value) bound ; channel: cmsearch_annots
  port 4: (value) bound ; channel: trna_annots
  port 5: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:orthology_plane:extract_products_from_models:run_extract_products_from_models
  status=ACTIVE
  port 0: (value) OPEN  ; channel: models
  port 1: (value) bound ; channel: parameters
  port 2: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:orthology_plane:diamond_orthology:run_diamond_egap
  status=ACTIVE
  port 0: (value) OPEN  ; channel: gnomon_prot_ids
  port 1: (value) bound ; channel: swiss_prot_ids
  port 2: (value) OPEN  ; channel: gnomon_prot_asn
  port 3: (value) bound ; channel: swiss_prot_asn
  port 4: (value) bound ; channel: params
  port 5: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:orthology_plane:find_orthologs:run_find_orthologs
  status=ACTIVE
  port 0: (value) bound ; channel: input_gencoll_asn
  port 1: (value) bound ; channel: ref_gencoll_asn
  port 2: (value) OPEN  ; channel: input_annotations
  port 3: (value) bound ; channel: ref_annotations
  port 4: (value) OPEN  ; channel: prot_hits
  port 5: (value) bound ; channel: blastdb
  port 6: (value) OPEN  ; channel: input_proteins_asn
  port 7: (value) bound ; channel: ref_proteins_asn
  port 8: (value) bound ; channel: input_genome_asn
  port 9: (value) bound ; channel: ref_genome_asn
  port 10: (value) bound ; channel: parameters
  port 11: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:locus_track:run_locus_track
  status=ACTIVE
  port 0: (value) OPEN  ; channel: annotation
  port 1: (value) bound ; channel: gencoll_asn
  port 2: (value) bound ; channel: lxr_data
  port 3: (value) bound ; channel: parameters
  port 4: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:locus_link:run_locus_link
  status=ACTIVE
  port 0: (value) bound ; channel: best_refseq_prot_hit
  port 1: (value) OPEN  ; channel: orthologs
  port 2: (value) OPEN  ; channel: annotation
  port 3: (value) bound ; channel: gencoll_asn
  port 4: (value) OPEN  ; channel: gnomon_lds2_source
  port 5: (value) OPEN  ; channel: best_prot_hit
  port 6: (value) OPEN  ; channel: track_loci
  port 7: (value) bound ; channel: comparisons
  port 8: (value) bound ; channel: curr_prev_compare
  port 9: (value) OPEN  ; channel: gnomon_biotypes
  port 10: (value) bound ; channel: lxr_data
  port 11: (value) bound ; channel: lineage_taxids
  port 12: (value) bound ; channel: name_cleanup_rules_file
  port 13: (value) bound ; channel: proteins_asn
  port 14: (value) bound ; channel: name_from_ortholog
  port 15: (value) bound ; channel: parameters
  port 16: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:final_asn_markup:final_asn
  status=ACTIVE
  port 0: (value) bound ; channel: assembly_name
  port 1: (value) bound ; channel: gencoll_asn
  port 2: (value) bound ; channel: genome_asn
  port 3: (value) bound ; channel: scaffolds
  port 4: (value) bound ; channel: chromosomes
  port 5: (value) OPEN  ; channel: annots
  port 6: (value) OPEN  ; channel: locus_link
  port 7: (value) OPEN  ; channel: locustypes
  port 8: (value) bound ; channel: params
  port 9: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:generate_fasta_from_annots:gnomon_asn2fasta
  status=ACTIVE
  port 0: (value) bound ; channel: gencoll_asn
  port 1: (value) OPEN  ; channel: scaffolds
  port 2: (value) bound ; channel: chromosomes
  port 3: (value) bound ; channel: params
  port 4: (cntrl) -     ; channel: $

[process] egapx:annot_proc_plane:annotwriter:run_annotwriter
  status=ACTIVE
  port 0: (value) OPEN  ; channel: accept_asn_file
  port 1: (cntrl) -     ; channel: $

[process] egapx:convert_annotations:run_converter
  status=ACTIVE
  port 0: (value) OPEN  ; channel: asn_files
  port 1: (value) bound ; channel: gff_params
  port 2: (value) bound ; channel: gtf_params
  port 3: (cntrl) -     ; channel: $

[process] egapx:convert_summary_files:run_convert_summary_files
  status=ACTIVE
  port 0: (value) OPEN  ; channel: report
  port 1: (value) OPEN  ; channel: quality_report
  port 2: (value) OPEN  ; channel: locustypes
  port 3: (value) bound ; channel: locus_tag_prefix
  port 4: (cntrl) -     ; channel: $

[process] egapx:busco:run_busco
  status=ACTIVE
  port 0: (value) OPEN  ; channel: proteins
  port 1: (value) bound ; channel: lineage
  port 2: (value) bound ; channel: lineage_download
  port 3: (value) bound ; channel: parameters
  port 4: (cntrl) -     ; channel: $

[process] export
  status=ACTIVE
  port 0: (value) OPEN  ; channel: genomic_gff
  port 1: (value) OPEN  ; channel: genomic_gtf
  port 2: (value) OPEN  ; channel: genomic_fasta
  port 3: (value) OPEN  ; channel: transcript_fasta
  port 4: (value) OPEN  ; channel: cds_fasta
  port 5: (value) OPEN  ; channel: proteins_fasta
  port 6: (value) OPEN  ; channel: validated
  port 7: (value) OPEN  ; channel: stats
  port 8: (value) OPEN  ; channel: annotated_genome_asn
  port 9: (value) OPEN  ; channel: annotation_data_comment
  port 10: (value) bound ; channel: star_bam
  port 11: (value) OPEN  ; channel: busco_results
  port 12: (value) OPEN  ; channel: gnomon_biotype_contam_rpt
  port 13: (value) OPEN  ; channel: user_gnomon_report
  port 14: (value) OPEN  ; channel: user_gnomon_quality_report
  port 15: (value) OPEN  ; channel: gnomon_summaries
  port 16: (value) bound ; channel: mask_stats
  port 17: (cntrl) -     ; channel: $

Jan-24 23:34:55.932 [main] DEBUG nextflow.Session - Session await > all processes finished
Jan-24 23:34:55.936 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jan-24 23:34:56.045 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jan-24 23:34:56.062 [main] WARN  n.processor.TaskPollingMonitor - Killing running tasks (12)
Jan-24 23:34:56.115 [main] DEBUG n.executor.local.LocalTaskHandler - Unable to kill egapx:gnomon_plane:chainer:run_chainer (6) -- command: kill -TERM 279862; exit: 1 
 bash: line 0: kill: (279862) - No such process

Jan-24 23:34:56.159 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1515; failedCount=37; ignoredCount=0; cachedCount=0; pendingCount=3; submittedCount=1; runningCount=-1; retriesCount=36; abortedCount=12; succeedDuration=79d 12h 56m 27s; failedDuration=2d 10h 35m 20s; cachedDuration=0ms;loadCpus=-7; loadMemory=0; peakRunning=12; peakCpus=88; peakMemory=720 GB; ]
Jan-24 23:34:56.159 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jan-24 23:34:56.195 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jan-24 23:34:58.260 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jan-24 23:34:58.604 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jan-24 23:34:59.025 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

[jianghao0799]

I can run the same data on v0.4.1 but not on v0.5.0.

[jiangxiuhan0310]

I can run the same data on v0.4.1 but not on v0.5.0.

OK! Thanks a lot, This is a piece of information that has been helpful to me.

[murphyte]

Note we are still investigating this issue. We haven't been able to reproduce it yet which may be in part due to some unrelated issues. We'll update when we know more. In the meantime, please use v0.4.1, although note that there are substantial improvements in functional annotation for plants with v0.5 so you may want to wait for a fix.