calling all output files together in next process for the creation of database folder

[awast]

Hi ,
I am again facing a new issue problem . I need to run gatk Genomicdbimport on all normal samples and need to create only one folder of the database but after running below script i am able to generate folder but for every sample . I have searched for this thing so i need to apply loop for it or any other way?

Below my script:

process panelnormalsone {
    
    publishDir params.trim, mode : 'copy'
  
    input:
    tuple val(pair_id), file(normal_samplefile)
    tuple val(pair_id), file(normal_samplefile_bai)

    output:
    tuple val(pair_id),file(panelvcfnormal_samplefile), emit : vcf_normal
    tuple val(pair_id),file(panelvcfnormal_samplefile_stats), emit : vcf_normal_stats
    tuple val(pair_id),file(panelvcfnormal_samplefile_tbi),  emit : vcf_normal_tbi

    script:
    panelvcfnormal_samplefile = pair_id + '_recal_somatic.vcf.gz'
    panelvcfnormal_samplefile_stats = pair_id + '_recal_somatic.vcf.gz.stats'
    panelvcfnormal_samplefile_tbi  =   pair_id + '_recal_somatic.vcf.gz.tbi'
    
    """
    /mnt/raid1/gatk_latest/gatk-4.1.9.0/./gatk --java-options "-Xmx24G" Mutect2 -R ${params.reference} -I $normal_samplefile --max-mnp-distance 0 -O $panelvcfnormal_samplefile
    """

}

process Genomics_DB_creation{
    
    publishDir params.pondb, mode : 'copy'
      
    input:
    tuple val(pair_id),file(panelvcfnormal_samplefile)
    tuple val(pair_id),file(panelvcfnormal_samplefile_stats)
    tuple val(pair_id),file(panelvcfnormal_samplefile_tbi)

    output:
     
    """ 
   ./gatk --java-options "-Xmx24G" GenomicsDBImport -R  ${params.reference} -L ${params.intervalfile} --genomicsdb-workspace-path !{params.pondb} $ {input_vcf} --merge-input-intervals true --reader-threads 30 
   """
}  
  
Genomics_DB_creation(panelnormalsone.out)
  | collect   

i have to run a command like this:

/gatk --java-options "-Xmx24G" GenomicsDBImport -R  ${params.reference} -L ${params.intervalfile} --genomicsdb-workspace-path !{params.pondb} -V file 1 -V file2  -V file3 -V file 4 -V file 4 -V file 4

But it takes one sample at a time and generated database for every input sample. ALL i want a single folder from all input samples file.Please suggest

[bentsherman]

Hi @awast , I converted your issue into a Q&A and edited your post for formatting. If I understand correctly, you want to collect all of the samples produced by the panelnormalsone process and feed them into a single run of the Genomics_DB_creation process. In that case, you should use the collect operator before the db creation process. Something like:

(samples, stats, tbi) = panelnormalsone( /* ... */ )
Genomics_DB_creation( samples.collect(), stats.collect(), tbi.collect() )

That will run your db creation process only once with all of the samples. You can construct that command line with some custom Groovy code:

script:
sampleOpts = samples.collect { p -> "-V ${p.name}" } .join(' ')
"""
/gatk --java-options "-Xmx24G" GenomicsDBImport -R  ${params.reference} -L ${params.intervalfile} --genomicsdb-workspace-path !{params.pondb} ${sampleOpts}
"""

[awast]

Hi Sir,
Thanks for the suggestion. i am trying to implement and will update.

[awast]

Hi Sir,
Yes, i want to collect all of the samples produced by the panelnormalsone process and feed them into a single run of the Genomics_DB_creation process. I have used the above command but getting an error as:

Process 'panelnormalsone' has been already used -- If you need to reuse the same component, include it with a different name or include it in a different workflow context.

i have already used panelnormalsone in calling previous process under workflow section :
panelnormalsone(previouprocess.out)

then how can i implement this:

(samples, stats, tbi) = panelnormalsone( /* ... */ )
Genomics_DB_creation( samples.collect(), stats.collect(), tbi.collect() 

[awast]

Sir ,
i have reached somewhere near about the required command as it is taking all the input files together but showing error
here is my code :

process panelnormalsone {
    
    publishDir params.trim, mode : 'copy'
  
    input:
    tuple val(pair_id), file(normal_samplefile)
    tuple val(pair_id), file(normal_samplefile_bai)

    output:
    tuple val(pair_id),file(panelvcfnormal_samplefile), emit : vcf_normal
    tuple val(pair_id),file(panelvcfnormal_samplefile_stats), emit : vcf_stats
    tuple val(pair_id),file(panelvcfnormal_samplefile_tbi),  emit : vcf_tbi
    

    script:
    panelvcfnormal_samplefile = pair_id + '_recal_somatic.vcf.gz'
    panelvcfnormal_samplefile_stats = pair_id + '_recal_somatic.vcf.gz.stats'
    panelvcfnormal_samplefile_tbi  =   pair_id + '_recal_somatic.vcf.gz.tbi'
    
    """

    /mnt/raid1/gatk_latest/gatk-4.1.9.0/./gatk --java-options "-Xmx24G" Mutect2 -R ${params.reference} -I $normal_samplefile --max-mnp-distance 0 -O $panelvcfnormal_samplefile
    """
   
}
   
process Genomics_DB_creation {

   input:
   file(panelvcfnormal_samplefile)
   file(panelvcfnormal_samplefile_stats)
   file(panelvcfnormal_samplefile_tbi)
   
   output:
   
        
    
   script:
   sampleOpts = panelvcfnormal_samplefile.collect { p -> ("-V ${p.name}") } .join(' ')
    
   """ 
    
    /mnt/raid1/gatk_latest/gatk-4.1.9.0/./gatk --java-options "-Xmx24G" GenomicsDBImport -R ${params.reference} -L ${params.intervalfile} --genomicsdb-workspace-path ${params.pondb} ${sampleOpts}
--merge-input-intervals true --reader-threads 30 
    
   """
   
} 
workflow {
panelnormalsone(normalsamplenamerun.out)
  samples = panelnormalsone.out.vcf_normal
  stats = panelnormalsone.out.vcf_stats
  tbi = panelnormalsone.out.vcf_tbi
  Genomics_DB_creation( samples.collect(), stats.collect(), tbi.collect() )
  
}

but after running this error is like:

 A USER ERROR has occurred: Failed to create reader from file://input.1
  
  ***********************************************************************
  ./gatk GenomicsDBImport -R reference file  -L interval file --genomicsdb-workspace-path pondb -V input.1 -V file1_N_recal_somatic.vcf.gz -V input.2 -V file2_N_recal_somatic.vcf.gz -V input.3 -V file3_N_recal_somatic.vcf.gz -V input.4  -V file4_N_recal_somatic.vcf.gz -V input.5 -V file5_N_recal_somatic.vcf.gz

[bentsherman]

It looks like you need to add a bit more logic to make it work. Notice that the panelnormalsone.out.vcf_normal channel emits tuples with an id and a filename, but in the DB creation process you only want the filenames. You can use the map operator to extract just the files:

workflow {
  panelnormalsone(normalsamplenamerun.out)
  samples = panelnormalsone.out.vcf_normal | map { t -> t[1] }
  stats = panelnormalsone.out.vcf_stats | map { t -> t[1] }
  tbi = panelnormalsone.out.vcf_tbi | map { t -> t[1] }
  Genomics_DB_creation( samples.collect(), stats.collect(), tbi.collect() )
}

[awast]

Sir,
Again a big thanks to you. After combining and implementing your both suggestions this step is working fine.