how to pair reads files that come out from a process? the fromFilePairs() appears cannot take process output.

[pan-genome]

I have a process, that is download reads from a internal data repository, then they are output into a reads channel. how I can build reads pair out of this output just the similar format as it is handled by fromFilePairs() method.

the content of the channel looks like this:
/workspace/nf-training-public/nf-training/work/8a/c151d982bf620c2d1c4982d1b4ecdc/new_gut_2.fq
/workspace/nf-training-public/nf-training/work/ea/ddf69ddb90d51d6f62225752687e72/new_gut_1.fq
/workspace/nf-training-public/nf-training/work/73/b15813b7a109222c9698c80c6df52d/new_liver_1.fq
/workspace/nf-training-public/nf-training/work/e1/d88bee780215f3f4d4466f500c746b/new_lung_1.fq
/workspace/nf-training-public/nf-training/work/e8/25719de3418190e9c8f986c208c0a7/new_liver_2.fq
/workspace/nf-training-public/nf-training/work/bb/31c7df8f17079f714277a1e79ce2e6/new_lung_2.fq

[bentsherman]

You probably can't use fromFilePairs directly because it is a channel factory, not an operator. However you should be able to chain together some operators to get the same behavior. Here is a sketch:

1. Given a channel of the output fastq files
2. Map each file to a tuple with the tissue type, e.g. ['gut', 'new_gut_2.fq']
3. Use groupTuple to group each pair by tissue type

You will then have tuples like ['gut', 'new_gut_1.fq', 'new_gut_2.fq']. You may need some extra logic to make sure they are in order, and from there you can manipulate the tuple however you want.

[pan-genome]

thanks @bentsherman, just tried .map().groupTuple() approach and it did the trick, sometime I do need the pairs in certain order e.g. a_1.fq a_2.fq. not the other way. do you have any suggestion what type of logic I can use here?

[programmeratlarge]

Hi @pan-genome - would you mind sharing the details of what you did with the map and groupTuple approach. I have a very similar problem and I'm struggling with the syntax a bit.

[bentsherman]

You will need another map step to manipulate the tuple, and Groovy list has a sort() method.

[programmeratlarge]

I have a very similar problem, but it's not obvious to me what the map syntax should be. I have one process that outputs a tuple that looks like:

[reads_id,
[/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample1_R2.fastq,
/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample1_R1.fastq,
/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample2_R2.fastq,
/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample2_R1.fastq,
/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample3_R2.fastq,
/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample3_R1.fastq,]]

and I want to split that into R1,R2 read pairs to send to the next process. As you suggest, so comination of map and groupTuple should do it, but the exact syntax in eluding me. Thanks for any help.

[bentsherman]

Let's see if @mribeirodantas can help you work out the syntax

[mribeirodantas]

Thanks for mentioning me, @bentsherman ! I love these challenges 😆

workflow {
  Channel
    .of(
      file('/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample1_R2.fastq'),
      file('/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample1_R1.fastq'),
      file('/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample2_R2.fastq'),
      file('/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample2_R1.fastq'),
      file('/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample3_R2.fastq'),
      file('/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample3_R1.fastq')
    )
    .set { my_samples }

  my_samples
    .map { it -> [it.name.split('_')[0], it] }
    .groupTuple()
    .view()
}

Final output:

[Sample1, [/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample1_R2.fastq, /local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample1_R1.fastq]]
[Sample2, [/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample2_R2.fastq, /local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample2_R1.fastq]]
[Sample3, [/local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample3_R2.fastq, /local/workdir/nextflow_scripts/work/f6/feeb76efb94fee70a1eafb79e7e250/Sample3_R1.fastq]]

To disentangle a bit what the map and groupTuple operators are doing:

1. Get the /path/to/file element from the channel, extract the beginning of the filename and convert the whole element into a tuple like ['file', '/path/to/file'].
2. Group the tuples in this new channel by the first element (default) of every tuple

[programmeratlarge]

That looks like what I need - I'll give it a try. Thanks!

[BoonsMcGee]

I think I was cc'ed unto this thread by mistake. Would you be able to remove me?

…

On Fri, Jan 6, 2023, 3:52 PM paulmunn ***@***.***> wrote: That looks like what I need - I'll give it a try. Thanks! — Reply to this email directly, view it on GitHub <#3053 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMS3UARDXMFBQABV4LXX2I3WRCAYDANCNFSM6AAAAAATICHKXQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mribeirodantas]

Hello, @BoonsMcGee. There is an "unsubscribe" button on the right side. I can't do it for you.