Feature request: demultiplexing in both forward and reverse directions

[krdav]

Hello and thanks for a great piece of software.
I am looking forwards to the v3 update, so far it looks very promising.

Currently, I am using the demultiplexing feature a lot but in one instance ran into the issue that the sequencing library was prepare using a ligation method which does not preserve the directionality of the barcodes. I.e. samples are barcoded with known barcode sequences but then adapter ligation causes a reversal of the direction in ~50% of the cases.

I would like AdapterRemoval to look for barcodes in both directions and demultiplex them into the same file. Either by allowing multiple barcodes for a single filename e.g.:

cat barcodes.txt
sample_1 ATGCGGA TGAATCT
sample_1 AGATTCA TCCGCAT
sample_2 ATGGATT ATAGTGA
sample_2 TCACTAT AATCCAT
sample_7 CAAAACT TCGCTGC
sample_7 GCAGCGA AGTTTTG

Which currently raises this reasonable excecption:
Duplicate sample name 'sample_1'; combining different barcodes for one sample is not supported. Please ensure that all sample names are unique!

Or alternatively, adding a --reverse option. That looks for barcodes in both orientations.

Of course, all of this can be achieved by specifying these reverse barcodes with unique barcode names and post processing to merge, but having it integrated into the demultiplexing step would be preferred.

[MikkelSchubert]

Thank you for the kind words.

I'm not opposed to adding something like an --allow-duplicate-samples option for the case where you, for whatever reason, have tagged the same sample with multiple barcodes. That will catch mistakes in the common case, while also enable you to deal with your data in a convenient manner.

It does add a bit of complexity, since reads needs to be trimmed with an adapter corresponding to the barcode that were identified (among other things), but I have some tentative ideas for how I can implement that without it becoming a huge mess. I'll give it a try in the v3 branch, since that's where I'm currently focusing most of my attention and since I can afford to make a mess there, and then back-port it to v2 once I've gotten it working.

Best,
Mikkel

[krdav]

Related to #56

[krdav]

Thanks, Mikkel!

Just to be sure; the mixed directionality thing is not so uncommon. It happens in cases where the Illumina adapters are added using blunt-end or TA ligation.

An --allow-duplicate-samples option would solve my issue for sure, but if it is primarily going to be used for this kind of mixed direction reads, then solving that problem in particular might be more powerful. From the user perspective it gets easier because only one set of barcodes have to be added (the reverse complementary being inferred by AR), from an implementation side of view it might also be easier as only two scenarios exist per barcode(-pair): 1) forward direction and 2) forward and reverse direction. I am also thinking that it would be easier to write a test for this case.

On the flip-side the --allow-duplicate-samples option would be more generalizable.

Regardless, I will be looking forward to test what you come up with.

[MikkelSchubert]

Hi Kristian,

Thank you for the additional context.
The main difficulty is just handling multiple barcodes per sample, once that is implemented then it would be simple enough to support automatically adding reversed barcodes as well.

There is a minor complication, though, that you might have an suggestion for how to handle:
In normal operation AR will generate and use adapter sequences with the barcodes included, so that the reads are trimmed/merged correctly. However, with --demultiplex-only and samples with multiple barcodes, that information is needed downstream but is not readily available. I guess I could append the barcode belonging to the mate read (which will be located in the 3' of the read itself) to the FASTQ header as meta-data or use a tag or read group in SAM/BAM output. So that way the information is available if needed. What do you think?

Best,
Mikkel

[krdav]

Well, it is hard for me to provide any useful recommendations as I haven't looked much at the source code. However, I think I understand your problem and I cannot think of any better ideas than storing the barcode information as read meta data. For fastq appending it to the header and for SAM format using a suitable tag. There is actually already a barcode tag (BC-tag): http://samtools.github.io/hts-specs/SAMtags.pdf

[MikkelSchubert]

The master branch now features support for a --multiple-barcodes option to allow multiple barcodes for the same sample and a --reversible-barcodes option to match both barcode1-insert-barcode2 and barcode2'-insert-barcode1', where ' indicates reverse complement. If you have the time, then I'd appreciate it if you could give it a try.

Once I've tested it a bit more then I'll try to back-port it to 2.x. That is unfortunately a bit more work than I hoped due to the fact that lots of code operates under the assumption that samples only have one barcode (pair), but it should be doable.

The command-line argument names are also not set in stone, if you have any suggestions for something that more clearly communicates the intent

[krdav]

Wow. This sounds promising. I compiled the most recent version and will test it soon, then I will report back. Thanks for doing this!

[krdav]

Okay, now I have tested the latest AR3 version. Besides being impressively fast and having some (my subjective opinion) better default settings I found the following regarding my original request:

1. The --multiple-barcodes option works great. It is a nice feature in its own right and can be useful for nested barcoding.
2. The --reversible-barcodes did not work as I expected. Let me explain why:
   The normal configuration is this (5' to 3'):
   adapter1-barcode1-insert-barcode2-adapter2
   My request relates to this (5' to 3'):
   adapter1-[barcode1-insert-barcode2]'-adapter2 = adapter1-barcode2'-insert'-barcode1'-adapter2 (using your notation ' to indicate reverse complement)
   Let's call it the "reverse barcode configuration".

Because mate2 is read from the opposite direction, what is normally specified to AR is:
adapter1-barcode1-insert-barcode2'-adapter2'
This results in demultiplexing, adapter and barcode removal leaving just insert. Applying this for the reverse barcode configuration, we get this:
adapter1-barcode2'-insert'-barcode1''-adapter2' = adapter1-barcode2'-insert'-barcode1-adapter2'

In short, the --reversible-barcodes option should only swap the barcode1, barcode 2 order, and not reverse complement them (I see now that my own example using barcodes.txt has the sample problem). When I try this by manually adding barcodes for both directions and use --multiple-barcodes it works.

It looks like --reversible-barcodes is implemented as a special case of --multiple-barcodes by adding the reverse complement of barcodes for all samples, if missing. Now, let me try to argue that the reverse order barcodes are a special case. This is because they carry information about directionality which is lost if treated as just another of multiple barcodes.

Maybe a new optional column containing the direction (forward or reverse) can be added to the barcode input file. Then the --reversible-barcodes option would fill in the missing direction, but I guess at that point it is almost redundant because the user can add it themselves. Importantly, the direction used for demultiplexing must be transferred to the output, either as a tag in the fastq/SAM file and/or by only outputting the forward direction (which is what I prefer).

I hope this makes sense.
Thanks,
Kristian

[MikkelSchubert]

Hi Kristian,

Thank you for the detailed explanation.

Unfortunately I am having a bit of trouble following along, and I am not 100% sure that I understood you correctly.

To simplify things, would it be possible for you to attach

1. A couple of unprocessed read pairs in the normal orientation
2. A couple of unprocessed read pairs in the reverse orientation
3. The barcodes you'd pass to adapterremoval for each of (1) and (2)

If possible, then reads that fully overlap the insert + barcodes, but it is not required. And by a couple of reads I do mean just 1 or 2 read pairs: It'd just easier for me if I have actual sequences that I can look at and check my understanding of the process against. An already public dataset exhibiting this behavior would of course also be fine.

Sorry for the trouble.

Normalizing the orientation of the inserts should be possible, as should adding tags to FASTQ/SAM/BAM reads with the (original) orientation.

Best,
Mikkel

[krdav]

Of course!

Here is an example of a 196 nt. sequence, sequenced by 150 nt. paired-end reads.
The 196 nt. sequence is what is in between the two Illumina adapters, and it contains 20 nt. barcodes on either side.
This is what I would refer to as adapter1-barcode1-insert-barcode2-adapter2, where the barcode1 is 20 nt. insert is 156 nt. and barcode2 is 20 nt.
In one case the sequence is in the forward direction and in the other case it is in the reverse. This is the expected outcome when doing ligation-based attachment of the Illumina adapters.
The reverse sequence is what I refer to as adapter1-[barcode1-insert-barcode2]'-adapter2.

I run the example using this command:
adapterremoval3 --merge --out-prefix demux_test --in-file1 test1.fastq --in-file2 test2.fastq --barcode-list test_barcodes.txt --multiple-barcodes

And here are the data:

cat test_barcodes.txt
bc_fw	GAAAGTGACAGAGTGCCCCA	ACAACCGTACACCCTAATGA
bc_rv	ACAACCGTACACCCTAATGA	GAAAGTGACAGAGTGCCCCA

cat test1.fastq
@forward
GAAAGTGACAGAGTGCCCCACACCTTCGTTGCCCCCATTAACCACGTACAGGTAAACTCGCGCTCAGAAGACCCTGTATGTCCACGTGGCCAGAACGCTTATAGGAGCACTATTTGAAAAATATGTTAGGCGAGCAGCTCTGCCGTGACG
+
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
@reverse
ACAACCGTACACCCTAATGATTCTCCGTCCCAGCATGATTTGACTTCGTCACGGCAGAGCTGCTCGCCTAACATATTTTTCAAATAGTGCTCCTATAAGCGTTCTGGCCACGTGGACATACAGGGTCTTCTGAGCGCGAGTTTACCTGTA
+
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

cat test2.fastq
@forward
ACAACCGTACACCCTAATGATTCTCCGTCCCAGCATGATTTGACTTCGTCACGGCAGAGCTGCTCGCCTAACATATTTTTCAAATAGTGCTCCTATAAGCGTTCTGGCCACGTGGACATACAGGGTCTTCTGAGCGCGAGTTTACCTGTA
+
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
@reverse
GAAAGTGACAGAGTGCCCCACACCTTCGTTGCCCCCATTAACCACGTACAGGTAAACTCGCGCTCAGAAGACCCTGTATGTCCACGTGGCCAGAACGCTTATAGGAGCACTATTTGAAAAATATGTTAGGCGAGCAGCTCTGCCGTGACG
+
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

[MikkelSchubert]

Thank for the explanation; that clears it up for me!

With regards to the orientation of the insert, just to make sure that I understand you correctly there as well:

When the "reverse" combination of barcodes is detected, am I understanding correctly that you would like AdapterRemoval output the insert sequence in the "original" orientation and/or record the detected direction with a tag in FQ/SAM/BAM records?

That is possible implement, but it's not obvious to me how you would handle unmerged reads, since you cant just reverse complement one (merged) sequence. How have you been post-processing your output when you manually specify both directions for your barcodes?

Going back to your earlier comments, the --reversible-barcodes option should probably be changed such that it errors out if a "reverse" combination of barcodes is already present in the barcode table, instead of just ignoring it if the sample is the same; that way the barcodes you provide would uniquely determine what is considered the "forward" direction. Would that work for you?

Best,
Mikkel

[krdav]

When the "reverse" combination of barcodes is detected, am I understanding correctly that you would like AdapterRemoval output the insert sequence in the "original" orientation and/or record the detected direction with a tag in FQ/SAM/BAM records?

Yes, that is correct.

How have you been post-processing your output when you manually specify both directions for your barcodes?

Rather crudely by ignoring/discarding unmerged reads. For what I do, these are a small fraction and typically indicate bad quality. Now thinking about a more principled way, you can of course also reverse complement unmerged reads but then they would also have to swap from read 1 to 2, and vice versa. That would be quite confusing and probably not recommendable. I guess in that case the detected direction must be recorded with a tag in FQ/SAM/BAM records.

Going back to your earlier comments, the --reversible-barcodes option should probably be changed such that it errors out if a "reverse" combination of barcodes is already present in the barcode table, instead of just ignoring it if the sample is the same; that way the barcodes you provide would uniquely determine what is considered the "forward" direction. Would that work for you?

That makes sense and it will work for me. I do think that being able to manually specify forward/reverse direction in the barcode file explicitly provides some value for the user in terms of comprehension, but it is absolutely not critical.

[MikkelSchubert]

Thank you for the explanation.

I'll have to think about it a bit more, and will probably think of other issues when I try to implement it, but the preliminary plan is as follows:

1. Change --reversible-barcodes so that it takes an optional argument specifying the orientation of barcodes in the table: Either forward, reverse, or explicit (or some such name), defaulting to forward.
2. If the forward or reverse option is used, AdapterRemoval will assume that barcodes in the (3 column) table have that orientation and auto-genrate the opposite sequences. This requires (checks) that the opposite sequences are not already present in the table.
3. If the explicit option is used, a 4th column is expected in the barcode table, specifying either the forward or reverse direction [1]. The opposite pair will be generated if not already present and will fail on mismatches between samples/orientations in existing/generated sequences. The opposite pair will not be generated (if missing) and it is permitted to label the forward and reverse barcodes using different sample names.
4. --reversible-barcodes will not imply --multiple-barcodes, so that a "explicit" table containing the wrong forward or reverse sequences will result in an error rather than resulting in 2 forward pairs and 2 reverse pairs being generated. Using --multiple-barcodes would then disable this check. This is basically how it works in the current implementation.

I considered if the direction column could be auto detected, but I think being explicit is the better option, since it avoids either failing to detect an intended orientation column or failing due to "invalid" values in what wasn't supposed to be a orientation column. I could possibly check and warn if a valid orientation column is present but not used

In addition, when --reversible-barcodes is enabled

1. Require that --merge is set; it could be implied, but generally I prefer explicit to implicit.
2. Record the orientation in the output reads, for both merged and unmerged reads.
3. Write merged reads in the forward orientation, and do nothing to other reads (other than adding tags).
4. Maybe collect some statistics about the frequency of each orientation for barcode pair

[1] Or some such labels. Abbreviations like fwd, +, rev, and - could also easily be supported.

[krdav]

On the top four points:

1. Great
2. Great
3. Consider if it makes sense to not generate the opposite barcode pair. After all the option here is explicit and the user can just add all pairs themselves. This also leaves the door open for specifying some barcodes with both directions and some without while demultiplexed into the same file (although I can't think of any reasons one would want that...).
4. Very reasonable.

On the bottom four points:

1. I also prefer explicit
2. Great
3. Sounds good
4. That would be very nice indeed since there is always the risk of orientation bias.

[MikkelSchubert]

Regarding (3), yes, that does make sense to only include the listed barcodes, though I'll have to be very clear in the documentation. For most people it'll probably be easier to just provide list all in one direction, so that the opposite direction can be generated automatically, but it can be nice to have an escape hatch for unusual data