update documentation (#142)

The documentation is still a work in progress, but this moves it significantly closer to being ready for the final release of v3.

Author: MikkelSchubert

CHANGES.md

@@ -1,5 +1,109 @@
 # Changelog
 
+## [3.0.0-alpha3] - 2025-05-19
+
+This is the third alpha release of AdapterRemoval v3. As with the previous alpha
+releases, changes that affect how AdapterRemoval is used (e.g. by removing
+options) or that result in different output compared to previous versions are
+marked with the label "[**BREAKING**]".
+
+AdapterRemoval now uses `meson` for its build process, and `meson` is therefore
+a build-time requirement. A `Makefile` is still provided to simplify setting up
+and running the build. See the installation instructions in the documentation
+for more information.
+
+Major changes include support for hardware accelerated alignments using NEON on
+modern Apple hardware, support for samples being identified by multiple
+barcodes, support for handling barcodes in that may ligate in different
+orientations, improved support for SAM/BAM output, and (optional) duplication
+plot in HTML report.
+
+### Added
+
+- Multiple barcodes/barcode pairs may now be used to identify the same sample,
+  via the `--multiple-barcodes` flag. The number of hits per barcode/barcode
+  pair is reported in the HTML/JSON reports.
+- Added support for handling barcodes that may ligate in different orientations
+  (via `--barcode-orientation`) and for normalizing the orientation of merged
+  reads (via `--normalize-orientation`).
+- The `--use-colors` parameter may now be used to controls color output.
+  Options are auto (default; enabled when run interactively), always, or never.
+- The title of the HTML report can now be set via `--report-title`.
+- Input files are now checked for duplicate filenames, in order to help
+  prevents accidental data duplication.
+- Alignments are now accelerated on Apple hardware using NEON instructions, for
+  a roughly 3-fold increase in throughput.
+- A duplication plot is now included in the HTML report if this is enabled,
+  instead of only being reported in the JSON file.
+
+### Changed
+
+- [**BREAKING**] Changed  `CO` tags for read-groups in SAM/BAM files to `DS`
+  (description) tags, in order to match the specification.
+- [**BREAKING**] A number of changes have been made to the JSON report layout,
+  including the moving, removal, and addition of sections. The layout is
+  described in `schema.json`.
+- [**BREAKING**] The minimum allowed/default value for `--min-adapter-overlap`
+  was set to 1. In practice this has no effect, since length 0 alignments were
+  never considered, but may break scripts running AdapterRemoval.
+- [**BREAKING**] Drop support for raw error-rates to `--trim-mott-rate`, which
+  was renamed to `--trim-mott-quality` to match other trimming options.
+- [**BREAKING**] SAM/BAM output is now combined into a single file by default,
+  including discarded reads. This can be overridden by setting the individual
+  `--out-*` options.
+- [**BREAKING**] Dropped `PG` tag from read-groups/records in SAM/BAM output.
+- [**BREAKING**] Dropped (minimal) read-groups for SAM/BAM output. If desired,
+  read-group information can be added with `--read-group`.
+- [**BREAKING**] The `--report-duplication` option now supports k/m/g suffixes,
+  and defaults to `100k` if used without an explicit value.
+- [**BREAKING**] The `--read-group` option no longer attempts to unescape
+  special characters. Instead, tags must be separated using embedded tabs
+  (`--read-group $'ID:A\tSM:B'`) or provided as individual arguments
+  (`--read-group 'ID:A' 'SM:B'`).
+- Improved checks for conflicting command-line options.
+- Barcodes are now recorded in FASTQ headers demultiplexing without trimming.
+- The `$schema` URL is now included in the JSON report
+- Makefile features are now enabled/disable with `true`/`false` instead of
+  `yes`/`no`.
+- Vega-lite is now loaded in the background, when opening the HTML reports,
+  making the report readable before Vega-lite has loaded.
+- Optimized alignments involving multiple possible adapter sequences, by only
+  once performing the alignments that involve no adapter sequences.
+- Optimized alignments involving multiple possible adapter sequences, by
+  sorting the list of adapter sequences by hits. This increasing the odds that
+  a good alignment is found early so that worse alignments can be skipped.
+- The old Makefile was replaced with the Meson build system, but a wrapper
+  Makefile is still provided/used as a convenience for setting the recommended
+  build options.
+- A number of small improvements were made to the `--help` text.
+- Improved error messages when mismatching (paired) read names are detected.
+- Singleton reads are now included in the overall summary statistics in
+  JSON/HTML reports.
+- Hardening flags are now enabled by default during compilation. This comes with
+  a small performance cost, but most distros are also expected to enable similar
+  flags by default.
+
+### Fixed
+
+- NA values were being written with '%' or 'bp' suffixes in HTML report.
+- Some plots were omitted for merged reads in HTML report.
+- Mate 2 adapters were reverse-complemented in JSON report when demultiplexing.
+- SAM/BAM headers were not being written in demultiplexing mode.
+- Mate 1/2 statistics were sampled independently, and thus potentially not
+  derived from the same read pairs.
+- The JSON/HTML reports would give different time-stamps for the run, since one
+  gave the start time and the other the end time. Now start time is always used.
+- Fixed failure when reading paired FASTQ files where read lengths differed
+  between the two files
+- Fixed report files and unidentified reads getting additional suffixes when
+  filenames were specified manually during demultiplexing.
+- Fixed `/dev/null` being listed as the path for some files when demultiplexing,
+  and these outputs were disabled.
+- Reverted the removal of support for '.' as equivalent to 'N' in FASTQ reads.
+  This is found in some older data-sets (#112).
+- Fixed misleading IO error messages, that would include descriptions of
+  unrelated errors in some cases.
+
 ## [2.3.4] - 2024-08-24
 
 This release adds a new couple of command-line options for handling non-ACGTN
@@ -151,6 +255,8 @@ Feedback is very welcome in the mean time.
   `--gzip` option when manually specifying output files.
 - Added options `--prefix-read1`, `--prefix-read2`, and `--prefix-merged` for
   adding custom prefixes to the names of FASTQ reads.
+- Added support for trimming poly-X tails. Trimming can be done for a single
+  nucleotide (e.g. poly-A) or for any combination of A, C, G, and T tails.
 
 ### Changed
 
@@ -166,9 +272,9 @@ Feedback is very welcome in the mean time.
   instead of `Q_match ~= Q_a + Q_b`, and that same-quality mismatches are
   assigned 'N' instead of one being picked at random. Motivated in part by
   `doi:10.1186/s12859-018-2579-2`. This can be changed using `--merge-strategy`.
-- The `--merge` option no longer has any effect when processing SE data;
-  previously this option would treat reads with at `--minalignmentlength`
-  adapter as pseudo-merged reads.
+- [**BREAKING**] The `--merge` option no longer has any effect when processing
+  SE data; previously this option would treat reads with at
+  `--minalignmentlength` adapter as pseudo-merged reads.
 - [**BREAKING**] Merged reads are no longer given a `M_` name prefix and merged
   reads that have been trimmed after merging are no longer given an `MT_` name
   prefix. Instead, see the new option `--prefix-merged`.
@@ -690,7 +796,7 @@ dramatic effects on the use of the program so please read these notes carefully
 ### Changed
 
 - Updated trimming of qualities.
-- The programs handles lower vs upper case issues by translating all sequences
+- The program handles lower vs upper case issues by translating all sequences
   to upper case.
 - The program now checks for inconsistent parameters.
 
@@ -702,6 +808,7 @@ dramatic effects on the use of the program so please read these notes carefully
 
 - Initial release
 
+[3.0.0-alpha3]: https://github.com/MikkelSchubert/adapterremoval/compare/3.0.0-alpha2...3.0.0-alpha3
 [2.3.4]: https://github.com/MikkelSchubert/adapterremoval/compare/v2.3.3...v2.3.4
 [3.0.0-alpha2]: https://github.com/MikkelSchubert/adapterremoval/compare/3.0.0-alpha1...3.0.0-alpha2
 [3.0.0-alpha1]: https://github.com/MikkelSchubert/adapterremoval/compare/v2.3.3...3.0.0-alpha1

README-v3.md

@@ -0,0 +1,56 @@
+# AdapterRemoval v3 - Dang fast(Q) processing
+
+[![build](https://github.com/MikkelSchubert/adapterremoval/actions/workflows/build-and-test.yaml/badge.svg)](https://github.com/MikkelSchubert/adapterremoval/actions/workflows/build-and-test.yml) [![docs](https://readthedocs.org/projects/adapterremoval/badge/?version=latest)](https://adapterremoval.readthedocs.io/) [![coverage](https://coveralls.io/repos/github/MikkelSchubert/adapterremoval/badge.svg?branch=master)](https://coveralls.io/github/MikkelSchubert/adapterremoval)
+
+AdapterRemoval trims adapter sequences and low quality bases from High-Throughput Sequencing (HTS) data in FASTQ format. For paired-end data, AdapterRemoval can merge overlapping paired-ended reads into (longer) consensus sequences. Additionally, AdapterRemoval can demultiplex FASTQ reads, and construct a consensus adapter sequence for paired-ended reads, if this information is not available.
+
+AdapterRemoval v3 is a major revision of AdapterRemoval v2 that aims at simplifying usage via sensible default settings. In addition, v3 adds support for detailed QC reports, trimming of poly-X tails and improved trimming of low quality bases, and greatly increased throughput. See below for more information.
+
+AdapterRemoval v3 is still a work in progress, but alpha release 3 is [available for download](https://github.com/MikkelSchubert/adapterremoval/releases/tag/v3.0.0-alpha3/). Documentation is available at [Read the Docs](https://adapterremoval.readthedocs.io/en/v3.0.0-alpha3/), including a guide on how to migrate from v2. For questions, bug reports, and/or suggestions, please use the [GitHub tracker](https://github.com/MikkelSchubert/adapterremoval/issues/).
+
+## Major features
+
+- Trimming of adapters sequences from single-end and paired-end FASTQ reads
+  - Trimming of multiple, different adapters or adapter pairs
+- Detailed [human-readable](https://mikkelschubert.github.io/adapterremoval/examples/3.0.0-alpha3.html) and [machine-readable](https://mikkelschubert.github.io/adapterremoval/examples/3.0.0-alpha3.json) QC reports **(v3)**
+  - The ability to perform QC-only runs with or without read processing **(v3)**
+- Barcode based demultiplexing with or without trimming of adapter sequences
+  - Support for samples identified by multiple barcode pairs **(v3)**
+  - Support for mixed orientation barcodes **(v3)**
+- Support for multiple methods for trimming low quality bases/reads
+  - Quality trimming using windows or constants thresholds
+  - Quality trimming using the modified Mott algorithm **(v3)**
+  - Poly-X tail trimming, supporting any combination of trailing bases **(v3)**
+- Filtering of reads based on complexity **(v3)**
+- Reconstruction of adapter sequences by pair-wise alignment of paired-end reads
+- Merging of overlapping read-pairs into higher-quality consensus sequences
+- Support for reading interleaved FASTQ files
+- Support for arbitrary splitting/interleaving of output files **(v3)**
+- Support for writing BGZF compressed SAM and BAM files **(v3)**
+- Support for SSE2, AVX2, AVX512, and NEON accelerated alignments **(v3)**
+
+## Performance
+
+AdapterRemoval v3 features greatly increased throughput compared to AdapterRemoval v2. This is accomplished through support for additional SIMD instruction sets, improved parallelization of I/O and computationally expensive tasks, including block based compression of output files, as well as defaulting to a lower compression level.
+
+Please note that these results are preliminary:
+
+![Throughput for ARv2, ARv3, and fastp](https://raw.githubusercontent.com/MikkelSchubert/adapterremoval/master/docs/images/throughput.svg)
+
+Point labels indicate the number of worker threads configured for each program, while the X-axis indicates observed CPU-usage for a given number of worker threads. The Y-axis indicates millions of 150bp paired-end reads processed per second, for gzipped input and output, with merging enabled and duplication estimation disabled.
+
+Benchmarking was performed on an Intel i9-11900K with 8 physical cores, and plotting is therefore limited to ~8 CPUs. The CPU usage of fastp being higher than the number of worker threads, is due to additional threads being used (compressed) I/O. ARv2 does not scale past 4 threads.
+
+## Documentation
+
+For a detailed description of program installation and usage, please refer to the [online documentation](https://adapterremoval.readthedocs.io/en/latest). A summary of command-line options may also be found in the [manual page](https://adapterremoval.readthedocs.io/en/latest/manpage.html), accessible via the command `man adapterremoval3` once AdapterRemoval has been installed.
+
+## Citation
+
+If you use AdapterRemoval v3, then please cite the paper::
+
+    Schubert, Lindgreen, and Orlando (2016). AdapterRemoval v2: rapid adapter trimming, identification, and read merging. BMC Research Notes, 12;9(1):88 <http://bmcresnotes.biomedcentral.com/articles/10.1186/s13104-016-1900-2>
+
+AdapterRemoval was originally published in Lindgreen 2012:
+
+    Lindgreen (2012): AdapterRemoval: Easy Cleaning of Next Generation Sequencing Reads, BMC Research Notes, 5:337 <http://www.biomedcentral.com/1756-0500/5/337/>