Merge branch 'ar/changelog-051rc1' into 'master'

Prepare docs 0.5.1-rc1

See merge request machine-learning/modkit!306

Author: ArtRand

CHANGELOG.md

@@ -4,6 +4,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [v0.5.1-rc1]
+### Fixes
+- [pileup] Fixes bug where N_diff was incorrectly summed when `--combine-strands` was used. (N.B. %-methylation was still correct).
+- Fixes bug where fallback threshold was incorrect. Will now be highest observed explicit canonical probability, was highest bin.
+- Fixes conversion to bigWig when chromosomes aren't sorted exactly the way that `sort` would do it.
+### Adds
+- [cli] Adds shell completions command, thanks to @killercup
+- [dmr] Adds 2OMe mod codes
+- [validate] Adds better logging when NM tag is missing or reads fail.
+- Allows bgzip-compressed FASTA references across all commands
+- [bedmethyl, tobigwig] Allow using header from modBAM instead of sizes.
+
 ## [v0.5.0]
 ### Adds
 - [open-chromatin] Adds open chromatin prediction subcommand for 6mA MTase-treated DNA

book/src/advanced_usage.md

@@ -2698,52 +2698,82 @@ Logging Options:
 Make a BigWig track from a bedMethyl file or stream. For details on the BigWig
 format see https://doi.org/10.1093/bioinformatics/btq351
 
-Usage: modkit bedmethyl tobigwig [OPTIONS] --sizes <CHROMSIZES> --mod-codes <MOD_CODES> <IN_BEDMETHYL> <OUT_FP>
+Usage: modkit bedmethyl tobigwig [OPTIONS] --mod-codes <MOD_CODES> <--sizes <CHROMSIZES>|--header <INPUT_BAM>> <IN_BEDMETHYL> <OUT_FP>
 
 Arguments:
-  <IN_BEDMETHYL>  Input bedmethyl, uncompressed, "-" or "stdin" indicates an
-                  input stream
-  <OUT_FP>        Output bigWig filename
+  <IN_BEDMETHYL>
+          Input bedmethyl, uncompressed, "-" or "stdin" indicates an input
+          stream
+
+  <OUT_FP>
+          Output bigWig filename
 
 Options:
-  -g, --sizes <CHROMSIZES>     A chromosome sizes file. Each line should be have
-                               a chromosome and its size in bases, separated by
-                               whitespace. A fasta index (.fai) works as well
-  -m, --mod-codes <MOD_CODES>  Make a bigWig track where the values are the
-                               percent of bases with this modification, use
-                               multiple comma-separated codes to combine counts.
-                               For example --mod-code m makes a track of the 5mC
-                               percentages and --mod-codes h,m will make a track
-                               of the combined counts from 5hmC and 5mC.
-                               Combining counts for different primary bases will
-                               cause an error (e.g. --mod-codes a,h)
-  -h, --help                   Print help
+  -g, --sizes <CHROMSIZES>
+          A chromosome sizes file. Each line should be a chromosome and its size
+          in bases, separated by whitespace. A fasta index (.fai) works as well.
+          Use instead of the bam header
+
+  -b, --header <INPUT_BAM>
+          modBAM from which the pileup was generated. Chromosome sizes are
+          gathered from the header. Use instead of the chromosome sizes file
+
+  -m, --mod-codes <MOD_CODES>
+          Make a bigWig track where the values are the percent of bases with
+          this modification, use multiple comma-separated codes to combine
+          counts. For example --mod-code m makes a track of the 5mC percentages
+          and --mod-codes h,m will make a track of the combined counts from 5hmC
+          and 5mC. Combining counts for different primary bases will cause an
+          error (e.g. --mod-codes a,h will error)
+
+  -h, --help
+          Print help (see a summary with '-h')
 
 Output Options:
       --negative-strand-values
           Report the percentages on the negative strand as negative values. The
           data range will be [-100, 100]
+
   -z, --nzooms <NZOOMS>
-          Set the maximum of zooms to create [default: 10]
+          Set the maximum of zooms to create
+          
+          [default: 10]
+
       --zooms <ZOOMS>...
           Set the zoom resolutions to use (overrides the --nzooms argument)
+
   -u, --uncompressed
           Don't use compression
+
       --block-size <BLOCK_SIZE>
-          Number of items to bundle in r-tree [default: 256]
+          Number of items to bundle in r-tree
+          
+          [default: 256]
+
       --items-per-slot <ITEMS_PER_SLOT>
-          Number of data points bundled at lowest level [default: 1024]
+          Number of data points bundled at lowest level
+          
+          [default: 1024]
 
 Compute Options:
-  -t, --nthreads <NTHREADS>  Set the number of threads to use. This tool will
-                             typically use ~225% CPU on a HDD. SDDs may be
-                             higher. (IO bound) [default: 6]
-      --inmemory             Do not create temporary files for intermediate data
+  -t, --nthreads <NTHREADS>
+          Set the number of threads to use. This tool will typically use ~225%
+          CPU on a HDD. SDDs may be higher. (IO bound)
+          
+          [default: 6]
+
+      --inmemory
+          Do not create temporary files for intermediate data
+
+      --force-chromosome-ordering
+          If input bedMethyl has sorting of the same scheme as `sort`, this
+          option may speed up conversion
 
 Logging Options:
       --log-filepath <LOG_FILEPATH>
           Specify a file for debug logs to be written to, otherwise ignore them.
           Setting a file is recommended. (alias: log)
+
       --suppress-progress
           Hide the progress bar
 ```

docs/advanced_usage.html

@@ -392,7 +392,7 @@ <h2 id="pileup"><a class="header" href="#pileup">pileup</a></h2>
           --filter-threshold 0.9 will specify a threshold value of 0.70 for
           adenine and 0.9 for all other base modification calls
 
-      --mod-thresholds &lt;MOD_THRESHOLDS&gt;
+      --mod-threshold &lt;MOD_THRESHOLDS&gt;
           Specify a passing threshold to use for a base modification,
           independent of the threshold for the primary sequence base or the
           default. For example, to set the pass threshold for 5hmC to 0.8 use
@@ -585,7 +585,7 @@ <h2 id="adjust-mods"><a class="header" href="#adjust-mods">adjust-mods</a></h2>
       --filter-probs
           Filter out the lowest confidence base modification probabilities
 
-      --only-mapped
+      --mapped-only
           Only use base modification probabilities from bases that are aligned
           when estimating the filter threshold (i.e. ignore soft-clipped, and
           inserted bases)
@@ -787,7 +787,7 @@ <h2 id="sample-probs"><a class="header" href="#sample-probs">sample-probs</a></h
           Only sample base modification probabilities that are aligned to the
           positions in this BED file. (alias: include-positions)
 
-      --only-mapped
+      --mapped-only
           Only use base modification probabilities that are aligned (i.e. ignore
           soft-clipped, and inserted bases)
 
@@ -903,7 +903,7 @@ <h2 id="summary"><a class="header" href="#summary">summary</a></h2>
           --filter-threshold 0.9 will specify a threshold value of 0.70 for
           adenine and 0.9 for all other base modification calls
 
-      --mod-thresholds &lt;MOD_THRESHOLDS&gt;
+      --mod-threshold &lt;MOD_THRESHOLDS&gt;
           Specify a passing threshold to use for a base modification,
           independent of the threshold for the primary sequence base or the
           default. For example, to set the pass threshold for 5hmC to 0.8 use
@@ -940,7 +940,7 @@ <h2 id="summary"><a class="header" href="#summary">summary</a></h2>
           Only summarize base modification probabilities that are aligned to the
           positions in this BED file. (alias: include-positions)
 
-      --only-mapped
+      --mapped-only
           Only use base modification probabilities that are aligned (i.e. ignore
           soft-clipped, and inserted bases)
 
@@ -1375,7 +1375,7 @@ <h2 id="pileup-hemi"><a class="header" href="#pileup-hemi">pileup-hemi</a></h2>
           --filter-threshold 0.9 will specify a threshold value of 0.70 for
           adenine and 0.9 for all other base modification calls
 
-      --mod-thresholds &lt;MOD_THRESHOLDS&gt;
+      --mod-threshold &lt;MOD_THRESHOLDS&gt;
           Specify a passing threshold to use for a base modification,
           independent of the threshold for the primary sequence base or the
           default. For example, to set the pass threshold for 5hmC to 0.8 use
@@ -1513,10 +1513,10 @@ <h2 id="entropy"><a class="header" href="#entropy">entropy</a></h2>
       --filter-threshold &lt;FILTER_THRESHOLD&gt;
           Specify the filter threshold globally or for the canonical calls. When
           specified, base modification call probabilities will be required to be
-          greater than or equal to this number. If `--mod-thresholds` is also
+          greater than or equal to this number. If `--mod-threshold` is also
           specified, _this_ value will be used for canonical calls
 
-      --mod-thresholds &lt;MOD_THRESHOLDS&gt;
+      --mod-threshold &lt;MOD_THRESHOLDS&gt;
           Specify a passing threshold to use for a base modification,
           independent of the threshold for the primary sequence base or the
           default. For example, to set the pass threshold for 5hmC to 0.8 use
@@ -2075,7 +2075,7 @@ <h2 id="extract-calls"><a class="header" href="#extract-calls">extract calls</a>
           --filter-threshold 0.9 will specify a threshold value of 0.70 for
           adenine and 0.9 for all other base modification calls
 
-      --mod-thresholds &lt;MOD_THRESHOLDS&gt;
+      --mod-threshold &lt;MOD_THRESHOLDS&gt;
           Specify a passing threshold to use for a base modification,
           independent of the threshold for the primary sequence base or the
           default. For example, to set the pass threshold for 5hmC to 0.8 use
@@ -2827,52 +2827,82 @@ <h2 id="bedmethyl-tobigwig"><a class="header" href="#bedmethyl-tobigwig">bedmeth
 <pre><code class="language-text">Make a BigWig track from a bedMethyl file or stream. For details on the BigWig
 format see https://doi.org/10.1093/bioinformatics/btq351
 
-Usage: modkit bedmethyl tobigwig [OPTIONS] --sizes &lt;CHROMSIZES&gt; --mod-codes &lt;MOD_CODES&gt; &lt;IN_BEDMETHYL&gt; &lt;OUT_FP&gt;
+Usage: modkit bedmethyl tobigwig [OPTIONS] --mod-codes &lt;MOD_CODES&gt; &lt;--sizes &lt;CHROMSIZES&gt;|--header &lt;INPUT_BAM&gt;&gt; &lt;IN_BEDMETHYL&gt; &lt;OUT_FP&gt;
 
 Arguments:
-  &lt;IN_BEDMETHYL&gt;  Input bedmethyl, uncompressed, "-" or "stdin" indicates an
-                  input stream
-  &lt;OUT_FP&gt;        Output bigWig filename
+  &lt;IN_BEDMETHYL&gt;
+          Input bedmethyl, uncompressed, "-" or "stdin" indicates an input
+          stream
+
+  &lt;OUT_FP&gt;
+          Output bigWig filename
 
 Options:
-  -g, --sizes &lt;CHROMSIZES&gt;     A chromosome sizes file. Each line should be have
-                               a chromosome and its size in bases, separated by
-                               whitespace. A fasta index (.fai) works as well
-  -m, --mod-codes &lt;MOD_CODES&gt;  Make a bigWig track where the values are the
-                               percent of bases with this modification, use
-                               multiple comma-separated codes to combine counts.
-                               For example --mod-code m makes a track of the 5mC
-                               percentages and --mod-codes h,m will make a track
-                               of the combined counts from 5hmC and 5mC.
-                               Combining counts for different primary bases will
-                               cause an error (e.g. --mod-codes a,h)
-  -h, --help                   Print help
+  -g, --sizes &lt;CHROMSIZES&gt;
+          A chromosome sizes file. Each line should be a chromosome and its size
+          in bases, separated by whitespace. A fasta index (.fai) works as well.
+          Use instead of the bam header
+
+  -b, --header &lt;INPUT_BAM&gt;
+          modBAM from which the pileup was generated. Chromosome sizes are
+          gathered from the header. Use instead of the chromosome sizes file
+
+  -m, --mod-codes &lt;MOD_CODES&gt;
+          Make a bigWig track where the values are the percent of bases with
+          this modification, use multiple comma-separated codes to combine
+          counts. For example --mod-code m makes a track of the 5mC percentages
+          and --mod-codes h,m will make a track of the combined counts from 5hmC
+          and 5mC. Combining counts for different primary bases will cause an
+          error (e.g. --mod-codes a,h will error)
+
+  -h, --help
+          Print help (see a summary with '-h')
 
 Output Options:
       --negative-strand-values
           Report the percentages on the negative strand as negative values. The
           data range will be [-100, 100]
+
   -z, --nzooms &lt;NZOOMS&gt;
-          Set the maximum of zooms to create [default: 10]
+          Set the maximum of zooms to create
+          
+          [default: 10]
+
       --zooms &lt;ZOOMS&gt;...
           Set the zoom resolutions to use (overrides the --nzooms argument)
+
   -u, --uncompressed
           Don't use compression
+
       --block-size &lt;BLOCK_SIZE&gt;
-          Number of items to bundle in r-tree [default: 256]
+          Number of items to bundle in r-tree
+          
+          [default: 256]
+
       --items-per-slot &lt;ITEMS_PER_SLOT&gt;
-          Number of data points bundled at lowest level [default: 1024]
+          Number of data points bundled at lowest level
+          
+          [default: 1024]
 
 Compute Options:
-  -t, --nthreads &lt;NTHREADS&gt;  Set the number of threads to use. This tool will
-                             typically use ~225% CPU on a HDD. SDDs may be
-                             higher. (IO bound) [default: 6]
-      --inmemory             Do not create temporary files for intermediate data
+  -t, --nthreads &lt;NTHREADS&gt;
+          Set the number of threads to use. This tool will typically use ~225%
+          CPU on a HDD. SDDs may be higher. (IO bound)
+          
+          [default: 6]
+
+      --inmemory
+          Do not create temporary files for intermediate data
+
+      --force-chromosome-ordering
+          If input bedMethyl has sorting of the same scheme as `sort`, this
+          option may speed up conversion
 
 Logging Options:
       --log-filepath &lt;LOG_FILEPATH&gt;
           Specify a file for debug logs to be written to, otherwise ignore them.
           Setting a file is recommended. (alias: log)
+
       --suppress-progress
           Hide the progress bar
 </code></pre>
@@ -2939,7 +2969,7 @@ <h2 id="modbam-check-tags"><a class="header" href="#modbam-check-tags">modbam ch
           Check tags on non-primary alignments as well. Keep in mind this may
           incur a double-counting of the read with its primary mapping
 
-      --only-mapped
+      --mapped-only
           Only check alignments that are mapped
 
       --region &lt;REGION&gt;

docs/filtering.html

@@ -183,7 +183,7 @@ <h1 id="partitioning-pass-and-fail-base-modification-calls"><a class="header" hr
 For example to set a threshold for cytosine modifications at 0.8 and adenine modifications at 0.9 provide <code>--filter-threshold C:0.8 --filter-threshold A:0.9</code>.
 Pass threshold values per base modification can also be specified.
 For example, to specify a threshold for canonical adenine at 0.8 and 6mA at 0.9 use <code>--filter-threshold A:0.8 --mod-thresholds a:0.9</code>.
-Or to specify a threshold of 0.8 for 5mC, 0.9 for 5hmC, and 0.85 for canonical cytosine: <code>--filter-threshold C:0.85 --mod-thresholds m:0.8 --mod-thresholds h:0.9</code></p>
+Or to specify a threshold of 0.8 for 5mC, 0.9 for 5hmC, and 0.85 for canonical cytosine: <code>--filter-threshold C:0.85 --mod-threshold m:0.8 --mod-threshold h:0.9</code></p>
 <p>Keep in mind that the <code>--mod-threshold</code> option will treat <code>A</code>, <code>C</code>, <code>G</code>, and <code>T</code> and "any-mod" as per the <a href="https://samtools.github.io/hts-specs/SAMtags.pdf">specification</a>.</p>
 <h2 id="further-details"><a class="header" href="#further-details">Further details</a></h2>
 <ol>