Merge branch 'dev'

Author: hasindu2008

README.md

@@ -30,7 +30,6 @@ You can also use conda to install *slow5tools* as `conda install slow5tools -c b
 
 ### Building a release
 
-
 Users are recommended to build from the  [latest release](https://github.com/hasindu2008/slow5tools/releases) tar ball.
 
 Quick example for Ubuntu :
@@ -70,7 +69,7 @@ make
 
 ### Other building options
 
-- You can optionally enable [*zstd* compression](https://facebook.github.io/zstd) support when building *slow5lib* by invoking `make zstd=1`. This requires __zstd 1.3 or higher development libraries__ installed on your system (*libzstd1-dev* package for *apt*, *libzstd-devel* for *yum/dnf* and *zstd* for *homebrew*). SLOW5 files compressed with *zstd* offer slightly smaller file size and better performance compared to the default *zlib*. However, *zlib* runtime library is available by default on almost all distributions unlike *zstd* and thus files compressed with *zlib* will be more 'portable' (also see [notes](https://github.com/hasindu2008/slow5tools#notes)).
+- You can optionally enable [*zstd* compression](https://facebook.github.io/zstd) support when building *slow5lib* by invoking `make zstd=1`. This requires __zstd 1.3 or higher development libraries__ installed on your system (*libzstd1-dev* package for *apt*, *libzstd-devel* for *yum/dnf* and *zstd* for *homebrew*). SLOW5 files compressed with *zstd* offer smaller file size and better performance compared to the default *zlib*. However, *zlib* runtime library is available by default on almost all distributions unlike *zstd* and thus files compressed with *zlib* will be more 'portable' (also see [notes](https://github.com/hasindu2008/slow5tools#notes)).
 
 - *slow5tools* from version 0.3.0 onwards by default requires vector instructions (SSSE3 or higher for Intel/AMD and neon for ARM). If your processor is an ancient processor with no such vector instructions, invoke make as `make no_simd=1`.
 
@@ -85,11 +84,17 @@ make
     Similarly, to locally build *zstd* and link against that:
 
     ```
-    scripts/install-zstd.sh        # download and compiles HDF5 in the current folder
+    scripts/install-zstd.sh        # download and compiles zstd in the current folder
     ./configure --enable-localzstd
-    make
+    make			# don't run make zstd=1. libzstd.a is statically linked this time.
     ```
 
+- On Mac M1 or in any system if `./configure` cannot find the hdf5 libraries installed through the package manager, you can specify the location as *LDFLAGS=-L/path/to/shared/lib/ CPPFLAGS=-I/path/to/headers/*. For example on Mac M1:
+	```
+	./configure LDFLAGS=-L/opt/homebrew/lib/ CPPFLAGS=-I/opt/homebrew/include/
+	make
+	```
+
 - You can build a docker image as follows.
 	```
 	git clone https://github.com/hasindu2008/slow5tools && cd slow5tools
@@ -99,7 +104,7 @@ make
 
 ## Usage
 
-Visit the [man page](https://hasindu2008.github.io/slow5tools/commands.html) for all the commands and options.
+Visit the [man page](https://hasindu2008.github.io/slow5tools/commands.html) for all the commands and options. See [here](https://hasindu2008.github.io/slow5tools/oneliners.html) for example bash one-liners with slow5tools. A guide on using BLOW5 for archiving and steps to verify if data integrity is preserved is [here](https://hasindu2008.github.io/slow5tools/archive.html). A script for performing real-time FAST5 to BLOW5 conversion during sequencing is provided [here](https://github.com/hasindu2008/slow5tools/tree/master/scripts/realtime-f2s).
 
 ### Examples
 
@@ -141,6 +146,24 @@ slow5tools s2f blow5_dir -d fast5
 
 Visit [here](https://hasindu2008.github.io/slow5tools/workflows.html) for example workflows.
 
+
+### Troubleshooting/Questions
+
+Visit the [frequently asked questions](https://hasindu2008.github.io/slow5tools/faq.html) or open an [issue](https://github.com/hasindu2008/slow5tools/issues).
+
+
+### Upcoming features and optimisations
+
+Following are some features and optimisations in our todo list which will be implemented based on the need. If anyone is interested please request [here](https://github.com/hasindu2008/slow5tools/issues). Contributions are welcome.
+
+- pipelining input, processing and output in *merge, get, etc.* (expected runtime improvement upto 2X)
+- reading from stdin for *view*
+- binary releases for ARM64 processors on Linux
+- binary releases for MacOS
+- decoupling conversion modules (currently f2s and s2f; any future formats) so that slow5tools only deal with S/BLOW5 files and thus can be easily compiled. Currently, compiling slow5tools is not straight forward due to the HDF5 (FAST5) dependency
+- any other features that are potentially useful to many
+
+
 ### Notes
 
 *slow5lib* from version 0.3.0 onwards has built in [StreamVByte](https://github.com/lemire/streamvbyte) compression support to enable even smaller file sizes, which is applied to the raw signal by default when producing BLOW5 files.  *zlib* compression is then applied by default to each record. If *zstd* is used instead of *zlib* on top of *StreamVByte*, it is similar to ONT's latest [vbz](https://github.com/nanoporetech/vbz_compression) compression. BLOW5 files compressed with *zstd+StreamVByte* are still significantly smaller than vbz compressed FAST5 files.

docs/getting_started.md

@@ -84,6 +84,12 @@ make
     ./configure --enable-localzstd
     make
     ```
+	
+- On Mac M1 or in any system if `./configure` cannot find the hdf5 libraries installed through the package manager, you can specify the location as *LDFLAGS=-L/path/to/shared/lib/ CPPFLAGS=-I/path/to/headers/*. For example on Mac M1:
+	```
+	./configure LDFLAGS=-L/opt/homebrew/lib/ CPPFLAGS=-I/opt/homebrew/include/
+	make
+
 
 - You can build a docker image as follows.
 	```

docs/workflows.md

@@ -33,7 +33,7 @@ f5c eventalign -t 8 -r reads.fq -g ref.fa -b reads.bam --slow5 signals.blow5 > m
 
 ## Nanopolish
 
-[Nanopolish](https://github.com/jts/nanopolish) master branch now supports SLOW5 file format.
+[Nanopolish](https://github.com/jts/nanopolish) version 0.14.0 onwards supports SLOW5 file format.
 
 ```bash
 #convert fast5 files to slow5 files using 8 I/O processes
@@ -57,3 +57,10 @@ slow5tools f2s fast5_dir -d blow5_dir -p 8
 # run sigmap
 ./sigmap -m -r ref.fa -p <model> -x index -s blow5_dir -o mapping.paf -t 8
 ```
+
+## Bonito
+
+SLOW5 support for ONT's Bonito basecaller is now available as a [pull request](https://github.com/nanoporetech/bonito/pull/252) along with usage instructions and benchmarks.
+
+
+

scripts/realtime-f2s/pipeline.sh

@@ -12,6 +12,9 @@ die() {
 
 LOG=start_end_trace.log
 
+MAX_PROC=$(nproc)
+MAX_PROC=$(echo "${MAX_PROC}/2" | bc)
+
 ## Handle flags
 while getopts "d:l:f:" o; do
     case "${o}" in
@@ -35,10 +38,12 @@ shift $((OPTIND-1))
 
 $SLOW5TOOLS --version &> /dev/null || die "[pipeline.sh] slow5tools not found in path. Exiting."
 
+echo "[pipeline.sh] Starting pipeline with $MAX_PROC max processes"
 #test -e ${LOG}  && rm ${LOG}
-
+counter=0
 while read FILE
 do
+(
     F5_FILEPATH=$FILE # first argument
     F5_DIR=${F5_FILEPATH%/*} # strip filename from .fast5 filepath
     PARENT_DIR=${F5_DIR%/*} # get folder one heirarchy higher
@@ -86,5 +91,12 @@ do
 
     echo "[pipeline.sh] $F5_FILEPATH" >> $TMP_FILE
     echo -e "${F5_FILEPATH}\t${SLOW5_FILEPATH}\t${START_TIME}\t${END_TIME}" >> ${LOG}
-
+)&
+    ((counter++))
+    if [ $counter -ge $MAX_PROC ]; then
+        echo "[pipeline.sh] Waiting for $counter jobs to finish."
+        wait
+        counter=0
+    fi
 done
+wait

slow5lib

@@ -1,7 +1,7 @@
 #ifndef CMD_H
 #define CMD_H
 
-#define SLOW5TOOLS_VERSION "0.4.0"
+#define SLOW5TOOLS_VERSION "0.4.0-dirty"
 
 #define DEFAULT_NUM_THREADS 8
 #define DEFAULT_NUM_PROCESSES 8

src/cmd.h

@@ -124,10 +124,6 @@ int get_main(int argc, char **argv, struct program_meta *meta) {
     opt_t user_opts;
     init_opt(&user_opts);
 
-    // Default options
-    int32_t num_threads = DEFAULT_NUM_THREADS;
-    int64_t read_id_batch_capacity = DEFAULT_BATCH_SIZE;
-
     // Input arguments
     char* read_list_file_in = NULL;
 
@@ -272,7 +268,7 @@ int get_main(int argc, char **argv, struct program_meta *meta) {
 
         // Setup multithreading structures
         core_t core;
-        core.num_thread = num_threads;
+        core.num_thread = user_opts.num_threads;
         core.fp = slow5file;
         core.format_out = user_opts.fmt_out;
         core.press_method = press_out;
@@ -287,7 +283,7 @@ int get_main(int argc, char **argv, struct program_meta *meta) {
         bool end_of_file = false;
         while (!end_of_file) {
             int64_t num_ids = 0;
-            while (num_ids < read_id_batch_capacity) {
+            while (num_ids < user_opts.read_id_batch_capacity) {
                 char *buf = NULL;
                 size_t cap_buf = 0;
                 ssize_t nread;

src/get.c

@@ -146,7 +146,7 @@ int view_main(int argc, char **argv, struct program_meta *meta) {
         EXIT_MSG(EXIT_FAILURE, argv, meta);
         return EXIT_FAILURE;
     } else if (optind != argc - 1) { // TODO handle more than 1 file?
-        ERROR(">1 input file%s", "");
+        ERROR("more than 1 input file is given%s", "");
         fprintf(stderr, HELP_SMALL_MSG, argv[0]);
         EXIT_MSG(EXIT_FAILURE, argv, meta);
         return EXIT_FAILURE;

src/view.c

@@ -1 +1,2 @@
 *.idx
+example_multi_rg_v0.2.0_none_none.blow5