Upgrade to py3: version 1.0 deprecation of py2.7 (#180)

* 🎨 Apply formatting on pipeline and validation directories with 2to3
🎨 Apply black formatting on 2to3 output
🔥 Remove unused imports in recursive_dbscan.py
⬆️ Add requirements.txt for easier installation using conda create -n autometa -c conda-forge -c bioconda --file=requirements.txt

* 🎨 Apply formatting on pipeline and validation directories with 2to3
🎨 Apply black formatting on 2to3 output
🔥 Remove unused imports in recursive_dbscan.py

* 🐳 Update Dockerfile for py3 installation
🔥 Remove unused imports in ML_recruitment.py

* 🎨 Add EOF line to Dockerfile

* 🔥 Remove unused import from run_autometa.py
🐳🎨 hmmpress markers databases

* 📝 Update installation instructions

* Reverse compatibility note for py2.7 and py3

* 🎨📝 Reformatting headers in installation

* 🔥 Remove typo

* 🎨🐛 Allow for gzipped prot.accession2taxid when checking databases

* 🐛 Chain decode to subprocess.check_output(...) when checking downloaded md5

* 🎨 Add text=True param to subprocess.check_output(...) calls
🎨 run 2to3 and black on lca_functions.pyx

* 🎨 Add subprocess.check_output(..., text=True) to validation dir scripts

* 🐛 Add `db_dir_path` param to run_blast2lca(...) in make_taxonomy_table.py

* 🐳 Add pipeline and validation dirs to /Users/rees/miniconda3/bin:/Users/rees/miniconda3/condabin:/usr/local/opt/mysql-client/bin:/Users/rees/bin:/usr/local/bin:/usr/local/Cellar/pyenv-virtualenv/HEAD-5419dc7/shims:/Users/rees/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/Applications/Postgres.app/Contents/Versions/latest/bin

* 🐳🎨 Add EOF char to Dockerfile

* 🐳🔥 Remove apt-get comment
🐳📝 Update copyright to 2021 info

* 🐛 Add encoding instance check and convert bytes objects to str in init_logger(...)

* 🎨🐛🔥 Change length_trim(...) func to handle gzipped fastas using SeqIO rather than perl script
🔥 Deprecated fasta_length_trim.pl from run_autometa.py

* 🔥 Remove logger emit message (logging is not imported in run_autometa_docker.py)

* 🎨 black formatting

* Update README.md

Co-authored-by: Jason Kwan <[email protected]>

* :art add /output/ prefix to coverage table output in make_taxonomy_table_docker.py
🎨 reformat some commands with f-strings

* 🎨 change shebangs from python2.7 to python
🎨 add database check if taxonomy table is specified in run_autometa_docker

* :memo Add note on python versions in README
📝🐛 move change directory command above hmmpress in environment build example in README

* 🐛🐳 Fix apt-get update error during docker build

Error message:

InRelease' changed its 'Suite' value from 'stable-updates' to 'oldstable-updates'

Suggested fix was found here: https://vitux.com/fixed-n-repository-http-security-debian-org-buster-updates-inrelease-changed-its-version-value-from-to-10/

- 📝 Add docker build command to readme
- 📝 Add 🐳 to git commit style messages in CONTRIBUTING

* 🎨 Add implementation to handle gzipped assemblies in make_taxonomy_table.py

* 🎨🐛 Fix path incorrect file path handling when gzipped assembly is provided to length_filter(...) causing diamond blastp to use wrong filepath as query

* 🎨🐛 Fix incorrect path handling where .filtered.orfs gets appended twice and is passed in wrong to run_diamond

* 🎨 Reformatting some print statements with f-strings for ease of reading

* 🔥🐛 Remove hardcoded extension in orfs_fpath param for run_diamond(...) func in make_taxonomy_table.py

* 📝 Update readme to fix markdown-lint warnings and add `--db_dir` parameter to example usage

* 📝 replace databases path to new GH repo name

NOTE: This assumes Autometa was installed in the user\'s home directory

* 📝 Replace any filepath instance of autometa with Autometa and \~ with $HOME

* Update pipeline/make_taxonomy_table.py

Co-authored-by: Jason Kwan <[email protected]>

Author: evanroyrees

.github/CONTRIBUTING.md

@@ -85,6 +85,7 @@ While the above must be satisfied, additional prerequisites may be present, depe
     * :white_check_mark: `:white_check_mark:` when adding tests
     * :arrow_up: `:arrow_up:` when upgrading dependencies
     * :arrow_down: `:arrow_down:` when downgrading dependencies
+    * :whale: `:whale:` when fixing the docker build
 
 ### Documentation Style Guide
 

Dockerfile

@@ -1,7 +1,8 @@
-FROM continuumio/anaconda
+FROM continuumio/miniconda3
 LABEL maintainer="[email protected]"
 
-# Copyright 2018 Ian J. Miller, Evan Rees, Izaak Miller, Jason C. Kwan
+# Copyright 2021 Copyright 2021 Ian J. Miller, Evan R. Rees, Siddharth Uppal,
+# Chase Clark, Andrew Lail, Kyle Wolf, Shaurya Chanana, Izaak Miller, Jason C. Kwan
 #
 # This file is part of Autometa.
 #
@@ -18,16 +19,31 @@ LABEL maintainer="[email protected]"
 # You should have received a copy of the GNU Affero General Public License
 # along with Autometa. If not, see <http://www.gnu.org/licenses/>.
 
-SHELL ["/bin/bash", "-c"]
-ENV PATH="/opt/conda/bin:$PATH"
-RUN apt-get update
-RUN apt-get install -y prodigal hmmer build-essential zlib1g-dev bowtie2 bedtools libatlas-base-dev libncurses5-dev libncursesw5-dev libbz2-dev liblzma-dev
-RUN conda install -y tqdm joblib biopython
-RUN mkdir diamond && cd diamond && wget http://github.com/bbuchfink/diamond/releases/download/v0.9.14/diamond-linux64.tar.gz && tar xvf diamond-linux64.tar.gz
-RUN wget https://github.com/samtools/samtools/releases/download/1.6/samtools-1.6.tar.bz2
-RUN tar -vxjf samtools-1.6.tar.bz2
-RUN cd samtools-1.6 && ./configure --prefix=/samtools && make && make install
-RUN git clone https://github.com/danielfrg/tsne.git && cd tsne && python setup.py install
-RUN git clone https://github.com/KwanLab/Autometa.git && cd Autometa/pipeline && python setup_lca_functions.py build_ext --inplace
+RUN apt-get update --allow-releaseinfo-change \
+    && apt-get install -y build-essential \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 
-ENV PATH="/diamond:/Autometa/pipeline:/samtools/bin:${PATH}"
+COPY requirements.txt ./
+RUN conda install -c bioconda -c conda-forge python=3.7 --file=requirements.txt \
+    && conda clean --all -y
+
+COPY . ./
+RUN cd pipeline \
+    && python setup_lca_functions.py build_ext --inplace \
+    && cd - \
+    && hmmpress -f single-copy_markers/Bacteria_single_copy.hmm \
+    && hmmpress -f single-copy_markers/Archaea_single_copy.hmm
+
+ENV PATH="/pipeline:/validation:${PATH}"
+# Test pipeline entrypoints
+RUN recursive_dbscan.py -h \
+    && calculate_read_coverage.py -h \
+    && run_autometa.py -h \
+    && make_contig_table.py -h \
+    && make_marker_table.py -h \
+    && cluster_taxonomy.py -h \
+    && lca.py -h \
+    && cluster_process.py -h \
+    && make_taxonomy_table.py -h \
+    && add_contig_taxonomy.py &> /dev/null