Feat rule based fragmenter (computational-metabolomics#17)

* Compatibility with conda version of geng; remove geng tool from package

* Incorporate pkl files into connectivity database

* Add nauty as dependency

* Add pickle as test dependency

* Switch from strings to pickles for connectivity graphs

* Use blob instead of text to store pickled dictionary

* No longer write substructures to .smi

* Add option to build to select only frequent substructures

* Add connectivity filter to k_configs

* Incorporate connectivity filter into MSn build method

* Build substructures for each set of masses independently

* Call itertools.product on substructures within multiprocessing portion of build

* Configure run script for current create_isomorphism_database inputs

* Built subsets should be empty list, not None

* Update variable names, remove debug options, update docstrings

* Add annotate_msn and generate_structures user functions

* Move stage at which multiprocessing step is performed

* Allow for multiple output options in build

* Remove ppm option for retrieving elemental composition from substructure db

* Allow list of mc/exact_mass to be passed to generate_structures

* Use TemporaryDirectory to store unittest results

* Let generate_structures return/yield smiles

* Implement build_msn to incorporate considerations for building structures from MS/MS

* Implement annotate_msn to provide an interface to build_msn

* Add/update build docstrings

* Remove unnecessary build parameters

* Pass data dictionary to user-facing build functions rather than separate mc, exact_mass, MSn masses

* Update variable naming conventions

* Add newline between smiles in out file

* Update SubstructureDb for removal of .pkl files

* Add function create_substructure_database

* Bring tests up to date with variable renaming

* Bring scripts up to date with variable renaming

* Simplify loading of test data and remove teardown

* Remove unused class ConnectivityDb and update SubstructureDb parameters

* Implement additional non-msn build tests

* Improve temporary table cleaning logic

* Fix issues with new build functions

* Allow tests to load auxiliary test data

* Implement msn tests and update k_config test for new parameter

* Correctly specify ppm in generate_structures

* Minor docstring and code reformatting

* Add binder dir

* Add example notebook

* Remove scripts

* Implement basic notebook

* Add small substructures to database prior to msn annotation

* Complete notebook example

* Fix logic for when smi_out_dir is None

* Rename example_msms.ipynb to workflow.ipynb

* Add pip to install metaboblend

* Add data dir, remove databases dir, move test data to data dir

* Write notebook databases to notebook_data

* Unzip test data

* Simplify test paths

* Remove databases from gitignore

* Use test databases for notebook

* Implement simple hydrogenation rules

* Get bond types rather than number of available atoms for hydrogen rule calculations

* Don't count dummy atoms for bond type calculations

* Remove dummy atom mass

* Use max_degree of 6 and 2 available_atoms by default for create_substructure_database

* Account for the fact we use neutral peaks (i.e. have removed adduct ion)

* Modify hydrogen re-arrangement rules for doulbe bonds

* Update databases tests

* Implement test for calculate_possible_hydrogenations using reference numbers

* Add test for calculate_hydrogen_rearrangements

* Update hydrogen re-arrangement calculation function documentation

* Update remaining unit tests

* Add hydrogen re-arrangement compound HMDB XMLs

* Record even substructures

* Record even substructures in results DB

* Add indexes to improve combine_ecs function performance

* Improve results DB hierarchy and implement aggregation of scoring metrics

* Define SQLite functions to calculate scores via queries alone

* Record max BDE in spectra results table

* Calculate frequency in the absence of scores (for non-MSn method)

* Retain substructures does not cause substructures not to be initially recorded

* Add additional scoring metrics

* Update results db test data

* Define ppm error and valence of fragment prior to re-ordering

* Configure checks on recording of putative structure information

* Calculate scores at substructure combination level

* Convert True to 1 and False to 0 for conversion to SQLite boolean type

* Index results DB

* Use a loop in place of pool.map

* Minor performance improvements

* Merge minor performance improvements

* Use the minimum absolute error for getting possible fragment ions

* Add separate absolute error options for MSn peak and full structure

* Use 0.005 for abs_error_precursor

* Drop indexes before inserting into results DB

* Add results table index on ms_id_num and structure_smiles

* Update results DB tests

* Add table for generating unique structure smiles IDs

* Calculate cosine spectrum similarity

* Allow for the specification of weights for the results database scoring calculations

* Aggregate structure scores but force floating point division

* Select fragment and substructure id when calculating results scores for the correlated query

* Update results DB tests with updated scores

* Don't create indexes until structure scoring

* Don't include valence=0 substructures in the substructure database

* Add max BDE parameter for building

* Remove redundant connectivity graphs

* Update data to test filter records function

* Update dictionary pickle with Python 3.7

* Update file header

* Update contact information

* Update setup.py

* Update tests for RDKit changes

* Update README

* Keep functioning buttons

* Update testing workflow

* Use python 3.7

* Remove unused dependencies

* Use only the channel conda-forge

* Add pillow and pyqt dependencies

* Remove list definition in function arguments

* Add algorithms test

* Merge database tests into single file

* Restructure modules

* Restructure tests

* Update outdated imports

* Omit notebooks from coverage

Co-authored-by: Ralf Weber <[email protected]>

Author: jackgisby

.coveragerc

@@ -1,2 +1,2 @@
 [run]
-omit = tests/*,setup.py,metaboblend/__main__.py
+omit = tests/*,setup.py,metaboblend/__main__.py,notebooks/

.github/workflows/build-test.yml

@@ -8,8 +8,8 @@ jobs:
 
     strategy:
       matrix:
-        os: [ubuntu-latest, windows-latest, macos-latest]
-        python-version: [3.6, 3.7, 3.8]
+        os: [ ubuntu-latest, windows-latest, macos-latest ]
+        python-version: [ 3.7, 3.8, 3.9 ]
 
     env:
       OS: ${{ matrix.os }}
@@ -19,21 +19,29 @@ jobs:
       - uses: actions/checkout@v2
 
       - name: Setup conda - Python ${{ matrix.python-version }}
-        uses: s-weigand/setup-conda@v1
+        uses: conda-incubator/setup-miniconda@v2
         with:
-          update-conda: true
+          auto-update-conda: true
+          activate-environment: metaboblend
           python-version: ${{ matrix.python-version }}
-          conda-channels: anaconda, conda-forge
+          environment-file: environment.yml
+          channels: anaconda, conda-forge
 
-      - name: Install dependencies
+      - name: Build MetaboBlend
+        shell: bash -l {0}
         run: |
+          python setup.py install
+          metaboblend --help
 
-          python --version
-          conda env update --file environment.yml --name base
+      - name: Test with pytest-cov
+        shell: bash -l {0}
+        run: |
+          conda install pytest codecov pytest-cov -c conda-forge
+          pytest --cov ./ --cov-config=.coveragerc --cov-report=xml
 
       - name: Lint with flake8
+        shell: bash -l {0}
         run: |
-
           conda install flake8
 
           # stop build if there are Python syntax errors or undefined names
@@ -42,15 +50,6 @@ jobs:
           # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
           flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
 
-      - name: Test with pytest-cov
-        run: |
-
-          python setup.py install
-          metaboblend --help
-
-          conda install pytest codecov pytest-cov -c conda-forge
-          pytest --cov ./ --cov-config=.coveragerc --cov-report=xml
-
       - name: Upload code coverage to codecov
         uses: codecov/codecov-action@v1
         with:

.gitignore

@@ -70,6 +70,8 @@ target/
 
 # Jupyter Notebook
 .ipynb_checkpoints
+notebooks/notebook_data
+notebooks/notebook_data/*
 
 # pyenv
 .python-version
@@ -105,4 +107,4 @@ ENV/
 # ignore test files
 */libgcc_s_dw2-1.dll
 */libstdc++-6.dll
-tests/test*
+tests/tmp*

README.rst

@@ -1,6 +1,9 @@
 MetaboBlend
 ===========
-|Version| |Py versions| |Git| |Bioconda| |Build Status| |License| |RTD doc| |codecov| |binder|
+..
+    |Version| |Py versions| |Bioconda|  |RTD doc| |License|  |binder|
+
+|Git| |Build Status| |codecov|
 
 Python package for *de novo* structural elucidation of small molecules in mass spectrometry-based Metabolomics
 
@@ -32,12 +35,11 @@ will help you to make the PR if you are new to `git`.
 Developers & Contributors
 -------------------------
  - Ralf J. M. Weber ([email protected]) - `University of Birmingham (UK) <https://www.birmingham.ac.uk/staff/profiles/biosciences/weber-ralf.aspx>`_
- - Jack Gisby ([email protected]) - `University of Birmingham (UK) <http://www.birmingham.ac.uk/index.aspx>`_
-
+ - Jack Gisby ([email protected]) - `University of Birmingham (UK) <http://www.birmingham.ac.uk/index.aspx>`_, `Imperial College London (UK) <https://www.imperial.ac.uk/>`_
 
 Licenses
 --------
-MetaboBlend is licensed under the GNU General Public License v3.0 (see `LICENSE file <https://github.com/computational-metabolomics/metaboblend/blob/master/LICENSE>`_ for licensing information). Copyright © 2019 - 2020 Ralf Weber
+MetaboBlend is licensed under the GNU General Public License v3.0 (see `LICENSE file <https://github.com/computational-metabolomics/metaboblend/blob/master/LICENSE>`_ for licensing information). Copyright © 2019 - 2020 Jack Gisby, Ralf Weber
 
 
 .. |Build Status| image:: https://github.com/computational-metabolomics/metaboblend/workflows/metaboblend/badge.svg

binder/environment.yml

@@ -0,0 +1,17 @@
+name: metaboblend
+channels:
+  - conda-forge
+  - bioconda
+dependencies:
+  - python=3.7
+  - numpy
+  - scipy
+  - pandas
+  - networkx
+  - rdkit
+  - biopython
+  - matplotlib
+  - nauty
+  - pip
+  - pip:
+    - -e ../

docs/source/conf.py

@@ -20,7 +20,7 @@
 # -- Project information -----------------------------------------------------
 
 project = 'MetaboBlend'
-copyright = '2020, Ralf Weber'
+copyright = '2020, Jack Gisby, Ralf Weber'
 author = 'Jack Gisby, Ralf Weber'
 
 # -- General configuration ---------------------------------------------------

docs/source/license.rst

@@ -1,4 +1,4 @@
 License
 -------
 TODO: change package name
-*MetaboBlend* is licensed under the GNU General Public License v3.0 (see `LICENSE file <https://github.com/computational-metabolomics/metaboblend/blob/master/LICENSE>`_ for licensing information). Copyright © 2019 - 2020 Ralf Weber
+*MetaboBlend* is licensed under the GNU General Public License v3.0 (see `LICENSE file <https://github.com/computational-metabolomics/metaboblend/blob/master/LICENSE>`_ for licensing information). Copyright © 2019 - 2020 Jack Gisby, Ralf Weber

environment.yml

@@ -1,14 +1,12 @@
 name: metaboblend
 channels:
   - conda-forge
-  - bioconda
 dependencies:
- - python>=3.6
+ - python>=3.7
+ - pillow!=9.2.0
+ - pyqt
+ - matplotlib
  - numpy
- - scipy
- - pandas
  - networkx
  - rdkit
- - biopython
- - matplotlib
  - nauty

metaboblend/__init__.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
-# Copyright © 2019-2020 Ralf Weber
+# Copyright © 2019-2020 Jack Gisby, Ralf Weber
 #
 # This file is part of MetaboBlend.
 #
@@ -19,7 +19,7 @@
 # along with MetaboBlend.  If not, see <https://www.gnu.org/licenses/>.
 #
 
-__author__ = 'Ralf Weber ([email protected])'
-__credits__ = 'Ralf Weber ([email protected])'
+__authors__ = ['Ralf Weber ([email protected])', 'Jack Gisby ([email protected])']
+__credits__ = ['Ralf Weber ([email protected])', 'Jack Gisby ([email protected])']
 __version__ = '0.1.0'
 __license__ = 'GPLv3'

metaboblend/__main__.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 #
-# Copyright © 2019-2020 Ralf Weber
+# Copyright © 2019-2020 Jack Gisby, Ralf Weber
 #
 # This file is part of MetaboBlend.
 #