Swap Packaging to Poetry (#60)

* Swapping packaging to Poetry
- main thing is removal of setup.py and requirements.txt, instead we have pyproject.toml and poetry.lock. pyproject is the project description that contains everything that setup.py does (i was careful to keep it as similar as possible to the existing one while also adding version constraints that allow a successful build/install), and the poetry.lock contains the exact specification of what to install
- I bumped up the minimum python from 3.5 to 3.6.1 to be able to install a few packages, both 3.5 and 3.6 are EOL and we should think of bumping this up further.

Also
- added some type hints to the DLCLive class which was bothering me to not have tab completion when i've been using it lol
- DLCLive object tries to mutate something that is by default a tuple, which should cause an error if anyone tries to use tflite and dynamic (instead of warning and repairing as is intended)
- Using Pathlib to resolve the path and check that it exists when loading configuration, this gives more helpful error messages bc people know where DLC is looking in their filesystem/resolves ambiguity of relative paths
- added a CITATION.cff file <3

* oop forgot to quote the pseudo-imported Processor class

* did not see we had github actions in here!
- removed pip install requirements
- removed pytest tests (i don't see any!?)
- installing package and running test from already checked-out copy rather than pip installing from git://
- the packaging script doesn't look like it's doing anything, but i'd be happy to write one that autodeploys on tagged commits to master

* - added argument parser to be able to suppress display on testing, and then call it when doing the tests
- also install and run using poetry because windows paths are an abominable hell
- tensorflow 2.7 only support python >3.7

* ohhhh Display is still instantiated even if the `display` arg is false because it gets instantiated before the check happens whether it should be saved or not.
also limited tf versions for python <3.7 because it has to like do the ridiculous pip thing where it downloads all of them for some reason

* arg not being passed through, does windows want to run poetry yet?

* OH need to put argparser within the function

- also cleaned up file handling in test function. Previously the video file was downloaded and not used, and lots of implicit paths that seem to be causing people trouble when running it.

* try catch removing temporary directory because WINDOWS GOD DAMN IT

* ok 3.6 is EOL anyway and it's causing all the problems

Author: sneakers-the-rat

.github/workflows/python-package.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       # You can use PyPy versions in python-version. For example, pypy2 and pypy3
       matrix:
-        python-version: [3.6, 3.7, 3.8]
+        python-version: [3.7, 3.8, 3.9]
 
     steps:
     - uses: actions/checkout@v2
@@ -29,4 +29,4 @@ jobs:
       run: python -c "import sys; print(sys.version)"
     #test installation of DLC-core dependencies:
     - name: Install dependencies
-      run: pip install -r requirements.txt
+      run: pip install .

.github/workflows/testing.yml

@@ -31,11 +31,6 @@ jobs:
         uses: actions/setup-python@v2
         with:
           python-version: ${{ matrix.python-version }}
-
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip setuptools wheel
-          pip install -r requirements.txt
       - name: Install ffmpeg
         run: |
           if [ "$RUNNER_OS" == "Linux" ]; then
@@ -47,12 +42,8 @@ jobs:
             choco install ffmpeg
           fi
         shell: bash
-
-      - name: Run pytest tests
-        run: |
-          pip install pytest
-          python -m pytest
-      - name: Run functional tests
+      - name: Install and test
         run: |
-          pip install git+git://github.com/${{ github.repository }}.git@${{ github.sha }}
-          dlc-live-test
+          python -m pip install --upgrade pip wheel poetry
+          python -m poetry install
+          python -m poetry run dlc-live-test --nodisplay

CITATION.cff

@@ -0,0 +1,55 @@
+# This CITATION.cff file was generated with cffinit.
+# Visit https://bit.ly/cffinit to generate yours today!
+
+cff-version: 1.2.0
+title: >-
+  Real-time, low-latency closed-loop feedback using
+  markerless posture tracking
+message: >-
+  If you utilize our tool, please [cite Kane et al,
+  eLife
+  2020](https://elifesciences.org/articles/61909).
+  The preprint is available here:
+  https://www.biorxiv.org/content/10.1101/2020.08.04.236422v2
+type: software
+authors:
+  - given-names: Gary
+    name-particle: A
+    family-names: Kane
+    affiliation: >-
+      The Rowland Institute at Harvard, Harvard
+      University, Cambridge, United States
+  - given-names: Gonçalo
+    family-names: Lopes
+    affiliation: 'NeuroGEARS Ltd, London, United Kingdom'
+  - given-names: Jonny
+    name-particle: L
+    family-names: Saunders
+    affiliation: >-
+      Institute of Neuroscience, Department of
+      Psychology, University of Oregon, Eugene,
+      United States
+  - given-names: Alexander
+    family-names: Mathis
+    affiliation: >-
+      The Rowland Institute at Harvard, Harvard
+      University, Cambridge, United States; Center
+      for Neuroprosthetics, Center for Intelligent
+      Systems, & Brain Mind Institute, School of Life
+      Sciences, Swiss Federal Institute of Technology
+      (EPFL), Lausanne, Switzerland
+  - given-names: Mackenzie
+    name-particle: W
+    family-names: Mathis
+    affiliation: >-
+      The Rowland Institute at Harvard, Harvard
+      University, Cambridge, United States; Center
+      for Neuroprosthetics, Center for Intelligent
+      Systems, & Brain Mind Institute, School of Life
+      Sciences, Swiss Federal Institute of Technology
+      (EPFL), Lausanne, Switzerland
+    email: [email protected]
+date-released: 2020-08-05
+doi: "10.7554/eLife.61909"
+license: "AGPL-3.0-or-later"
+version:  "1.0.2"

dlclive/check_install/check_install.py

@@ -7,42 +7,88 @@
 
 
 import os
+import sys
 import shutil
+import warnings
+
 from dlclive import benchmark_videos
 import urllib.request
+import argparse
+from pathlib import Path
+import tarfile
+
+
+def urllib_pbar(count, blockSize, totalSize):
+    percent = int(count * blockSize * 100 / totalSize)
+    outstr = f"{round(percent)}%"
+    sys.stdout.write(outstr)
+    sys.stdout.write("\b"*len(outstr))
+    sys.stdout.flush()
 
+def main(display:bool=None):
+    parser = argparse.ArgumentParser(
+        description="Test DLC-Live installation by downloading and evaluating a demo DLC project!")
+    parser.add_argument('--nodisplay', action='store_false', help="Run the test without displaying tracking")
+    args = parser.parse_args()
 
-def main():
+    if display is None:
+        display = args.nodisplay
+
+    if not display:
+        print('Running without displaying video')
 
     # make temporary directory in $HOME
     print("\nCreating temporary directory...\n")
-    home = os.path.expanduser("~")
-    tmp_dir = os.path.normpath(f"{home}/dlc-live-tmp")
-    os.makedirs(tmp_dir, exist_ok=True)
-    os.chdir(tmp_dir)
+    tmp_dir = Path().home() / 'dlc-live-tmp'
+    tmp_dir.mkdir(mode=0o775,exist_ok=True)
+
+    video_file = str(tmp_dir / 'dog_clip.avi')
+    model_tarball = tmp_dir / 'DLC_Dog_resnet_50_iteration-0_shuffle-0.tar.gz'
+    model_dir = model_tarball.with_suffix('').with_suffix('') # remove two suffixes (tar.gz)
+
 
     # download dog test video from github:
+    print(f"Downloading Video to {video_file}")
     url_link = "https://github.com/DeepLabCut/DeepLabCut-live/blob/master/check_install/dog_clip.avi?raw=True"
-    urllib.request.urlretrieve(url_link, "dog_clip.avi")
-    video_file = os.path.join(url_link, "dog_clip.avi")
+    urllib.request.urlretrieve(url_link, video_file, reporthook=urllib_pbar)
 
     # download exported dog model from DeepLabCut Model Zoo
-    print("Downloading full_dog model from the DeepLabCut Model Zoo...")
-    model_url = "http://deeplabcut.rowland.harvard.edu/models/DLC_Dog_resnet_50_iteration-0_shuffle-0.tar.gz"
-    os.system(f"curl {model_url} | tar xvz")
+    if Path(model_tarball).exists():
+        print('Tarball already downloaded, using cached version')
+    else:
+        print("Downloading full_dog model from the DeepLabCut Model Zoo...")
+        model_url = "http://deeplabcut.rowland.harvard.edu/models/DLC_Dog_resnet_50_iteration-0_shuffle-0.tar.gz"
+        urllib.request.urlretrieve(model_url, str(model_tarball), reporthook=urllib_pbar)
+
+    print('Untarring compressed model')
+    model_file = tarfile.open(str(model_tarball))
+    model_file.extractall(str(model_dir.parent))
+    model_file.close()
+
+    # assert these things exist so we can give informative error messages
+    assert Path(video_file).exists()
+    assert Path(model_dir).exists() and Path(model_dir).is_dir()
 
     # run benchmark videos
     print("\n Running inference...\n")
-    model_dir = "DLC_Dog_resnet_50_iteration-0_shuffle-0"
-    print(video_file)
-    benchmark_videos(model_dir, video_file, display=True, resize=0.5, pcutoff=0.25)
+    # model_dir = "DLC_Dog_resnet_50_iteration-0_shuffle-0"
+    # print(video_file)
+    benchmark_videos(str(model_dir), video_file, display=display, resize=0.5, pcutoff=0.25)
 
     # deleting temporary files
     print("\n Deleting temporary files...\n")
-    shutil.rmtree(tmp_dir)
+    try:
+        shutil.rmtree(tmp_dir)
+    except PermissionError:
+        warnings.warn(f'Could not delete temporary directory {str(tmp_dir)} due to a permissions error, but otherwise dlc-live seems to be working fine!')
 
     print("\nDone!\n")
 
 
 if __name__ == "__main__":
-    main()
+
+
+    display = args.nodisplay
+
+
+    main(display=args.nodisplay)

dlclive/dlclive.py

@@ -11,6 +11,9 @@
 import warnings
 import numpy as np
 import tensorflow as tf
+import typing
+from pathlib import Path
+from typing import Optional, Tuple, List
 
 try:
     TFVER = [int(v) for v in tf.__version__.split(".")]
@@ -32,7 +35,8 @@
 from dlclive.display import Display
 from dlclive import utils
 from dlclive.exceptions import DLCLiveError, DLCLiveWarning
-
+if typing.TYPE_CHECKING:
+    from dlclive.processor import Processor
 
 class DLCLive(object):
     """
@@ -99,23 +103,23 @@ class DLCLive(object):
 
     def __init__(
         self,
-        model_path,
-        model_type="base",
-        precision="FP32",
+        model_path:str,
+        model_type:str="base",
+        precision:str="FP32",
         tf_config=None,
-        cropping=None,
-        dynamic=(False, 0.5, 10),
-        resize=None,
-        convert2rgb=True,
-        processor=None,
-        display=False,
-        pcutoff=0.5,
-        display_radius=3,
-        display_cmap="bmy",
+        cropping:Optional[List[int]]=None,
+        dynamic:Tuple[bool, float, float]=(False, 0.5, 10),
+        resize:Optional[float]=None,
+        convert2rgb:bool=True,
+        processor:Optional['Processor']=None,
+        display:typing.Union[bool, Display]=False,
+        pcutoff:float=0.5,
+        display_radius:int=3,
+        display_cmap:str="bmy",
     ):
 
         self.path = model_path
-        self.cfg = None
+        self.cfg = None # type: typing.Optional[dict]
         self.model_type = model_type
         self.tf_config = tf_config
         self.precision = precision
@@ -125,11 +129,12 @@ def __init__(
         self.resize = resize
         self.processor = processor
         self.convert2rgb = convert2rgb
-        self.display = (
-            Display(pcutoff=pcutoff, radius=display_radius, cmap=display_cmap)
-            if display
-            else None
-        )
+        if isinstance(display, Display):
+            self.display = display
+        elif display:
+            self.display = Display(pcutoff=pcutoff, radius=display_radius, cmap=display_cmap)
+        else:
+            self.display = None
 
         self.sess = None
         self.inputs = None
@@ -141,7 +146,7 @@ def __init__(
         # checks
 
         if self.model_type == "tflite" and self.dynamic[0]:
-            self.dynamic[0] = False
+            self.dynamic = (False, *self.dynamic[1:])
             warnings.warn(
                 "Dynamic cropping is not supported for tensorflow lite inference. Dynamic cropping will not be used...",
                 DLCLiveWarning,
@@ -158,14 +163,14 @@ def read_config(self):
             error thrown if pose configuration file does nott exist
         """
 
-        cfg_path = os.path.normpath(self.path + "/pose_cfg.yaml")
-        if not os.path.isfile(cfg_path):
+        cfg_path = Path(self.path).resolve() / "pose_cfg.yaml"
+        if not cfg_path.exists():
             raise FileNotFoundError(
-                f"The pose configuration file for the exported model at {cfg_path} was not found. Please check the path to the exported model directory"
+                f"The pose configuration file for the exported model at {str(cfg_path)} was not found. Please check the path to the exported model directory"
             )
 
         ruamel_file = ruamel.yaml.YAML()
-        self.cfg = ruamel_file.load(open(cfg_path, "r"))
+        self.cfg = ruamel_file.load(open(str(cfg_path), "r"))
 
     @property
     def parameterization(self) -> dict: