docs: contributing: Notes on development dependencies

Signed-off-by: John Andersen <[email protected]>

Author: pdxjohnny

CHANGELOG.md

@@ -5,6 +5,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+### Added
+- Notes on development dependencies in `setup.py` files to codebase notes.
 
 ## [0.3.3] - 2020-02-10
 ### Added

docs/contributing/codebase.rst

@@ -153,3 +153,92 @@ The command line equivalent for the model is...
 
 The reason it's called ``arg`` right now is because the parsing of the command
 line came first and the argument is stored there when it's not nested.
+
+setup.py
+--------
+
+There are various ``setup.py`` files throughout the codebase, one for the main
+package, one for each plugin, and one in ``skel/``. There are also
+``setup_common.py`` files.
+
+Checking For Development Version
+++++++++++++++++++++++++++++++++
+
+You might have asked yourself, what is this thing?
+
+.. code-block:: python
+
+    INSTALL_REQUIRES = [] + (
+        ["dffml>=0.3.3"]
+        if not any(
+            list(
+                map(
+                    os.path.isfile,
+                    list(
+                        map(
+                            lambda syspath: os.path.join(
+                                syspath, "dffml.egg-link"
+                            ),
+                            sys.path,
+                        )
+                    ),
+                )
+            )
+        )
+        else []
+    )
+
+This code is needed because ``python`` will use extracted versions of packages
+over development versions if they are installed.
+
+``pip`` will download and extract a package (thereby installing it) if it sees
+it in the ``INSTALL_REQUIRES`` list. This wrecks havoc with our development
+workflow.
+
+For example, when we put the main package, ``dffml`` in the ``INSTALL_REQUIRES``
+list of a plugin, ``pip`` will go off and download the appropriate version from
+PyPi and extract it to a place Python searches for packages. Now when we run
+anything we'll end up using the version ``pip`` just installed instead of the
+version we're developing on locally.
+
+The solution to this is to add the above code block to ``setup.py`` files. The
+innermost list is ``sys.path``, which is all the places Python is going to look
+for packages when there is an ``import`` statement. We use ``map`` to apply a
+function to each directory in ``sys.path``. The map will take the directory name
+and add ``dffml.egg-link`` to it. We add this because when you install something
+in development mode (``dffml`` in this case) ``pip`` creates this ``.egg-link``
+file. In the file is the path to the source code you're working on. Therefore,
+if that file exists, then the package is installed in development mode. The next
+``map`` then checks if any of the file paths generated by the previous ``map``
+exist. If ``any`` of them exist, then there is a ``.egg-link`` file somewhere in
+Python's search path, which means the package (``dffml`` in this case, hence the
+``dffml.egg-link``) is installed in development mode.
+
+If the package is installed in development mode, then we don't want ``pip`` to
+install it from PyPi, since that would cause the development version not to be
+used.
+
+Notes on Various Subsystems
+---------------------------
+
+DFFML is comprised of various subsystems. The following are some notes
+that might be helpful when working on each of them.
+
+Working on ``skel/``
+++++++++++++++++++++
+
+The packages in ``skel/`` are used to create new DFFML packages.
+
+For example, to create a new package containing operations we run the following.
+
+.. code-block:: console
+
+    $ dffml service dev create operations dffml-operations-feedface
+
+If you want to work on any of the packages in ``skel/``, you'll need to run the
+``skel link`` command first fromt he ``dev`` service. This will symlink required
+files in from ``common/`` so that testing will work.
+
+.. code-block:: console
+
+    $ dffml service dev skel link

docs/contributing/index.rst

@@ -145,5 +145,4 @@ Working on DFFML
     git
     testing
     docs
-    subsystems
     debugging