docs: Plugins

* Add spec field to Definition to allow for documenting of complex types

Signed-off-by: John Andersen <[email protected]>

Author: John Andersen

CHANGELOG.md

@@ -5,6 +5,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+### Added
+- Definition spec field to specify a class representative of key value pairs for
+  definitions with primitives which are dictionaries
+- Auto generation of documentation for operation implementations, models, and
+  sources. Generated docs include information on configuration options and
+  inputs and outputs for operation implementations.
 ### Changed
 - OperationImplementation add_label and add_orig_label methods now use op.name
   instead of ENTRY_POINT_ORIG_LABEL and ENTRY_POINT_NAME.

dffml/df/types.py

@@ -15,6 +15,8 @@ class Definition(NamedTuple):
     name: str
     primitive: str
     lock: bool = False
+    # spec is a NamedTuple which could be populated via a dict
+    spec: NamedTuple = None
 
     def __repr__(self):
         return self.name

dffml/operation/output.py

@@ -11,12 +11,6 @@
 )
 from ..df.exceptions import DefinitionNotInContext
 
-group_by_spec = Definition(name="group_by_spec", primitive="Dict[str, Any]")
-
-group_by_output = Definition(
-    name="group_by_output", primitive="Dict[str, List[Any]]"
-)
-
 
 class GroupBySpec(NamedTuple):
     group: Definition
@@ -42,6 +36,15 @@ async def resolve(
         return cls(**exported)
 
 
+group_by_spec = Definition(
+    name="group_by_spec", primitive="Dict[str, Any]", spec=GroupBySpec
+)
+
+group_by_output = Definition(
+    name="group_by_output", primitive="Dict[str, List[Any]]"
+)
+
+
 @op(
     name="group_by",
     inputs={"spec": group_by_spec},

docs/community.rst

@@ -0,0 +1,23 @@
+Community
+=========
+
+Miscellaneous questions can be asked on
+`Gitter <https://gitter.im/dffml/community>`_ or in the
+`GitHub Issue <https://github.com/intel/dffml/issues/12>`_ meant for questions.
+
+We hold weekly meetings on Tuesday's at 9 AM PST. Meetings are held via Google
+Hangouts.
+
+If you have questions about you might go about integrating machine learning, or
+using data flow programming, hop on the meeting and we'll do Q & A at the end of
+the agenda.
+
+- `Join Meeting <https://hangouts.google.com/call/WpnVSFrZFJl-QfLReYcfAEEE>`_
+- `Meeting Minutes <https://docs.google.com/document/d/16u9Tev3O0CcUDe2nfikHmrO3Xnd4ASJ45myFgQLpvzM/>`_
+
+If you have developed a plugin for DFFML and would like it listed in the docs,
+please open an `issue <https://github.com/intel/dffml/issues/new?assignees=&labels=documentation&template=new_plugin.md&title=plugin%3A+new%3A+>`_.
+
+.. raw:: html
+
+    <iframe src="https://calendar.google.com/calendar/embed?src=0b6g0ki0qsq1fd1sr703ceeioc%40group.calendar.google.com&ctz=America%2FLos_Angeles" marginheight="0" marginwidth="0" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true" width="688" height="600" frameborder="0"></iframe>

docs/concepts/index.rst

@@ -1,9 +1,12 @@
-Models
-======
+Concepts
+========
+
+The following documents cover the core concepts of DFFML. They will be useful to
+understand as you work with the code to implement new plugins, or use existing
+pluings in the eco system.
 
 .. toctree::
-    :glob:
     :maxdepth: 2
     :caption: Contents:
 
-    base
+    data_flow

docs/index.rst

@@ -28,7 +28,10 @@ set of feature generators which gather data from git repositories.
    :maxdepth: 2
    :caption: Contents:
 
+   community
    usage/*
+   concepts/index
+   plugins/index
    api
 
 

docs/plugins/dffml_model.rst

@@ -0,0 +1,38 @@
+Models
+======
+
+Models are implementations of :class:`dffml.model.model.Model`, they
+abstract the usage of machine learning models.
+
+dffml_model_tensorflow
+----------------------
+
+dnn
+~~~
+
+*Core*
+
+Implemented using Tensorflow's DNNClassifier. Models are saved under the
+``directory`` in subdirectories named after the hash of their feature names.
+
+**Args**
+
+- directory: String
+
+  - default: /home/user/.cache/dffml/tensorflow
+  - Directory where state should be saved
+
+- steps: Integer
+
+  - default: 3000
+  - Number of steps to train the model
+
+- epochs: Integer
+
+  - default: 30
+  - Number of iterations to pass over all repos in a source
+
+- hidden: List of integers
+
+  - default: [12, 40, 15]
+  - List length is the number of hidden layers in the network. Each entry in the list is the number of nodes in that hidden layer