docs: Add about page and update shouldi

Signed-off-by: John Andersen <[email protected]>

Author: John Andersen

CHANGELOG.md

@@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - shouldi example got a bandit operation which tells users not to install if
   there are more than 5 issues of high severity and confidence.
 - dev service got the ability to run a single operation in a standalone fashion.
+- About page to docs.
 ### Changed
 - feature/codesec became it's own branch, binsec
 - BaseOrchestratorContext `run_operations` strict is default to true. With

docs/about.rst

@@ -0,0 +1,91 @@
+About
+=====
+
+This project is for your if:
+
+- You want to apply machine learning by just bringing your data
+
+  - No need to deal with the various specifics of ML libraries
+
+- You want to do machine learning on a new problem that you don't have a dataset
+  for, so you need to generate it.
+
+- You want to harness the power of directed graph execution to write testable,
+  maintainable code.
+
+Machine Learning
+----------------
+
+Python was chosen because of the machine learning community’s preference towards
+it. In addition to the data flow side of DFFML, there is a machine learning
+focused side. It provides a standardized way to defining, training, and using
+models. It also allows for wrapping existing models so as to expose them via the
+standardized API. Models can then be integrated into data flows as operations.
+This enables trivial layering of models to create complex features. See
+:ref:`plugin_models` for existing models and usage.
+
+Data Flows - Directed Graph Execution
+-------------------------------------
+
+The idea behind this project is to provide a way to link together various new
+or existing pieces of code and run them via an orchestration engine that
+forwards the data between them all. Similar a microservice architecture but with
+the orchestration being preformed according to a directed graph. This offers
+greater flexibility in that interaction between services can easily be modified
+without changing code, only the graph (known as the dataflow).
+
+This is an example of the dataflow for a meta static analysis tool for Python,
+``shouldi``. We take the package name (package) and feed it through operations,
+which are just functions (but could be anything, some SaaS web API endpoint for
+instance). All the data generated by running these operations is queriable,
+allowing us to structure the output in whatever way is most fitting for our
+application.
+
+.. image:: /images/shouldi-dataflow.png
+
+Consistant API
+--------------
+
+DFFML decouples the interface through which the flow is accessed from the flow
+itself. For instance, data flows can be run via the library, HTTP API, CLI, or
+any communication channel (next targets are Slack and IRC). Data flows are also
+asynchronous in nature, allowing them to be used to build any event driven
+application (Chat, IoT data, etc.). The way in which operations are defined and
+executed by the orchestrator will let us take existing API endpoints and code in
+other languages and combine them into one cohesive workflow. The architecture
+itself is programming language agnostic, the first implementation has been
+written in Python.
+
+Plugins
+-------
+
+We take a community driven approach to content. Architecture is plugin based,
+which means anyone can swap out any piece by writing their own plugin and
+publishing it to the Python Package Index. This means that developers can
+publish operations and machine learning models that work out of the box with
+everything else maintained as a part of the core repository and with other
+developers models and operations. :doc:`tutorials/index` show how to create your
+own plugins.
+
+Team
+----
+
+We have a team of volunteers working on the project. We hold weekly meetings
+and have a mailing list and chat. If you want to get involved, ask questions, or
+get help getting started, see :doc:`community`.
+
+We participated in Google Summer of Code 2019 under the Python Software
+Foundation. A big thanks to our students, Yash and Sudharsana!
+
+- `GSoC 2019 Student Contributions <https://github.com/intel/dffml/wiki/GSoC-2019#student-contributions>`_
+
+Users
+-----
+
+The following is a list of organizations and projects using DFFML. Please let us
+know if you are using DFFML and we'll add you to the list. If you want help
+using DFFML, see the :doc:`community` page.
+
+- Intel
+
+  - Open Source Software dependency security viability analysis

docs/community.rst

@@ -20,21 +20,10 @@ the agenda.
 - `Meeting Minutes <https://docs.google.com/document/d/16u9Tev3O0CcUDe2nfikHmrO3Xnd4ASJ45myFgQLpvzM/>`_
 - `Recordings <https://www.youtube.com/channel/UCorEDRWGikwBH3dsJdDK1qA>`_
 
-The meeting link will be posted in the Gitter channel shortly before starting.
 To join the meeting, click on the Google calendar below for the day you are
 trying to join, then click "more details" which will open your Google calendar,
 then click "Join Hangouts".
 
-The goal of DFFML is to build a community driven library of plugins for dataset
-generation and model definition. So that we as developers and researchers can
-quickly and easily plug and play various pieces of data with various model
-implementations.
-
-The more we build up the library of plugins (which anyone can maintain, they
-don't have to be contributed upstream unless you want to) the more variations on
-model implementations, feature data generators, and database backend
-abstractions, we all have to work with.
-
 If you have developed a plugin for DFFML and would like it listed in the docs,
 please open an `issue <https://github.com/intel/dffml/issues/new?assignees=&labels=documentation&template=new_plugin.md&title=plugin%3A+new%3A+>`_.