Peter/v02 release update (#265)

* Update documentation and bump version

* Update docs and minor fix

* Update documentation

* Update README.md

* Fix legal headers and imports

* Fixed documentation errors

* minor

* Minor docs fix

* update api.rst

Author: peteriz

.gitignore

@@ -27,10 +27,9 @@ generated
 dist
 .nlp_architect_env/
 src/
-_generated_reqs.txt
 pylint.html
 pylint.txt
 flake8.txt
 nlp_architect/pipelines/bist-pretrained/*
 venv
-nlp_architect/api/ner-pretrained/*
+nlp_architect/api/ner-pretrained/*

CHANGELOG.md

@@ -21,7 +21,7 @@ DOC_PUB_RELEASE_PATH := $(DOC_PUB_PATH)/$(RELEASE)
 
 LIBRARY_NAME := nlp_architect
 VIRTUALENV_DIR := .nlp_architect_env
-REQ_FILE := _generated_reqs.txt
+REQ_FILE := requirements.txt
 ACTIVATE := $(VIRTUALENV_DIR)/bin/activate
 MODELS_DIR := $(LIBRARY_NAME)/models
 NLP_DIR := $(LIBRARY_NAME)/nlp
@@ -47,6 +47,8 @@ pylint: test_prepare
 	@. $(ACTIVATE); pylint $(PYLINT_CHECK_DIRS) | tee pylint.txt
 	@. $(ACTIVATE); python3 tests/utils/ansi2html.py pylint.txt > pylint.html
 
+style: flake pylint
+
 doc_prepare: doc_requirements.txt $(ACTIVATE)
 	@. $(ACTIVATE); pip3 install -r doc_requirements.txt > /dev/null 2>&1
 
@@ -64,24 +66,15 @@ html: doc_prepare $(ACTIVATE)
 clean:
 	@echo "Cleaning files.."
 	@rm -rf $(VIRTUALENV_DIR)
-	@rm -rf $(REQ_FILE)
 
 ENV_EXIST := $(shell test -d $(VIRTUALENV_DIR) && echo -n yes)
-REQ_EXIST := $(shell test -f $(REQ_FILE) && echo -n yes)
-
-$(REQ_FILE):
-ifneq ($(REQ_EXIST), yes)
-	@echo "Generating pip requirements file"
-	@bash generate_reqs.sh
-endif
 
 $(ACTIVATE): $(REQ_FILE)
 ifneq ($(ENV_EXIST), yes)
 	@echo "NLP Architect installation"
 	@echo "**************************"
 	@echo "Creating new environment"
 	@echo
-#	virtualenv -p python3 $(VIRTUALENV_DIR)
 	python3 -m venv $(VIRTUALENV_DIR)
 	@. $(ACTIVATE); pip3 install -U pip
 endif

Makefile

@@ -4,65 +4,19 @@
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/NervanaSystems/nlp-architect/blob/master/LICENSE)
 [![Docs](https://img.shields.io/readthedocs/pip/stable.svg)](http://nlp_architect.nervanasys.com)
 
-
-NLP Architect is an open-source Python library for exploring state-of-the-art 
-deep learning topologies and techniques for natural language processing and 
-natural language understanding. It is intended to be a platform for future 
+NLP Architect is an open-source Python library for exploring state-of-the-art
+deep learning topologies and techniques for natural language processing and
+natural language understanding. It is intended to be a platform for future
 research and collaboration.
 
-The current version of NLP Architect includes these features that we found 
-interesting from both research perspectives and practical applications:
-
-* NLP core models that allow robust extraction of linguistic features for 
-  NLP workflow: for example, dependency parser (BIST) and NP chunker
-* NLU modules that provide best in class performance: for example, intent 
-  extraction (IE), name entity recognition (NER) 
-* Modules that address semantic understanding: for example, colocations, most 
-  common word sense, NP embedding representation (e.g. NP2V)
-* Components instrumental for conversational AI: for example, ChatBot 
-  applications, including dialog system, sequence chunking and IE 
-* End-to-end DL applications using new topologies: for example, Q&A, machine 
-  reading comprehension
-
-<center> <img src="doc/source/assets/nlp_archtiect_v0.1.png"></center>
-
-The library consists of core modules (topologies), data pipelines, utilities 
-and end-to-end model examples with training and inference scripts. We look at 
-these as a set of building blocks that were needed for implementing NLP use 
-cases based on our pragmatic research experience. Each of the models includes 
-algorithm descriptions and results in the [documentation].
-
-Some of the components, with provided pre-trained models, are exposed as REST 
-service APIs through NLP Architect server. NLP Architect server is designed to 
-provide predictions across different models in NLP Architect. It also includes 
-a web front-end exposing the model annotations for visualizations. Currently, 
-we provide two services: BIST dependency parsing and NER annotations. We also 
-provide a template for developers to add a new service.
-
-<center> <img src="doc/source/assets/bist_service.png"></center>
-
-
-Because of its current research nature, several open source deep learning 
-frameworks are used in this repository including:
-
-* [Intel® neon]
-* [Intel® nGraph Python API]
-* [TensorFlow]
-* [Intel-Optimized TensorFlow]
-* [Keras]
-* [Dynet]
-
-Over time, the list of models included in this space will change; though all 
-generally run with Python 3.5+.
-
 ## Documentation
-Framework documentation on NLP models, algorithms, and modules, and instructions 
+Framework documentation on NLP models, algorithms, and modules, and instructions
 on how to contribute can be found at our main [documentation] site.
 
 ## Installation
-Installing NLP Architect within a virtual environment to ensure a self-contained 
-environment is recommended. To install NLP Architect within a new or already-existing 
-virtual environment, or to install it system-wide, see the "Custom installations" 
+Installing NLP Architect within a virtual environment to ensure a self-contained
+environment is recommended. To install NLP Architect within a new or already-existing
+virtual environment, or to install it system-wide, see the "Custom installations"
 section below.  
 
 
@@ -72,10 +26,10 @@ To get started, clone our repository:
     $ git clone https://github.com/NervanaSystems/nlp-architect.git
     $ cd nlp-architect
 
-Note that the `setuptools` package from a recent version of `pip` is needed to 
+Note that the `setuptools` package from a recent version of `pip` is needed to
 get the `make` command to build properly.
 
-    $ pip3 install setuptools  
+    $ pip3 install -U setuptools  
 
 
 Install in development mode (default):
@@ -90,18 +44,18 @@ Activate the newly-created virtual environment:
 
 Fire up your favorite IDE/text editor/terminal and start running models.
 
-### Custom installations 
+### Custom installations
 
 
-* **Python 3 `venv`**  The default installation instructions create a venv named 
-  `.nlp_architect_env`.  If this is not ideal for your needs, (for example, 
-  if you want to name it something else or use an already-existing virtual env), 
-  simply create or enter the local virtual environment you want to use, activate 
+* **Python 3 `venv`**  The default installation instructions create a venv named
+  `.nlp_architect_env`.  If this is not ideal for your needs, (for example,
+  if you want to name it something else or use an already-existing virtual env),
+  simply create or enter the local virtual environment you want to use, activate
   it, and install the library in development mode with:
 
     $ (py3_venv) make install_no_virt_env
 
-  Note that `setuptools` from a recent version of `pip` is needed to get the 
+  **Note** that `setuptools` from a recent version of `pip` is needed to get the
   `make` command to build properly.  
 
 
@@ -110,15 +64,71 @@ Fire up your favorite IDE/text editor/terminal and start running models.
     $ make sysinstall
 
 
+_Note that all installations use CPU-based installations of Tensorflow/Dynet/Neon/nGraph. To install GPU supported binaries please refer to the framework's website for installation instructions._
+
+## NLP Architect overview
+
+The current version of NLP Architect includes these features that we found
+interesting from both research perspectives and practical applications:
+
+* NLP core models and NLU modules that provide best in class performance: Intent
+  Extraction (IE), Name Entity Recognition (NER), Word Chunker, Dependency parser (BIST)
+* Modules that address semantic understanding: co-locations, most
+  common word sense, NP embedding representation (NP2Vec)
+* Components instrumental for conversational AI: ChatBot
+  applications (Memory Networks for Dialog, Key-Value Memory Networks), Noun Phrase extraction
+* End-to-end DL applications using new topologies: Q&A, machine
+  reading comprehension, Language modeling using Temporal Convolution
+  Networks (TCN), Unsupervised Cross-lingual embeddings.
+* Solutions using one or more models: Set Term expansion which
+  uses the included word chunker as a noun phrase extractor and NP2Vec.
+
+
+<center> <img src="doc/source/assets/nlp_architect_diag.png"></center>
+
+The library consists of core modules (topologies), data pipelines, utilities
+and end-to-end model examples with training and inference scripts. We look at
+these as a set of building blocks that were needed for implementing NLP use
+cases based on our pragmatic research experience. Each of the models includes
+algorithm descriptions and results in the [documentation].
+
+Some of the components, with provided pre-trained models, are exposed as REST
+service APIs through NLP Architect server. NLP Architect server is designed to
+provide predictions across different models in NLP Architect. It also includes
+a web front-end exposing the model annotations for visualizations. The server
+supports extensions via a template for developers to add a new service. For
+detailed documentation see this
+[page](http://nlp_architect.nervanasys.com/service.html).
+
+See and example of the BIST parser model UI:
+
+<center> <img src="doc/source/assets/bist_service.png"></center>
+
+### Deep Learning Frameworks
+Because of its current research nature, several open source deep learning
+frameworks are used in this repository including:
+
+* [Intel® neon]
+* [Intel® nGraph Python API]
+* [TensorFlow]
+* [Intel-Optimized TensorFlow]
+* [Keras]
+* [Dynet]
+
+NLP Architect is an active space of research and development; Throughout future
+releases new models, solutions, topologies and framework additions and changes
+will be made. We aim to make sure all models run with Python 3.5+. We
+encourage researchers and developers to contribute their work into the library.
+
 ## Disclaimer
-The NLP Architect is released as reference code for research purposes. It is 
-not an official Intel product, and the level of quality and support may not be 
-as expected from an official product. Additional algorithms and environments are 
-planned to be added to the framework. Feedback and contributions from the open 
+The NLP Architect is released as reference code for research purposes. It is
+not an official Intel product, and the level of quality and support may not be
+as expected from an official product. Additional algorithms and environments are
+planned to be added to the framework. Feedback and contributions from the open
 source and NLP research communities are more than welcome.
 
 ## Contact
-Contact the NLP Architect development team through Github issues or 
+Contact the NLP Architect development team through Github issues or
 email: [email protected]
 
 
@@ -127,6 +137,6 @@ email: [email protected]
 [Intel® nGraph Python API]:http://ngraph.nervanasys.com/docs/latest/python_api/index.html
 [Intel-Optimized TensorFlow]:https://software.intel.com/en-us/articles/intel-optimized-tensorflow-wheel-now-available
 [Intel® neon]:https://github.com/NervanaSystems/neon
-[TensorFlow]:https://www.tensorflow.org/ 
+[TensorFlow]:https://www.tensorflow.org/
 [Keras]:https://keras.io/
 [Dynet]:https://dynet.readthedocs.io/en/latest/

README.md

@@ -25,71 +25,71 @@ Contribution Process
 2. Clone and/or update your checked out copy of nlp-achitect to ensure you have the
    most recent commits from the master branch:
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    git clone https://github.com/NervanaSystems/nlp-architect.git
-    cd nlp-architect
-    git fetch origin
-    git checkout master
-    git pull
+      git clone https://github.com/NervanaSystems/nlp-architect.git
+      cd nlp-architect
+      git fetch origin
+      git checkout master
+      git pull
 
 3. Create a new feature branch for your work and switch to it.  Give it a
    meaningful name related to the task(s) at hand:
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    git checkout -b my_new_feature_branch
+      git checkout -b my_new_feature_branch
 
 4. Ideally you'd start by creating one or more unit tests with the
    functionality you expect your new feature to perform.  These should reside
    under the appropriate tests subdirectory of whatever you are changing.
    Then hack away at the code until you feel your feature is complete.  Once
    satisfied, run the code through the following checks:
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    make test   # ensure all are OK
-    make style  # ensure there are no style related issues
-    make lint   # (optional).  We still have a fair bit to clean up currently!
+      make test   # ensure all are OK
+      make style  # ensure there are no style related issues
+      make lint   # (optional).  We still have a fair bit to clean up currently!
 
 5. If necessary you may want to update and/or rebuild the documentation.
    This all exists under doc/source and is in
    `Sphinx reStructuredText format <http://sphinx-doc.org/rest.html>`_:
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    make html  # builds docs locally, starts a webserver so you can view
-    or
-    make doc   # builds the doc and starts a local server directly
+      make html  # builds docs locally, starts a webserver so you can view
+      or
+      make doc   # builds the doc and starts a local server directly
 
 6. Commit your changes and push your feature branch to your github fork.  Be
    sure to add a descriptive message and reference the github issue associated
    with your task (ex. #1).  You will also want to rebase your commits down to
    a single sensible commit to make things clean for the merge process:
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    git add my_updated_files
-    git commit -m "Added new awesome functionality.  Closes issue #1"
-    git push origin my_new_feature_branch
+      git add my_updated_files
+      git commit -m "Added new awesome functionality.  Closes issue #1"
+      git push origin my_new_feature_branch
 
 7. Create a new pull request to get your feature branch merged into master for
    others to use.  You'll first need to ensure your feature branch contains the
    latest changes from master.  Furthermore, internal devs will need to assign
    the request to someone else for a code review.  You must also ensure there
    are no errors when run through the items defined in step 4.
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    # (external contribs): make a new pull request:
-    https://github.com/NervanaSystems/nlp-architect/pulls
+      # (external contribs): make a new pull request:
+      https://github.com/NervanaSystems/nlp-architect/pulls
 
-    # merge latest master changes into your feature branch
-    git fetch origin
-    git checkout master
-    git pull origin master
-    git checkout my_new_feature_branch
-    git merge master  # you may need to manually resolve any merge conflicts
+      # merge latest master changes into your feature branch
+      git fetch origin
+      git checkout master
+      git pull origin master
+      git checkout my_new_feature_branch
+      git merge master  # you may need to manually resolve any merge conflicts
 
 8. If there are issues you can continue to push commits to your feature branch
    by following step 6.  They will automatically be added to this same merge
@@ -98,12 +98,12 @@ Contribution Process
 9. Once your change has been successfully merged, you can remove the source
    branch and ensure your local copy is up to date:
 
-.. code-block:: bash
+  .. code-block:: bash
 
-    git fetch origin
-    git checkout master
-    git pull
-    git branch -d my_new_feature_branch
-    git branch -d -r origin/my_new_feature_branch
+      git fetch origin
+      git checkout master
+      git pull
+      git branch -d my_new_feature_branch
+      git branch -d -r origin/my_new_feature_branch
 
 10. Give yourself a high five for a job well done!