Merge branch 'master' into alon/master_local

Author: Alon Eirew

.gitignore

@@ -1,36 +1,28 @@
-*.sublime-project
-*.sublime-workspace
 *.pyc
 *.pkl
 *.prm
 .DS_Store
-*@eaDir
 .pkgs
 *.egg-info
 .venv
-.venv[23]
 .styleenv
 .coverage
-build
+/build/
 generated
-*.ropeproject
-*.cubin
+/dist/
 *.hdf5
 *.h5
-*.html
-!solutions/set_expansion/ui/templates/*.html
+#!nlp_architect/server/angular-ui/dist/angular-ui/*.html
+#*.html
+doc/build/**
+!nlp_architect/solutions/set_expansion/ui/templates/*.html
 .vscode
 !tests/fixtures/data/server/*.gz
-!server/angular-ui/src/
-!server/angular-ui/dist/**/*.html
 *.log
 .idea/
 .nlp_architect_env/
 src/
 pylint.html
 pylint.txt
 flake8.txt
-nlp_architect/pipelines/bist-pretrained/*
-venv
-nlp_architect/api/ner-pretrained/*
-tests/fixtures/data/chunker/
+tests/fixtures/data/chunker/(

README.md

@@ -1,4 +1,4 @@
-<p align="center"><img src="doc/source/assets/nlp_architect_header.png" width="400"/></p>
+<p align="center"><img src="https://raw.githubusercontent.com/NervanaSystems/nlp-architect/master/doc/source/assets/nlp_architect_header.png" width="400"/></p>
 <p align="center">
 <a href="https://github.com/NervanaSystems/nlp-architect/blob/master/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg"/></a> <a href="http://nlp_architect.nervanasys.com"><img src="https://img.shields.io/readthedocs/pip/stable.svg"/></a> <a href="https://doi.org/10.5281/zenodo.1477518"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.1477518.svg" alt="DOI"></a> <a href="https://github.com/NervanaSystems/nlp-architect/blob/master/LICENSE"><img src="https://img.shields.io/badge/release-v0.3-blue.svg"/></a>
 </p>
@@ -27,7 +27,7 @@ interesting from both research perspectives and practical applications:
 * Solutions using one or more models: Set Term expansion which
   uses the included word chunker as a noun phrase extractor and NP2Vec, Topics and trend analysis for analyzing temporal corpora.
 
-<center> <img src="doc/source/assets/nlp_architect_diag.png"></center>
+<center> <img src="https://raw.githubusercontent.com/NervanaSystems/nlp-architect/master/doc/source/assets/nlp_architect_diag.png"></center>
 
 The library consists of core modules (topologies), data pipelines, utilities
 and end-to-end model examples with training and inference scripts. We look at
@@ -44,7 +44,7 @@ detailed documentation see this
 [page](http://nlp_architect.nervanasys.com/service.html).
 
 NLP Architect server in action
-<center> <img src="doc/source/assets/service_cards.png"></center>
+<center> <img src="https://raw.githubusercontent.com/NervanaSystems/nlp-architect/master/doc/source/assets/service_cards.png"></center>
 
 NLP Architect utilizes the following open source deep learning frameworks:
 
@@ -58,41 +58,40 @@ on how to contribute can be found at our main [documentation] site.
 
 ## Installation
 ### Prerequisites
-To get started, clone our repository:
 
-    git clone https://github.com/NervanaSystems/nlp-architect.git
-    cd nlp-architect
+Make sure `pip` and `setuptools` and `venv` are up to date before installing.
+
+    pip3 install -U pip setuptools
+
+We recommend installing NLP Architect in a virtual environment to self-contain
+the work done using the library. 
 
-Installing NLP Architect within a virtual environment to ensure a self-contained
-environment is recommended.
 To create and activate a new virtual environment:
 
     python3 -m venv .nlp_architect_env
     source .nlp_architect_env/bin/activate
 
-Make sure `pip` and `setuptools` and `venv` are up to date before installing.
-
-    pip3 install -U pip setuptools
+### Installing using `pip`
 
-### Backend
+To install NLP Architect using `pip` package manager:
 
-NLP Architect supports CPU, GPU and Intel Optimized Tensorflow (MKL-DNN).
-Select the desired backend using a dedicated environment variable.
+    pip install nlp-architect
+    
+### Installing from source
 
-CPU (default):
-
-    export NLP_ARCHITECT_BE=CPU
-
-MKL-DNN (Linux OS only):
+To get started, clone our repository:
 
-    export NLP_ARCHITECT_BE=MKL
+    git clone https://github.com/NervanaSystems/nlp-architect.git
+    cd nlp-architect
 
-GPU (Linux OS only):
+#### Selecting a backend
 
-    export NLP_ARCHITECT_BE=GPU
+NLP Architect supports CPU, GPU and Intel Optimized Tensorflow (MKL-DNN) backends.
+Users can select the desired backend using a dedicated environment variable (default: CPU). (MKL-DNN and GPU backends are supported only on Linux)
 
+    export NLP_ARCHITECT_BE=CPU/MKL/GPU
 
-### Installation
+#### Installation
 NLP Architect is installed using `pip` and it is recommended to install in development mode.
 
 Default:
@@ -115,23 +114,11 @@ Once installed, the `nlp_architect` command provides additional options to work
 | nlp_architect.data      	| Datasets, data loaders and data classes              	|
 | nlp_architect.models    	| NLP, NLU and End-to-End neural models                	|
 | nlp_architect.pipelines 	| End-to-end NLP apps                                  	|
+| nlp_architect.server     	| API Server and demos UI                              	|
+| nlp_architect.solutions 	| Solution applications                                	|
 | nlp_architect.utils     	| Misc. I/O, metric, pre-processing and text utilities 	|
 | examples                	| Example files for each model                         	|
-| solutions               	| Solution applications                                	|
-| server                  	| API Server and demos UI                              	|
 | tutorials               	| Misc. Jupyter tutorials                              	|
-<!--
-Below are examples of NLP Architect server in action
-
-BIST Parser UI
-<center> <img src="doc/source/assets/bist_service.png"></center>
-
-NER Parser UI
-<center> <img src="doc/source/assets/ner_service.png"></center>
-
-Spacy NER Parser UI
-<center> <img src="doc/source/assets/spacy_ner_service.png"></center>-->
-
 
 NLP Architect is an active space of research and development; Throughout future
 releases new models, solutions, topologies and framework additions and changes

doc/source/_quick_install.rst

@@ -14,8 +14,8 @@
 .. limitations under the License.
 .. ---------------------------------------------------------------------------
 
-Quick Installation wizard
--------------------------
+Quick Install
+-------------
 Select the desired configuration of your system:
 
 .. raw:: html
@@ -26,6 +26,18 @@ Select the desired configuration of your system:
         <col class="description">
         </colgroup><tbody valign="top">
         <tr><td class="option-group">
+        <kbd><span class="option">
+            <strong>Install from</strong>
+        </span></kbd></td>
+        <td>
+            <label class="radio">
+              <input v-model="form.source" type="radio" value="1">Pip
+            </label>
+            <label class="radio">
+              <input v-model="form.source" type="radio" value="0" checked>GitHub
+            </label>
+        </td></tr>
+        <tr><td class="option-group">
         <kbd><span class="option">
             <strong>Create virtualenv?</strong>
         </span></kbd></td>
@@ -72,9 +84,7 @@ Run the following commands to install NLP Architect:
 .. raw:: html
 
     <div class="code python highlight-default notranslate"><div class="highlight">
-    <pre><span class="n" v-html="get_env()"></span>
-    <span class="n">export NLP_ARCHITECT_BE={{ get_be() }}</span>
-    <span class="n">pip3 install {{ get_mode() }}.</span>
+    <pre v-html="get_commands()">
     </pre></div>
     </div>
     <script src="https://cdn.jsdelivr.net/npm/vue"></script>

doc/source/api.rst

@@ -94,7 +94,7 @@ Dataset implementations and data loaders (check deep learning framework compatib
 ---------------------------
 .. currentmodule:: nlp_architect.pipelines
 
-NLP pipelines modules using models implemented from ``nlp_architect.models``
+NLP pipelines modules using NLP Architect models
 
 .. autosummary::
     :toctree: generated/
@@ -132,5 +132,7 @@ Common types of data structures used by NLP models
     :toctree: generated/
     :nosignatures:
 
+    nlp_architect.common.core_nlp_doc.CoreNLPDoc
+    nlp_architect.common.high_level_doc.HighLevelDoc
     nlp_architect.common.cdc.mention_data.MentionDataLight
     nlp_architect.common.cdc.mention_data.MentionData

doc/source/chunker.rst

@@ -126,7 +126,7 @@ Running inference on a trained model using an input file (text based, each line
 
 .. code:: python
 
-	python examples/chunker/inference.py --model_name <model_name> --input <input_file>.txt
+    python examples/chunker/inference.py --model_name <model_name> --input <input_file>.txt
 
 
 .. _CONLL2000: https://www.clips.uantwerpen.be/conll2000/chunking/

doc/source/code_docs/nlp_architect.api.rst

@@ -23,4 +23,6 @@
 
 .. autoclass:: nlp_architect.api.ner_api.NerApi
 
-.. autoclass:: nlp_architect.api.spacy_ner_api.SpacyNerApi
+.. autoclass:: nlp_architect.api.intent_extraction.IntentExtractionApi
+
+.. autoclass:: nlp_architect.api.machine_comprehension.MachineComprehensionApi

doc/source/code_docs/service.rst

@@ -17,7 +17,7 @@
 Service class documentation
 ###########################
 
-.. autoclass:: server.service.Service
+.. autoclass:: nlp_architect.server.service.Service
 
 .. autoclass:: nlp_architect.common.core_nlp_doc.CoreNLPDoc
 

doc/source/conf.py

@@ -20,7 +20,7 @@
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
-from nlp_architect.version import nlp_architect_version
+from nlp_architect.version import NLP_ARCHITECT_VERSION
 
 sys.path.insert(0, os.path.abspath('../..'))
 
@@ -87,7 +87,7 @@
 # built documents.
 #
 # The short X.Y.Z version.
-version = nlp_architect_version()
+version = NLP_ARCHITECT_VERSION
 # The full version, including git-hash and alpha/beta/rc tags.
 release = version
 

doc/source/cross_doc_coref.rst

@@ -49,6 +49,7 @@ The system achieved the following:
 * Best in class results achieve on ECB+ Event Cross Document Co-Reference (**79.0% F1**) using the sieves set *[Head Lemma, Exact Match, Wikipedia Redirect, Wikipedia Disambiguation and Fuzzy Head]*
 
 .. [1] ECB+: Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution.
+
 In Proceedings of the 9th international conference on Language Resources and Evaluation (LREC2014)
 ECB+ annotation is held copyright by Agata Cybulska, Piek Vossen and the VU University of Amsterdam.
 

doc/source/identifying_semantic_relation.rst

@@ -270,35 +270,38 @@ Generate Scripts
 
 ::
 
-    python nlp_architect/data/cdc_resources/gen_scripts/create_reference_dict_dump.py --ref_dict=<ref.dict1.tsv downloaded file> --mentions=<in_mentions.json> --output=<output.json>
+    python -m nlp_architect.data.cdc_resources.gen_scripts.create_reference_dict_dump.py --ref_dict=<ref.dict1.tsv downloaded file> --mentions=<in_mentions.json> --output=<output.json>
 
 **Generate VerbOcean:**
 
 ::
 
-    python nlp_architect/data/cdc_resources/gen_scripts/create_verbocean_dump.py --vo=<verbocean.unrefined.2004-05-20.txt downloaded file> --mentions=<in_mentions.json> --output=<output.json>
+    python -m nlp_architect.data.cdc_resources.gen_scripts.create_verbocean_dump.py --vo=<verbocean.unrefined.2004-05-20.txt downloaded file> --mentions=<in_mentions.json> --output=<output.json>
 
 **Generate WordEmbedding Glove:**
 
 ::
 
-    python nlp_architect/data/cdc_resources/gen_scripts/create_word_embed_glove_dump.py --mentions=<in_mentions.json> --glove=glove.840B.300d.txt --output=<output.pickle>
+    python -m nlp_architect.data.cdc_resources.gen_scripts.create_word_embed_glove_dump.py --mentions=<in_mentions.json> --glove=glove.840B.300d.txt --output=<output.pickle>
 
 **Generate Wordnet:**
 
 ::
 
-    python nlp_architect/data/cdc_resources/gen_scripts/create_wordnet_dump.py --mentions=<in_mentions.json> --output=<output.json>
+    python -m nlp_architect.data.cdc_resources.gen_scripts.create_wordnet_dump.py --mentions=<in_mentions.json> --output=<output.json>
 
 **Generate Wikipedia:**
 
 ::
 
-    python nlp_architect/data/cdc_resources/gen_scripts/create_wiki_dump.py --mentions=<in_mentions.json> --output=<output.json>``
+    python -m nlp_architect.data.cdc_resources.gen_scripts.create_wiki_dump.py --mentions=<in_mentions.json> --output=<output.json>``
 
 .. note::
+
      **For a fast evaluation using Wikipedia at run time**, on live data, there is an option to generate a local ElasticSearch database of the entire Wiki site using this resource: `Wiki to Elastic <https://github.com/AlonEirew/wikipedia-to-elastic/>`_, It is highly recommended since using online evaluation against Wikipedia site can be very slow.
     In case you adopt elastic local database, Initiate ``WikipediaRelationExtraction`` relation extraction using ``WikipediaSearchMethod.ELASTIC``
  **Generate Wikipedia Snapshot using Elastic data instead of from online wikipedia site:**
- ::
-     python nlp_architect/data/cdc_resources/gen_scripts/create_wiki_dump.py --mentions=<in_mentions.json> --host=<elastic_host eg:localhost> --port=<elastic_port eg:9200> --index=<elastic_index> --output=<output.json>``
+
+::
+
+     python -m nlp_architect.data.cdc_resources.gen_scripts.create_wiki_dump.py --mentions=<in_mentions.json> --host=<elastic_host eg:localhost> --port=<elastic_port eg:9200> --index=<elastic_index> --output=<output.json>``