Skip to content
This repository was archived by the owner on Nov 8, 2022. It is now read-only.

Commit 2723c9b

Browse files
peterizchinnikrishna2231
peteriz
authored and
chinnikrishna2231
committed
Installation 0.3 (#319)
* New installation and command app * Added installation wizard * header * Update README.md * Removed keras/neon/ngraph and old models * update setup * New installation and command app Added installation wizard header Update README.md Removed keras/neon/ngraph and old models update setup * updated readme * Update documentation * new installation * added publication * merged overview with index * import fixes * updated contribution doc * updates + jenkins * Added installation wizard header Update README.md Removed keras/neon/ngraph and old models update setup updated readme Update documentation * new installation * added publication * merged overview with index * import fixes * updated contribution doc updates + jenkins * jenkins * let make this work * jenkins * Fix setup.py * minor * jenkins * fix open file * fix * fix nlp_architect script * minor fix * fix jenkins * fix exit codes * Added new server UI images, changes to README * Added deployment tutorial * Clear picture * fix * fixed spacy import * updated pre-trained models tests * flake error fix * fixed tests * flake errors * Added correct cards picture to service.rst
1 parent 70efad8 commit 2723c9b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+779
-2371
lines changed

‎.jenkins/Jenkinsfile‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,14 @@ timestamps {
5656
// Build using Makefile
5757
sh 'python3 --version'
5858
sh 'python3 -V -V'
59-
sh 'make clean'
60-
sh 'make'
59+
sh 'python3 -m venv .nlp_architect_env'
60+
sh """
61+
. .nlp_architect_env/bin/activate
62+
pip3 install -U pip
63+
pip3 install -U h5py
64+
export NLP_ARCHITECT_BE=CPU
65+
pip3 install -e .
66+
"""
6167
}
6268

6369
// Wrap this stage in a try/catch to prevent Jenkins from fast failing
@@ -69,7 +75,10 @@ timestamps {
6975
// Run tests using Makefile
7076
// TODO: Leverage Jenkins J/XUnit plugins to scan test results
7177
// by exporting test results as a known J/XUnit style artifact
72-
sh 'make test'
78+
sh """
79+
. .nlp_architect_env/bin/activate
80+
nlp_architect test
81+
"""
7382
}
7483
} catch (err) {
7584
echo "BUILD FAILURE - Caught Exception: ${err}"
@@ -82,23 +91,14 @@ timestamps {
8291
// however with this call wrapped in the below, we ignore the exception but we flag
8392
// the build as a failure.
8493
try {
85-
stage('Flake8') {
94+
stage('Style') {
8695
// Run style checks using Makefile
8796
// TODO: Leverage Jenkins Warnings plugins to scan for style errors
8897
// by exporting style check results as a known artifact
89-
sh 'make flake'
90-
}
91-
} catch (err) {
92-
echo "BUILD FAILURE - Caught Exception: ${err}"
93-
// Inform Jenkins this run is now in a FAILURE state
94-
currentBuild.result = 'FAILURE'
95-
}
96-
try {
97-
stage('Pylint') {
98-
// Run style checks using Makefile
99-
// TODO: Leverage Jenkins Warnings plugins to scan for style errors
100-
// by exporting style check results as a known artifact
101-
sh 'make pylint'
98+
sh """
99+
. .nlp_architect_env/bin/activate
100+
nlp_architect style
101+
"""
102102
}
103103
} catch (err) {
104104
echo "BUILD FAILURE - Caught Exception: ${err}"

‎MANIFEST.in‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎README.md‎

Lines changed: 85 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -1,105 +1,126 @@
1+
<p align="center"><img src="doc/source/assets/nlp_architect_header.png" width="400"/></p>
2+
<p align="center">
3+
<a href="https://github.com/NervanaSystems/nlp-architect/blob/master/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg"/></a> <a href="http://nlp_architect.nervanasys.com"><img src="https://img.shields.io/readthedocs/pip/stable.svg"/></a> <a href="https://github.com/NervanaSystems/nlp-architect/blob/master/LICENSE"><img src="https://img.shields.io/badge/release-v0.3-blue.svg"/></a>
4+
</p>
15

26
# NLP Architect by Intel® AI LAB
37

4-
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/NervanaSystems/nlp-architect/blob/master/LICENSE)
5-
[![Docs](https://img.shields.io/readthedocs/pip/stable.svg)](http://nlp_architect.nervanasys.com)
6-
78
NLP Architect is an open-source Python library for exploring state-of-the-art
89
deep learning topologies and techniques for natural language processing and
910
natural language understanding. It is intended to be a platform for future
1011
research and collaboration.
1112

12-
## Documentation
13-
Framework documentation on NLP models, algorithms, and modules, and instructions
14-
on how to contribute can be found at our main [documentation] site.
13+
## Overview
1514

16-
## Installation
17-
Installing NLP Architect within a virtual environment to ensure a self-contained
18-
environment is recommended. To install NLP Architect within a new or already-existing
19-
virtual environment, or to install it system-wide, see the "Custom installations"
20-
section below.
15+
The current version of NLP Architect includes these features that we found
16+
interesting from both research perspectives and practical applications:
2117

18+
* NLP core models and NLU modules that provide best in class performance: Intent
19+
Extraction (IE), Name Entity Recognition (NER), Word Chunker, Dependency parser (BIST)
20+
* Modules that address semantic understanding: co-locations, most
21+
common word sense, noun phrase embedding representation (NP2Vec), relation identification and cross document coreference.
22+
* Components instrumental for conversational AI: ChatBot
23+
applications (Memory Networks for Dialog, Key-Value Memory Networks), Intent Extraction.
24+
* End-to-end DL applications using and new topologies: Q&A, Machine
25+
Reading Comprehension, Language modeling using Temporal Convolution
26+
Networks (TCN), Unsupervised Cross-lingual embeddings, Sparse and quantized GNMT.
27+
* Solutions using one or more models: Set Term expansion which
28+
uses the included word chunker as a noun phrase extractor and NP2Vec, Topics and trend analysis for analyzing temporal corpora.
2229

23-
### Default installation (works with Python 3.n+ only)
24-
To get started, clone our repository:
30+
<center> <img src="doc/source/assets/nlp_architect_diag.png"></center>
2531

26-
$ git clone https://github.com/NervanaSystems/nlp-architect.git
27-
$ cd nlp-architect
32+
The library consists of core modules (topologies), data pipelines, utilities
33+
and end-to-end model examples with training and inference scripts. We look at
34+
these as a set of building blocks that were needed for implementing NLP use
35+
cases based on our pragmatic research experience. Each of the models includes
36+
algorithm descriptions and results in the [documentation].
2837

29-
Note that the `setuptools` package from a recent version of `pip` is needed to
30-
get the `make` command to build properly.
38+
Some of the components, with provided pre-trained models, are exposed as REST
39+
service APIs through NLP Architect server. NLP Architect server is designed to
40+
provide predictions across different models in NLP Architect. It also includes
41+
a web front-end exposing the model annotations for visualizations. The server
42+
supports extensions via a template for developers to add a new service. For
43+
detailed documentation see this
44+
[page](http://nlp_architect.nervanasys.com/service.html).
3145

32-
$ pip3 install -U setuptools
46+
NLP Architect server in action
47+
<center> <img src="doc/source/assets/service_cards.png"></center>
3348

49+
NLP Architect utilizes the following open source deep learning frameworks:
3450

35-
Install in development mode (default):
51+
* [TensorFlow]
52+
* [Intel-Optimized TensorFlow with MKL-DNN]
53+
* [Dynet]
3654

37-
$ make
55+
## Documentation
56+
Framework documentation on NLP models, algorithms, and modules, and instructions
57+
on how to contribute can be found at our main [documentation] site.
3858

59+
## Installation
60+
### Prerequisites
61+
To get started, clone our repository:
3962

40-
Activate the newly-created virtual environment:
63+
git clone https://github.com/NervanaSystems/nlp-architect.git
64+
cd nlp-architect
4165

42-
$ . .nlp_architect_env/bin/activate
66+
Installing NLP Architect within a virtual environment to ensure a self-contained
67+
environment is recommended.
68+
To create and activate a new virtual environment:
4369

70+
python3 -m venv .nlp_architect_env
71+
source .nlp_architect_env/bin/activate
4472

45-
Fire up your favorite IDE/text editor/terminal and start running models.
73+
Make sure `pip` and `setuptools` and `venv` are up to date before installing.
4674

47-
### Custom installations
75+
pip3 install -U pip setuptools
4876

77+
### Backend
4978

50-
* **Python 3 `venv`** The default installation instructions create a venv named
51-
`.nlp_architect_env`. If this is not ideal for your needs, (for example,
52-
if you want to name it something else or use an already-existing virtual env),
53-
simply create or enter the local virtual environment you want to use, activate
54-
it, and install the library in development mode with:
79+
NLP Architect supports CPU, GPU and Intel Optimized Tensorflow (MKL-DNN).
80+
Select the desired backend using a dedicated environment variable.
5581

56-
$ (py3_venv) make install_no_virt_env
82+
CPU (default):
5783

58-
**Note** that `setuptools` from a recent version of `pip` is needed to get the
59-
`make` command to build properly.
84+
export NLP_ARCHITECT_BE=CPU
6085

86+
MKL-DNN (Linux OS only):
6187

62-
* **System-wide install** -- A system install might require `sudo` permissions:
88+
export NLP_ARCHITECT_BE=MKL
6389

64-
$ make sysinstall
90+
GPU (Linux OS only):
6591

92+
export NLP_ARCHITECT_BE=GPU
6693

67-
_Note that all installations use CPU-based installations of Tensorflow/Dynet/Neon/nGraph. To install GPU supported binaries please refer to the framework's website for installation instructions._
6894

69-
## NLP Architect overview
95+
### Installation
96+
NLP Architect is installed using `pip` and it is recommended to install in development mode.
7097

71-
The current version of NLP Architect includes these features that we found
72-
interesting from both research perspectives and practical applications:
98+
Default:
7399

74-
* NLP core models and NLU modules that provide best in class performance: Intent
75-
Extraction (IE), Name Entity Recognition (NER), Word Chunker, Dependency parser (BIST)
76-
* Modules that address semantic understanding: co-locations, most
77-
common word sense, NP embedding representation (NP2Vec)
78-
* Components instrumental for conversational AI: ChatBot
79-
applications (Memory Networks for Dialog, Key-Value Memory Networks), Noun Phrase extraction
80-
* End-to-end DL applications using new topologies: Q&A, machine
81-
reading comprehension, Language modeling using Temporal Convolution
82-
Networks (TCN), Unsupervised Cross-lingual embeddings.
83-
* Solutions using one or more models: Set Term expansion which
84-
uses the included word chunker as a noun phrase extractor and NP2Vec.
100+
pip3 install .
85101

102+
Development mode:
86103

87-
<center> <img src="doc/source/assets/nlp_architect_diag.png"></center>
104+
pip3 install -e .
88105

89-
The library consists of core modules (topologies), data pipelines, utilities
90-
and end-to-end model examples with training and inference scripts. We look at
91-
these as a set of building blocks that were needed for implementing NLP use
92-
cases based on our pragmatic research experience. Each of the models includes
93-
algorithm descriptions and results in the [documentation].
106+
Once installed, the `nlp_architect` command provides additional options to work with the library, issue `nlp_architect -h` to see all options.
94107

95-
Some of the components, with provided pre-trained models, are exposed as REST
96-
service APIs through NLP Architect server. NLP Architect server is designed to
97-
provide predictions across different models in NLP Architect. It also includes
98-
a web front-end exposing the model annotations for visualizations. The server
99-
supports extensions via a template for developers to add a new service. For
100-
detailed documentation see this
101-
[page](http://nlp_architect.nervanasys.com/service.html).
108+
## Packages
102109

110+
| Package | Description |
111+
|------------------------- |------------------------------------------------------ |
112+
| nlp_architect.api | Model server API interfaces |
113+
| nlp_architect.common | Common packages |
114+
| nlp_architect.contrib | Framework extensions |
115+
| nlp_architect.data | Datasets, data loaders and data classes |
116+
| nlp_architect.models | NLP, NLU and End-to-End neural models |
117+
| nlp_architect.pipelines | End-to-end NLP apps |
118+
| nlp_architect.utils | Misc. I/O, metric, pre-processing and text utilities |
119+
| examples | Example files for each model |
120+
| solutions | Solution applications |
121+
| server | API Server and demos UI |
122+
| tutorials | Misc. Jupyter tutorials |
123+
<!--
103124
Below are examples of NLP Architect server in action
104125
105126
BIST Parser UI
@@ -109,17 +130,8 @@ NER Parser UI
109130
<center> <img src="doc/source/assets/ner_service.png"></center>
110131
111132
Spacy NER Parser UI
112-
<center> <img src="doc/source/assets/spacy_ner_service.png"></center>
133+
<center> <img src="doc/source/assets/spacy_ner_service.png"></center>-->
113134

114-
### Deep Learning Frameworks
115-
Because of its current research nature, several open source deep learning
116-
frameworks are used in this repository including:
117-
118-
* [Intel® nGraph Python API]
119-
* [TensorFlow]
120-
* [Intel-Optimized TensorFlow]
121-
* [Keras]
122-
* [Dynet]
123135

124136
NLP Architect is an active space of research and development; Throughout future
125137
releases new models, solutions, topologies and framework additions and changes
@@ -137,12 +149,7 @@ source and NLP research communities are more than welcome.
137149
Contact the NLP Architect development team through Github issues or
138150
email: [email protected]
139151

140-
141-
142152
[documentation]:http://nlp_architect.nervanasys.com
143-
[Intel® nGraph Python API]:http://ngraph.nervanasys.com/docs/latest/python_api/index.html
144-
[Intel-Optimized TensorFlow]:https://software.intel.com/en-us/articles/intel-optimized-tensorflow-wheel-now-available
145-
[Intel® neon]:https://github.com/NervanaSystems/neon
153+
[Intel-Optimized TensorFlow with MKL-DNN]:https://software.intel.com/en-us/articles/intel-optimized-tensorflow-wheel-now-available
146154
[TensorFlow]:https://www.tensorflow.org/
147-
[Keras]:https://keras.io/
148155
[Dynet]:https://dynet.readthedocs.io/en/latest/

‎doc/source/CONTRIBUTING.rst‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,19 +48,16 @@ Contribution Process
4848

4949
.. code-block:: bash
5050
51-
make test # ensure all are OK
52-
make style # ensure there are no style related issues
53-
make lint # (optional). We still have a fair bit to clean up currently!
51+
nlp_architect test # ensure all are OK
52+
nlp_architect style # ensure there are no style related issues
5453
5554
5. If necessary you may want to update and/or rebuild the documentation.
5655
This all exists under doc/source and is in
5756
`Sphinx reStructuredText format <http://sphinx-doc.org/rest.html>`_:
5857

5958
.. code-block:: bash
6059
61-
make html # builds docs locally, starts a webserver so you can view
62-
or
63-
make doc # builds the doc and starts a local server directly
60+
nlp_architect doc # builds the doc and starts a local server directly
6461
6562
6. Commit your changes and push your feature branch to your github fork. Be
6663
sure to add a descriptive message and reference the github issue associated

‎doc/source/_quick_install.rst‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
.. ---------------------------------------------------------------------------
2+
.. Copyright 2017-2018 Intel Corporation
3+
..
4+
.. Licensed under the Apache License, Version 2.0 (the "License");
5+
.. you may not use this file except in compliance with the License.
6+
.. You may obtain a copy of the License at
7+
..
8+
.. http://www.apache.org/licenses/LICENSE-2.0
9+
..
10+
.. Unless required by applicable law or agreed to in writing, software
11+
.. distributed under the License is distributed on an "AS IS" BASIS,
12+
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
.. See the License for the specific language governing permissions and
14+
.. limitations under the License.
15+
.. ---------------------------------------------------------------------------
16+
17+
Quick Installation wizard
18+
-------------------------
19+
Select the desired configuration of your system:
20+
21+
.. raw:: html
22+
23+
<div id="app" class="wy-table-responsive">
24+
<table class="docutils option-list installation_table" frame="void" rules="none">
25+
<colgroup><col class="option">
26+
<col class="description">
27+
</colgroup><tbody valign="top">
28+
<tr><td class="option-group">
29+
<kbd><span class="option">
30+
<strong>Create virtualenv?</strong>
31+
</span></kbd></td>
32+
<td>
33+
<label class="radio">
34+
<input v-model="form.with_env" type="radio" value="1">Yes
35+
</label>
36+
<label class="radio">
37+
<input v-model="form.with_env" type="radio" value="0" checked>No
38+
</label>
39+
</td></tr>
40+
<tr><td class="option-group">
41+
<kbd><span class="option">
42+
<strong>Backend</strong>
43+
</span></kbd></td>
44+
<td>
45+
<label class="radio">
46+
<input v-model="form.backend" type="radio" value="CPU">CPU
47+
</label>
48+
<label class="radio">
49+
<input v-model="form.backend" type="radio" value="MKL" checked>MKL
50+
</label>
51+
<label class="radio">
52+
<input v-model="form.backend" type="radio" value="GPU" checked>GPU
53+
</label>
54+
</td></tr>
55+
<tr><td class="option-group">
56+
<kbd><span class="option">
57+
<strong>Install in developer mode?</strong>
58+
</span></kbd></td>
59+
<td>
60+
<label class="radio">
61+
<input v-model="form.inst_type" type="radio" value="0">Yes
62+
</label>
63+
<label class="radio">
64+
<input v-model="form.inst_type" type="radio" value="1" checked>No
65+
</label>
66+
</td></tr>
67+
</tbody>
68+
</table>
69+
70+
Run the following commands to install NLP Architect:
71+
72+
.. raw:: html
73+
74+
<div class="code python highlight-default notranslate"><div class="highlight">
75+
<pre><span class="n" v-html="get_env()"></span>
76+
<span class="n">export NLP_ARCHITECT_BE={{ get_be() }}</span>
77+
<span class="n">pip3 install {{ get_mode() }}.</span>
78+
</pre></div>
79+
</div>
80+
<script src="https://cdn.jsdelivr.net/npm/vue"></script>
81+
<script src="_static/install.js"></script>
82+
83+
It is recommended to install NLP Architect in development mode to utilize all its features, examples and solutions.

0 commit comments

Comments
 (0)