Merge branch 'master' into dev-frontend

Author: Hironsan

ISSUE_TEMPLATE.md renamed to .github/ISSUE_TEMPLATE.md

@@ -0,0 +1,21 @@
+---
+name: "\U0001F4AC Question"
+about: For the question to confirm suspiciously behaviors or feature usage. Please use StackOverflow if your question is general usage or help with your environment
+
+---
+
+How to reproduce the behaviour
+---------
+<!-- Before submitting an issue, make sure to check the docs and closed issues and FAQ to see if any of the solutions work for you. https://github.com/chakki-works/doccano/wiki/Frequently-Asked-Questions -->
+
+<!--
+Include a code example or the steps that led to the problem. Please try to be as specific as possible. -->
+
+Your Environment
+---------
+<!-- Include details of your environment. -->
+
+*   Operating System:
+*   Python Version Used:
+*   When you install doccano:
+*   How did you install doccano (Heroku button etc):

.github/ISSUE_TEMPLATE/01-question.md

@@ -0,0 +1,19 @@
+---
+name: "\U0001F6A8 Bug Report"
+about: For the bug report or unexpected behavior differing from the docs
+
+---
+
+How to reproduce the behaviour
+---------
+<!-- Before submitting an issue, make sure to check the docs and closed issues and FAQ to see if any of the solutions work for you. https://github.com/chakki-works/doccano/wiki/Frequently-Asked-Questions -->
+
+<!-- Include a code example or the steps that led to the problem. Please try to be as specific as possible. -->
+
+Your Environment
+---------
+<!-- Include details of your environment.-->
+*   Operating System:
+*   Python Version Used:
+*   When you install doccano:
+*   How did you install doccano (Heroku button etc):

.github/ISSUE_TEMPLATE/02-bug.md

@@ -0,0 +1,23 @@
+---
+name: "\U000023F3 Installation Problem"
+about: For the problem that you faced when installing doccano which none of the suggestions in the docs and other issues helped
+
+---
+
+<!-- Before submitting an issue, make sure to check the docs and closed issues and FAQ to see if any of the solutions work for you. https://github.com/chakki-works/doccano/wiki/Frequently-Asked-Questions -->
+
+How to reproduce the problem
+---------
+<!-- Include the details of how the problem occurred. Which option did you choose to install doccano? Did you come across an error? What else did you try? -->
+
+```bash
+# copy-paste the error message here
+```
+
+Your Environment
+---------
+<!-- Include details of your environment.-->
+*   Operating System:
+*   Python Version Used:
+*   When you install doccano:
+*   How did you install doccano (Heroku button etc):

.github/ISSUE_TEMPLATE/03-install.md

@@ -0,0 +1,9 @@
+---
+name: "\U0001F381 Feature Request"
+about: For the proposal to improve or enhance doccano
+
+---
+
+Feature description
+---------
+<!-- Please describe the feature: Which area of the library is it related to? What specific solution would you like? -->

.github/ISSUE_TEMPLATE/04-request.md

@@ -199,4 +199,4 @@ node_modules/
 bundle/
 webpack-stats.json
 
-.vscode/
+.vscode

.gitignore

@@ -41,16 +41,19 @@ deploy:
   - provider: script
     script: tools/cd.sh travis-${TRAVIS_BUILD_NUMBER}
     on:
+      repo: chakki-works/doccano
       branch: master
 
   - provider: script
     script: tools/cd.sh ${TRAVIS_TAG}
     on:
+      repo: chakki-works/doccano
       tags: true
 
   - provider: pages
     skip_cleanup: true
     github_token: $GITHUB_TOKEN
     local_dir: site
     on:
+        repo: chakki-works/doccano
         branch: master

.travis.yml

@@ -4,7 +4,7 @@ FROM python:${PYTHON_VERSION}-stretch AS builder
 ARG NODE_VERSION="8.x"
 RUN curl -sL "https://deb.nodesource.com/setup_${NODE_VERSION}" | bash - \
  && apt-get install --no-install-recommends -y \
-      nodejs=8.16.0-1nodesource1
+      nodejs
 
 COPY tools/install-mssql.sh /doccano/tools/install-mssql.sh
 RUN /doccano/tools/install-mssql.sh --dev

Dockerfile

@@ -3,27 +3,27 @@
 [![Codacy Badge](https://api.codacy.com/project/badge/Grade/98a0992c0a254d0ba23fd75631fe2907)](https://app.codacy.com/app/Hironsan/doccano?utm_source=github.com&utm_medium=referral&utm_content=chakki-works/doccano&utm_campaign=Badge_Grade_Dashboard)
 [![Build Status](https://travis-ci.org/chakki-works/doccano.svg?branch=master)](https://travis-ci.org/chakki-works/doccano)
 
-doccano is an open source text annotation tool for human. It provides annotation features for text classification, sequence labeling and sequence to sequence. So, you can create labeled data for sentiment analysis, named entity recognition, text summarization and so on. Just create project, upload data and start annotation. You can build dataset in hours.
+doccano is an open source text annotation tool for humans. It provides annotation features for text classification, sequence labeling and sequence to sequence tasks. So, you can create labeled data for sentiment analysis, named entity recognition, text summarization and so on. Just create a project, upload data and start annotating. You can build a dataset in hours.
 
 ## Demo
 
-You can enjoy [annotation demo](http://doccano.herokuapp.com).
+You can try the [annotation demo](http://doccano.herokuapp.com).
 
 ### [Named entity recognition](https://doccano.herokuapp.com/demo/named-entity-recognition/)
 
-First demo is one of the sequence labeling tasks, named-entity recognition. You just select text spans and annotate it. Since doccano supports shortcut key, so you can quickly annotate text spans.
+The first demo is a sequence labeling task: named-entity recognition. You just select text spans and annotate them. Doccano supports shortcut keys, so you can quickly annotate text spans.
 
 ![Named Entity Recognition](./docs/named_entity_annotation.gif)
 
 ### [Sentiment analysis](https://doccano.herokuapp.com/demo/text-classification/)
 
-Second demo is one of the text classification tasks, topic classification. Since there may be more than one category, you can annotate multi-labels.
+The second demo is a text classification task: sentiment analysis. Since there may be more than one category, you can annotate with multiple labels.
 
 ![Text Classification](./docs/text_classification.gif)
 
 ### [Machine translation](https://doccano.herokuapp.com/demo/translation/)
 
-Final demo is one of the sequence to sequence tasks, machine translation. Since there may be more than one responses in sequence to sequence tasks, you can create multi responses.
+The final demo is a sequence to sequence task: machine translation. Since there may be more than one response in sequence to sequence tasks, you can create multiple responses.
 
 ![Machine Translation](./docs/translation.gif)
 
@@ -54,43 +54,52 @@ git push heroku master
 
 Doccano can be deployed to AWS ([Cloudformation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html)) by clicking on the button below:
 
-[![AWS CloudFormation Launch Stack SVG Button](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3-external-1.amazonaws.com/cf-templates-10vry9l3mp71r-us-east-1/20190732wl-new.templatexloywxxyimi&stackName=doccano)
+[![AWS CloudFormation Launch Stack SVG Button](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home?#/stacks/create/review?stackName=doccano&templateURL=https://s3-external-1.amazonaws.com/cf-templates-10vry9l3mp71r-us-east-1/2019290i9t-AppSGl1poo4j8qpq)
 
 > Notice: (1) EC2 KeyPair cannot be created automatically, so make sure you have an existing EC2 KeyPair in one region. Or [create one yourself](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair). (2) If you want to access doccano via HTTPS in AWS, here is an [instruction](https://github.com/chakki-works/doccano/wiki/HTTPS-setting-for-doccano-in-AWS).
 
-
 ## Features
 
-* Collaborative annotation
-* Multi-Language support
-* Emoji :smile: support
-* (future) Auto labeling
+-   Collaborative annotation
+-   Multi-Language support
+-   Emoji :smile: support
+-   (future) Auto labeling
 
 ## Requirements
 
-* Python 3.6+
-* Django 2.1.7+
-* Node.js 8.0+
-* Google Chrome(highly recommended)
+-   Python 3.6+
+-   Django 2.1.7+
+-   Node.js 8.0+
+-   Google Chrome (highly recommended)
 
 ## Installation
 
+### Clone repository
+
 First of all, you have to clone the repository:
 
 ```bash
 git clone https://github.com/chakki-works/doccano.git
 cd doccano
 ```
 
+_Note for Windows developers: Be sure to configure git to correctly handle line endings or you may encounter `status code 127` errors while running the services in future steps. Running with the git config options below will ensure your git directory correctly handles line endings._
+
+```bash
+git clone https://github.com/chakki-works/doccano.git --config core.autocrlf=input
+```
+
+### Install doccano
+
 To install doccano, there are three options:
 
-**Option1: Pull the production Docker image**
+#### Option 1: Pull the production Docker image
 
 ```bash
 docker pull chakkiworks/doccano
 ```
 
-**Option2: Setup Python environment**
+#### Option 2: Setup Python environment
 
 First we need to install the dependencies. Run the following commands:
 
@@ -111,7 +120,7 @@ npm run build
 cd ..
 ```
 
-**Option3: Pull the development Docker-Compose images**
+#### Option 3: Pull the development Docker-Compose images
 
 ```bash
 docker-compose pull
@@ -123,7 +132,7 @@ docker-compose pull
 
 Let’s start the development server and explore it.
 
-Depending on your installation method, there are two options:
+Depending on your installation method, there are three options:
 
 #### Option 1: Running the Docker image as a Container
 
@@ -151,6 +160,12 @@ Next we need to create a user who can login to the admin site. Run the following
 python manage.py create_admin --noinput --username "admin" --email "[email protected]" --password "password"
 ```
 
+Create the admin, annotator, and annotation approver roles to assign to users. Run the following command:
+
+```bash
+python manage.py create_roles
+```
+
 Developers can also validate that the project works as expected by running the tests:
 
 ```bash
@@ -162,7 +177,9 @@ Finally, to start the server, run the following command:
 ```bash
 python manage.py runserver
 ```
+
 Optionally, you can change the bind ip and port using the command
+
 ```bash
 python manage.py runserver <ip>:<port>
 ```
@@ -175,50 +192,63 @@ We can use docker-compose to set up the webpack server, django server, database,
 docker-compose up
 ```
 
-Now, open a Web browser and go to <http://127.0.0.1:8000/login/>. You should see the login screen:
+_Note the superuser account credentials located in the `docker-compose.yaml` file:_
+```yml
+ADMIN_USERNAME: "admin"
+ADMIN_PASSWORD: "password"
+```
+
+### Confirm all doccano services are running
+Open a Web browser and go to <http://127.0.0.1:8000/login/>. You should see the login screen:
 
 <img src="./docs/login_form.png" alt="Login Form" width=400>
 
 ### Create a project
 
 Now, try logging in with the superuser account you created in the previous step. You should see the doccano project list page:
 
-<img src="./docs/projects.png" alt="projects" width=600>
+<img src="./docs/projects.png" alt="Projects page" width=600>
 
 There is no project created yet. To create your project, make sure you’re in the project list page and select `Create Project` button. You should see the following screen:
 
 <img src="./docs/create_project.png" alt="Project Creation" width=400>
 
-In this step, you can select three project types: text classificatioin, sequence labeling and sequence to sequence. You should select a type with your purpose.
+In this step, you can select three project types: text classification, sequence labeling and sequence to sequence. You should select a type with your purpose.
 
 ### Import Data
 
 After creating a project, you will see the "Import Data" page, or click `Import Data` button in the navigation bar. You should see the following screen:
 
 <img src="./docs/upload.png" alt="Upload project" width=600>
 
-You can upload two types of files:
-- `CSV file`: file must contain a header with a `text` column or be one-column csv file.
-- `JSON file`: each line contains a JSON object with a `text` key. JSON format supports line breaks rendering.
+You can upload the following types of files (depending on project type):
+
+-   `Text file`: file must contain one sentence/document per line separated by new lines.
+-   `CSV file`: file must contain a header with `"text"` as the first column or be one-column csv file. If using labels the second column must be the labels.
+-   `Excel file`: file must contain a header with `"text"` as the first column or be one-column excel file. If using labels the second column must be the labels. Supports multiple sheets as long as format is the same.
+-   `JSON file`: each line contains a JSON object with a `text` key. JSON format supports line breaks rendering.
 
 > Notice: Doccano won't render line breaks in annotation page for sequence labeling task due to the indent problem, but the exported JSON file still contains line breaks.
 
-`example.txt` (or `example.csv`)
-```python
+`example.txt/csv/xlsx`
+
+```txt
 EU rejects German call to boycott British lamb.
 President Obama is speaking at the White House.
 He lives in Newark, Ohio.
 ...
 ```
+
 `example.json`
+
 ```JSON
 {"text": "EU rejects German call to boycott British lamb."}
 {"text": "President Obama is speaking at the White House."}
 {"text": "He lives in Newark, Ohio."}
 ...
 ```
 
-Any other columns (for csv) or keys (for json) are preserved and will be exported in the `metadata` column or key as is.
+Any other columns (for csv/excel) or keys (for json) are preserved and will be exported in the `metadata` column or key as is.
 
 Once you select a TXT/JSON file on your computer, click `Upload dataset` button. After uploading the dataset file, we will see the `Dataset` page (or click `Dataset` button list in the left bar). This page displays all the documents we uploaded in one project.
 
@@ -228,18 +258,23 @@ Click `Labels` button in left bar to define your own labels. You should see the
 
 <img src="./docs/label_editor.png" alt="Edit label" width=600>
 
+### Assign Roles to Users
+
+Click `Users` button in left bar to assign project users to annotator, admin, or annotation approval roles.
+
+<img src="./docs/user_page.png" alt="Assign users to roles on project" width=600>
 
 ### Annotation
 
 Now, you are ready to annotate the texts. Just click the `Annotate Data` button in the navigation bar, you can start to annotate the documents you uploaded.
 
-<img src="./docs/annotation.png" alt="Edit label" width=600>
+<img src="./docs/annotation.png" alt="Annotate data" width=600>
 
 ### Export Data
 
 After the annotation step, you can download the annotated data. Click the `Edit data` button in navigation bar, and then click `Export Data`. You should see below screen:
 
-<img src="./docs/export_data.png" alt="Edit label" width=600>
+<img src="./docs/export_data.png" alt="Export data" width=600>
 
 You can export data as CSV file or JSON file by clicking the button. As for the export file format, you can check it here: [Export File Formats](https://github.com/chakki-works/doccano/wiki/Export-File-Formats). 
 
@@ -249,11 +284,14 @@ by adding `external_id` to the imported file. For example:
 
 Input file may look like this:
 `import.json`
+
 ```JSON
 {"text": "EU rejects German call to boycott British lamb.", "meta": {"external_id": 1}}
 ```
+
 and the exported file will look like this:
 `output.json`
+
 ```JSON
 {"doc_id": 2023, "text": "EU rejects German call to boycott British lamb.", "labels": ["news"], "username": "root", "meta": {"external_id": 1}}
 ```
@@ -270,7 +308,6 @@ As with any software, doccano is under continuous development. If you have reque
 
 Here are some tips might be helpful. [How to Contribute to Doccano Project](https://github.com/chakki-works/doccano/wiki/How-to-Contribute-to-Doccano-Project)
 
-
 ## Contact
 
 For help and feedback, please feel free to contact [the author](https://github.com/Hironsan).