Merge pull request #2524 from flairNLP/documentation-for-release-10

Flair release 0.10

Author: alanakbik

CONTRIBUTING.md

@@ -12,9 +12,36 @@ In case you just want to help out and don't know where to start,
 [issues with "help wanted" label](https://github.com/zalandoresearch/flair/labels/help%20wanted) are good for 
 first-time contributors. 
 
+
 ## Git Commit Guidelines
 
 If there is already a ticket, use this number at the start of your commit message. 
 Use meaningful commit messages that described what you did.
 
-**Example:** `GH-42: Added new type of embeddings: DocumentEmbedding.` 
+**Example:** `GH-42: Added new type of embeddings: DocumentEmbedding.` 
+
+
+## Running unit tests locally
+
+For contributors looking to get deeper into the API we suggest cloning the repository and checking out the unit
+tests for examples of how to call methods. Nearly all classes and methods are documented, so finding your way around
+the code should hopefully be easy.
+
+You need [Pipenv](https://pipenv.readthedocs.io/) for this:
+
+```bash
+pipenv install --dev && pipenv shell
+pytest tests/
+```
+
+To run integration tests execute:
+```bash
+pytest --runintegration tests/
+```
+The integration tests will train small models.
+Afterwards, the trained model will be loaded for prediction.
+
+To also run slow tests, such as loading and using the embeddings provided by flair, you should execute:
+```bash
+pytest --runslow tests/
+```

README.md

@@ -22,7 +22,7 @@ document embeddings, including our proposed **[Flair embeddings](https://www.acl
 * **A PyTorch NLP framework.** Our framework builds directly on [PyTorch](https://pytorch.org/), making it easy to
 train your own models and experiment with new approaches using Flair embeddings and classes.
 
-Now at [version 0.9](https://github.com/flairNLP/flair/releases)!
+Now at [version 0.10](https://github.com/flairNLP/flair/releases)!
 
 
 ## Join Us: Open Positions at HU-Berlin!
@@ -155,18 +155,6 @@ If you use the Flair framework for your experiments, please cite [this paper](ht
 }
 ```
 
-If you use the pooled version of the Flair embeddings (PooledFlairEmbeddings), please cite [this paper](https://www.aclweb.org/anthology/papers/N/N19/N19-1078/):
-
-```
-@inproceedings{akbik2019naacl,
-  title={Pooled Contextualized Embeddings for Named Entity Recognition},
-  author={Akbik, Alan and Bergmann, Tanja and Vollgraf, Roland},
-  booktitle = {{NAACL} 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
-  pages     = {724–728},
-  year      = {2019}
-}
-```
-
 If you use our new "FLERT" models or approach, please cite [this paper](https://arxiv.org/abs/2011.06993):
 
 ```
@@ -179,6 +167,17 @@ If you use our new "FLERT" models or approach, please cite [this paper](https://
     primaryClass={cs.CL}
 ```
 
+If you use our TARS approach for few-shot and zero-shot learning, please cite [this paper](https://kishaloyhalder.github.io/pdfs/tars_coling2020.pdf/):
+
+```
+@inproceedings{halder2020coling,
+  title={Task Aware Representation of Sentences for Generic Text Classification},
+  author={Halder, Kishaloy and Akbik, Alan and Krapac, Josip and Vollgraf, Roland},
+  booktitle = {{COLING} 2020, 28th International Conference on Computational Linguistics},
+  year      = {2020}
+}
+```
+
 ## Contact
 
 Please email your questions or comments to [Alan Akbik](http://alanakbik.github.io/).
@@ -189,30 +188,6 @@ Thanks for your interest in contributing! There are many ways to get involved;
 start with our [contributor guidelines](CONTRIBUTING.md) and then
 check these [open issues](https://github.com/flairNLP/flair/issues) for specific tasks.
 
-For contributors looking to get deeper into the API we suggest cloning the repository and checking out the unit
-tests for examples of how to call methods. Nearly all classes and methods are documented, so finding your way around
-the code should hopefully be easy.
-
-### Running unit tests locally
-
-You need [Pipenv](https://pipenv.readthedocs.io/) for this:
-
-```bash
-pipenv install --dev && pipenv shell
-pytest tests/
-```
-
-To run integration tests execute:
-```bash
-pytest --runintegration tests/
-```
-The integration tests will train small models.
-Afterwards, the trained model will be loaded for prediction.
-
-To also run slow tests, such as loading and using the embeddings provided by flair, you should execute:
-```bash
-pytest --runslow tests/
-```
 
 ## [License](/LICENSE)
 

flair/__init__.py

@@ -25,7 +25,7 @@
 
 import logging.config
 
-__version__ = "0.9"
+__version__ = "0.10"
 
 logging.config.dictConfig(
     {

resources/docs/TUTORIAL_2_TAGGING.md

@@ -281,7 +281,7 @@ As we can see, the frame detector makes a distinction in sentence 1 between two
 Similarly, in sentence 2 the frame detector finds a light verb construction in which 'have' is the light verb and
 'look' is a frame evoking word.
 
-### Tagging a List of Sentences
+## Tagging a List of Sentences
 
 Often, you may want to tag an entire text corpus. In this case, you need to split the corpus into sentences and pass a
 list of `Sentence` objects to the `.predict()` method.
@@ -361,6 +361,55 @@ are provided:
 | 'communicative-functions' | English | detecting function of sentence in research paper (BETA) | scholarly papers |  |
 | 'de-offensive-language' | German | detecting offensive language | [GermEval 2018 Task 1](https://projects.fzai.h-da.de/iggsa/projekt/) |  **75.71** (Macro F1) |
 
+
+## Experimental: Relation Extraction
+
+Relations hold between two entities. For instance, a text like "George was born in Washington" 
+names two entities and also expresses that there is a born_in relationship between
+both. 
+
+We added two experimental relation extraction models, 
+trained over a modified version of TACRED: `relations` and `relations-fast`. 
+Use these models together with an entity tagger, like so: 
+```python
+from flair.data import Sentence
+from flair.models import RelationExtractor, SequenceTagger
+
+# 1. make example sentence
+sentence = Sentence("George was born in Washington")
+
+# 2. load entity tagger and predict entities
+tagger = SequenceTagger.load('ner-fast')
+tagger.predict(sentence)
+
+# check which entities have been found in the sentence
+entities = sentence.get_labels('ner')
+for entity in entities:
+    print(entity)
+
+# 3. load relation extractor
+extractor: RelationExtractor = RelationExtractor.load('relations-fast')
+
+# predict relations
+extractor.predict(sentence)
+
+# check which relations have been found
+relations = sentence.get_labels('relation')
+for relation in relations:
+    print(relation)
+```
+
+This should print: 
+
+~~~
+PER [George (1)] (0.9971)
+LOC [Washington (5)] (0.9847)
+
+born_in [George (1) -> Washington (5)] (0.9998)
+~~~
+
+Indicating that a born_in relationship holds between "George" and "Washington"!
+
 ## Tagging new classes without training data
 
 In case you need to label classes that are not included you can also try

resources/docs/TUTORIAL_7_TRAINING_A_MODEL.md

@@ -150,8 +150,6 @@ from flair.datasets import CONLL_03
 from flair.embeddings import TransformerWordEmbeddings
 from flair.models import SequenceTagger
 from flair.trainers import ModelTrainer
-import torch
-from torch.optim.lr_scheduler import OneCycleLR
 
 # 1. get the corpus
 corpus = CONLL_03()
@@ -165,38 +163,32 @@ label_dict = corpus.make_label_dictionary(label_type=label_type)
 print(label_dict)
 
 # 4. initialize fine-tuneable transformer embeddings WITH document context
-embeddings = TransformerWordEmbeddings(
-    model='xlm-roberta-large',
-    layers="-1",
-    subtoken_pooling="first",
-    fine_tune=True,
-    use_context=True,
-)
+embeddings = TransformerWordEmbeddings(model='xlm-roberta-large',
+                                       layers="-1",
+                                       subtoken_pooling="first",
+                                       fine_tune=True,
+                                       use_context=True,
+                                       )
 
 # 5. initialize bare-bones sequence tagger (no CRF, no RNN, no reprojection)
-tagger = SequenceTagger(
-    hidden_size=256,
-    embeddings=embeddings,
-    tag_dictionary=label_dict,
-    tag_type='ner',
-    use_crf=False,
-    use_rnn=False,
-    reproject_embeddings=False,
-)
+tagger = SequenceTagger(hidden_size=256,
+                        embeddings=embeddings,
+                        tag_dictionary=label_dict,
+                        tag_type='ner',
+                        use_crf=False,
+                        use_rnn=False,
+                        reproject_embeddings=False,
+                        )
 
-# 6. initialize trainer with AdamW optimizer
-trainer = ModelTrainer(tagger, corpus, optimizer=torch.optim.AdamW)
-
-# 7. run training with XLM parameters (20 epochs, small LR, one-cycle learning rate scheduling)
-trainer.train('resources/taggers/sota-ner-flert',
-              learning_rate=5.0e-6,
-              mini_batch_size=4,
-              mini_batch_chunk_size=1,  # remove this parameter to speed up computation if you have a big GPU
-              max_epochs=20,  # 10 is also good
-              scheduler=OneCycleLR,
-              embeddings_storage_mode='none',
-              weight_decay=0.,
-              )
+# 6. initialize trainer
+trainer = ModelTrainer(tagger, corpus)
+
+# 7. run fine-tuning
+trainer.fine_tune('resources/taggers/sota-ner-flert',
+                  learning_rate=5.0e-6,
+                  mini_batch_size=4,
+                  mini_batch_chunk_size=1,  # remove this parameter to speed up computation if you have a big GPU
+                  )
 ```
 
 This will give you state-of-the-art numbers similar to the ones reported
@@ -214,9 +206,6 @@ code below:
 (If you don't have a big GPU to fine-tune transformers, try `DocumentPoolEmbeddings` or `DocumentRNNEmbeddings` instead; sometimes they work just as well!)
 
 ```python
-import torch
-from torch.optim.lr_scheduler import OneCycleLR
-
 from flair.data import Corpus
 from flair.datasets import TREC_6
 from flair.embeddings import TransformerDocumentEmbeddings
@@ -238,18 +227,15 @@ document_embeddings = TransformerDocumentEmbeddings('distilbert-base-uncased', f
 # 5. create the text classifier
 classifier = TextClassifier(document_embeddings, label_dictionary=label_dict, label_type=label_type)
 
-# 6. initialize trainer with AdamW optimizer
-trainer = ModelTrainer(classifier, corpus, optimizer=torch.optim.AdamW)
+# 6. initialize trainer
+trainer = ModelTrainer(classifier, corpus)
 
 # 7. run training with fine-tuning
-trainer.train('resources/taggers/question-classification-with-transformer',
-              learning_rate=5.0e-5,
-              mini_batch_size=4,
-              max_epochs=10,
-              scheduler=OneCycleLR,
-              embeddings_storage_mode='none',
-              weight_decay=0.,
-              )
+trainer.fine_tune('resources/taggers/question-classification-with-transformer',
+                  learning_rate=5.0e-5,
+                  mini_batch_size=4,
+                  max_epochs=10,
+                  )
 ```
 
 Once the model is trained you can load it to predict the class of new sentences. Just call the `predict` method of the
@@ -358,55 +344,46 @@ for `TextClassifier`.
 
 ```python
 from flair.data import Corpus
-from flair.datasets import WNUT_17
-from flair.embeddings import TokenEmbeddings, WordEmbeddings, StackedEmbeddings
-from typing import List
+from flair.datasets import UD_ENGLISH
+from flair.embeddings import WordEmbeddings
 from flair.models import SequenceTagger
 from flair.trainers import ModelTrainer
 
 # 1. get the corpus
-corpus: Corpus = WNUT_17().downsample(0.1)
+corpus: Corpus = UD_ENGLISH().downsample(0.1)
 
 # 2. what label do we want to predict?
-label_type = 'ner'
+label_type = 'upos'
 
 # 3. make the label dictionary from the corpus
 label_dict = corpus.make_label_dictionary(label_type=label_type)
 
-# 4. initialize embeddings
-embedding_types: List[TokenEmbeddings] = [
-    WordEmbeddings('glove')
-]
-
-embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)
-
-# 5. initialize sequence tagger
-tagger: SequenceTagger = SequenceTagger(hidden_size=256,
-                                        embeddings=embeddings,
+# 4. initialize sequence tagger
+tagger: SequenceTagger = SequenceTagger(hidden_size=128,
+                                        embeddings=WordEmbeddings('glove'),
                                         tag_dictionary=label_dict,
-                                        tag_type=label_type,
-                                        use_crf=True)
+                                        tag_type=label_type)
 
-# 6. initialize trainer
+# 5. initialize trainer
 trainer: ModelTrainer = ModelTrainer(tagger, corpus)
 
-# 7. start training
-trainer.train('resources/taggers/example-ner',
+# 6. train for 10 epochs with checkpoint=True
+path = 'resources/taggers/example-pos'
+trainer.train(path,
               learning_rate=0.1,
               mini_batch_size=32,
               max_epochs=10,
-              checkpoint=True)
+              checkpoint=True,
+              )
 
-# 8. stop training at any point
+# 7. continue training at later point. Load previously trained model checkpoint, then resume
+trained_model = SequenceTagger.load(path + '/checkpoint.pt')
 
-# 9. continue trainer at later point
-checkpoint = 'resources/taggers/example-ner/checkpoint.pt'
-trainer = ModelTrainer.load_checkpoint(checkpoint, corpus)
-trainer.train('resources/taggers/example-ner',
-              learning_rate=0.1,
-              mini_batch_size=32,
-              max_epochs=150,
-              checkpoint=True)
+# resume training best model, but this time until epoch 25
+trainer.resume(trained_model,
+               base_path=path + '-resume',
+               max_epochs=25,
+               )
 ```
 
 ## Scalability: Training with Large Datasets

setup.py

@@ -5,7 +5,7 @@
 
 setup(
     name="flair",
-    version="0.9",
+    version="0.10",
     description="A very simple framework for state-of-the-art NLP",
     long_description=open("README.md", encoding="utf-8").read(),
     long_description_content_type="text/markdown",