diff --git a/README.md b/README.md
index 8832637..be3be65 100644
--- a/README.md
+++ b/README.md
@@ -1,14 +1,11 @@
- 
+ 
- 
- 
- 
- 
+ 
---
@@ -16,30 +13,33 @@
**NLP-Models-Tensorflow**, Gathers machine learning and tensorflow deep learning models for NLP problems, **code simplify inside Jupyter Notebooks 100%**.
## Table of contents
- * [Text classification](https://github.com/huseinzol05/NLP-Models-Tensorflow#text-classification)
- * [Chatbot](https://github.com/huseinzol05/NLP-Models-Tensorflow#chatbot)
- * [Neural Machine Translation](https://github.com/huseinzol05/NLP-Models-Tensorflow#neural-machine-translation-english-to-vietnam)
- * [Embedded](https://github.com/huseinzol05/NLP-Models-Tensorflow#embedded)
- * [Entity-Tagging](https://github.com/huseinzol05/NLP-Models-Tensorflow#entity-tagging)
- * [POS-Tagging](https://github.com/huseinzol05/NLP-Models-Tensorflow#pos-tagging)
- * [Dependency-Parser](https://github.com/huseinzol05/NLP-Models-Tensorflow#dependency-parser)
- * [SQUAD Question-Answers](https://github.com/huseinzol05/NLP-Models-Tensorflow#squad-question-answers)
- * [Question-Answers](https://github.com/huseinzol05/NLP-Models-Tensorflow#question-answers)
- * [Abstractive Summarization](https://github.com/huseinzol05/NLP-Models-Tensorflow#abstractive-summarization)
- * [Extractive Summarization](https://github.com/huseinzol05/NLP-Models-Tensorflow#extractive-summarization)
- * [Stemming](https://github.com/huseinzol05/NLP-Models-Tensorflow#stemming)
- * [Generator](https://github.com/huseinzol05/NLP-Models-Tensorflow#generator)
- * [Topic Generator](https://github.com/huseinzol05/NLP-Models-Tensorflow#topic-generator)
- * [Language detection](https://github.com/huseinzol05/NLP-Models-Tensorflow#language-detection)
- * [OCR (optical character recognition)](https://github.com/huseinzol05/NLP-Models-Tensorflow#ocr-optical-character-recognition)
- * [Sentence-Pair classification](https://github.com/huseinzol05/NLP-Models-Tensorflow#sentence-pair)
- * [Speech to Text](https://github.com/huseinzol05/NLP-Models-Tensorflow#speech-to-text)
- * [Text to Speech](https://github.com/huseinzol05/NLP-Models-Tensorflow#text-to-speech)
- * [Old-to-Young Vocoder](https://github.com/huseinzol05/NLP-Models-Tensorflow#old-to-young-vocoder)
- * [Text Similarity](https://github.com/huseinzol05/NLP-Models-Tensorflow#text-similarity)
- * [Text Augmentation](https://github.com/huseinzol05/NLP-Models-Tensorflow#text-augmentation)
- * [Miscellaneous](https://github.com/huseinzol05/NLP-Models-Tensorflow#Miscellaneous)
- * [Attention](https://github.com/huseinzol05/NLP-Models-Tensorflow#attention)
+ * [Abstractive Summarization](#abstractive-summarization)
+ * [Chatbot](#chatbot)
+ * [Dependency Parser](#dependency-parser)
+ * [Entity Tagging](#entity-tagging)
+ * [Extractive Summarization](#extractive-summarization)
+ * [Generator](#generator)
+ * [Language Detection](#language-detection)
+ * [Neural Machine Translation](neural-machine-translation)
+ * [OCR](#ocr-optical-character-recognition)
+ * [POS Tagging](#pos-tagging)
+ * [Question-Answers](#question-answers)
+ * [Sentence pairs](#sentence-pair)
+ * [Speech-to-Text](#speech-to-text)
+ * [Spelling correction](#spelling-correction)
+ * [SQUAD Question-Answers](#squad-question-answers)
+ * [Stemming](#stemming)
+ * [Text Augmentation](#text-augmentation)
+ * [Text Classification](#text-classification)
+ * [Text Similarity](#text-similarity)
+ * [Text-to-Speech](#text-to-speech)
+ * [Topic Generator](#topic-generator)
+ * [Topic Modeling](#topic-modeling)
+ * [Unsupervised Extractive Summarization](#unsupervised-extractive-summarization)
+ * [Vectorizer](#vectorizer)
+ * [Old-to-Young Vocoder](#old-to-young-vocoder)
+ * [Visualization](#visualization)
+ * [Attention](#attention)
## Objective
@@ -49,125 +49,40 @@ I will attached github repositories for models that I not implemented from scrat
## Tensorflow version
-Tensorflow version 1.10 and above only, not included 2.X version.
+Tensorflow version 1.13 and above only, not included 2.X version. 1.13 < Tensorflow < 2.0
+
+```bash
+pip install -r requirements.txt
+```
## Contents
-### [Text classification](text-classification)
+### [Abstractive Summarization](abstractive-summarization)
-Trained on [English sentiment dataset](https://github.com/huseinzol05/NLP-Models-Tensorflow/tree/master/text-classification/data).
+Trained on [India news](abstractive-summarization/dataset).
-1. Basic cell RNN
-2. Bidirectional RNN
-3. LSTM cell RNN
-4. GRU cell RNN
-5. LSTM RNN + Conv2D
-6. K-max Conv1d
-7. LSTM RNN + Conv1D + Highway
-8. LSTM RNN with Attention
-9. Neural Turing Machine
-10. BERT
-11. Dynamic Memory Network
-12. XL-net
-
-Complete list (76 notebooks)
+Accuracy based on 10 epochs only, calculated using word positions.
-1. Basic cell RNN
-2. Basic cell RNN + Hinge
-3. Basic cell RNN + Huber
-4. Basic cell Bidirectional RNN
-5. Basic cell Bidirectional RNN + Hinge
-6. Basic cell Bidirectional RNN + Huber
-7. LSTM cell RNN
-8. LSTM cell RNN + Hinge
-9. LSTM cell RNN + Huber
-10. LSTM cell Bidirectional RNN
-11. LSTM cell Bidirectional RNN + Huber
-12. LSTM cell RNN + Dropout + L2
-13. GRU cell RNN
-14. GRU cell RNN + Hinge
-15. GRU cell RNN + Huber
-16. GRU cell Bidirectional RNN
-17. GRU cell Bidirectional RNN + Hinge
-18. GRU cell Bidirectional RNN + Huber
-19. LSTM RNN + Conv2D
-20. K-max Conv1d
-21. LSTM RNN + Conv1D + Highway
-22. LSTM RNN + Basic Attention
-23. LSTM Dilated RNN
-24. Layer-Norm LSTM cell RNN
-25. Only Attention Neural Network
-26. Multihead-Attention Neural Network
-27. Neural Turing Machine
-28. LSTM Seq2Seq
-29. LSTM Seq2Seq + Luong Attention
-30. LSTM Seq2Seq + Bahdanau Attention
-31. LSTM Seq2Seq + Beam Decoder
-32. LSTM Bidirectional Seq2Seq
-33. Pointer Net
-34. LSTM cell RNN + Bahdanau Attention
-35. LSTM cell RNN + Luong Attention
-36. LSTM cell RNN + Stack Bahdanau Luong Attention
-37. LSTM cell Bidirectional RNN + backward Bahdanau + forward Luong
-38. Bytenet
-39. Fast-slow LSTM
-40. Siamese Network
-41. LSTM Seq2Seq + tf.estimator
-42. Capsule layers + RNN LSTM
-43. Capsule layers + LSTM Seq2Seq
-44. Capsule layers + LSTM Bidirectional Seq2Seq
-45. Nested LSTM
-46. LSTM Seq2Seq + Highway
-47. Triplet loss + LSTM
-48. DNC (Differentiable Neural Computer)
-49. ConvLSTM
-50. Temporal Convd Net
-51. Batch-all Triplet-loss + LSTM
-52. Fast-text
-53. Gated Convolution Network
-54. Simple Recurrent Unit
-55. LSTM Hierarchical Attention Network
-56. Bidirectional Transformers
-57. Dynamic Memory Network
-58. Entity Network
-59. End-to-End Memory Network
-60. BOW-Chars Deep sparse Network
-61. Residual Network using Atrous CNN
-62. Residual Network using Atrous CNN + Bahdanau Attention
-63. Deep pyramid CNN
-64. Transformer-XL
-65. Transfer learning GPT-2 345M
-66. Quasi-RNN
-67. Tacotron
-68. Slice GRU
-69. Slice GRU + Bahdanau
-70. Wavenet
-71. Transfer learning BERT Base
-72. Transfer learning XL-net Large
-73. LSTM BiRNN global Max and average pooling
-74. Transfer learning BERT Base drop 6 layers
-75. Transfer learning BERT Large drop 12 layers
-76. Transfer learning XL-net Base
+Complete list (12 notebooks)
+
+1. LSTM Seq2Seq using topic modelling, test accuracy 13.22%
+2. LSTM Seq2Seq + Luong Attention using topic modelling, test accuracy 12.39%
+3. LSTM Seq2Seq + Beam Decoder using topic modelling, test accuracy 10.67%
+4. LSTM Bidirectional + Luong Attention + Beam Decoder using topic modelling, test accuracy 8.29%
+5. Pointer-Generator + Bahdanau, https://github.com/xueyouluo/my_seq2seq, test accuracy 15.51%
+6. Copynet, test accuracy 11.15%
+7. Pointer-Generator + Luong, https://github.com/xueyouluo/my_seq2seq, test accuracy 16.51%
+8. Dilated Seq2Seq, test accuracy 10.88%
+9. Dilated Seq2Seq + Self Attention, test accuracy 11.54%
+10. BERT + Dilated CNN Seq2seq, test accuracy 13.5%
+11. self-attention + Pointer-Generator, test accuracy 4.34%
+12. Dilated-CNN Seq2seq + Pointer-Generator, test accuracy 5.57%
Complete list (54 notebooks)
@@ -220,7 +135,7 @@ Trained on [Cornell Movie Dialog corpus](https://github.com/huseinzol05/NLP-Mode
47. Attention is all you need + Beam Search
48. Transformer-XL + LSTM
49. GPT-2 + LSTM
-50. Fairseq
+50. CNN Seq2seq
51. Conv-Encoder + LSTM
52. Tacotron + Greedy decoder
53. Tacotron + Beam decoder
@@ -228,103 +143,171 @@ Trained on [Cornell Movie Dialog corpus](https://github.com/huseinzol05/NLP-Mode
Complete list (55 notebooks)
+Accuracy based on arc, types and root accuracies after 15 epochs only.
-1. Basic cell Seq2Seq-manual
-2. LSTM Seq2Seq-manual
-3. GRU Seq2Seq-manual
-4. Basic cell Seq2Seq-API Greedy
-5. LSTM Seq2Seq-API Greedy
-6. GRU Seq2Seq-API Greedy
-7. Basic cell Bidirectional Seq2Seq-manual
-8. LSTM Bidirectional Seq2Seq-manual
-9. GRU Bidirectional Seq2Seq-manual
-10. Basic cell Bidirectional Seq2Seq-API Greedy
-11. LSTM Bidirectional Seq2Seq-API Greedy
-12. GRU Bidirectional Seq2Seq-API Greedy
-13. Basic cell Seq2Seq-manual + Luong Attention
-14. LSTM Seq2Seq-manual + Luong Attention
-15. GRU Seq2Seq-manual + Luong Attention
-16. Basic cell Seq2Seq-manual + Bahdanau Attention
-17. LSTM Seq2Seq-manual + Bahdanau Attention
-18. GRU Seq2Seq-manual + Bahdanau Attention
-19. LSTM Bidirectional Seq2Seq-manual + Luong Attention
-20. GRU Bidirectional Seq2Seq-manual + Luong Attention
-21. LSTM Bidirectional Seq2Seq-manual + Bahdanau Attention
-22. GRU Bidirectional Seq2Seq-manual + Bahdanau Attention
-23. LSTM Bidirectional Seq2Seq-manual + backward Bahdanau + forward Luong
-24. GRU Bidirectional Seq2Seq-manual + backward Bahdanau + forward Luong
-25. LSTM Seq2Seq-API Greedy + Luong Attention
-26. GRU Seq2Seq-API Greedy + Luong Attention
-27. LSTM Seq2Seq-API Greedy + Bahdanau Attention
-28. GRU Seq2Seq-API Greedy + Bahdanau Attention
-29. LSTM Seq2Seq-API Beam Decoder
-30. GRU Seq2Seq-API Beam Decoder
-31. LSTM Bidirectional Seq2Seq-API + Luong Attention + Beam Decoder
-32. GRU Bidirectional Seq2Seq-API + Luong Attention + Beam Decoder
-33. LSTM Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder
-34. GRU Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder
-35. Bytenet
-36. LSTM Seq2Seq + tf.estimator
-37. Capsule layers + LSTM Seq2Seq-API Greedy
-38. Capsule layers + LSTM Seq2Seq-API + Luong Attention + Beam Decoder
-39. LSTM Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder + Dropout + L2
-40. DNC Seq2Seq
-41. LSTM Bidirectional Seq2Seq-API + Luong Monotic Attention + Beam Decoder
-42. LSTM Bidirectional Seq2Seq-API + Bahdanau Monotic Attention + Beam Decoder
-43. End-to-End Memory Network + Basic cell
-44. End-to-End Memory Network + LSTM cell
-45. Attention is all you need
-46. Transformer-XL
-47. Attention is all you need + Beam Search
-48. Fairseq
-49. Conv-Encoder + LSTM
-50. Bytenet Greedy
-51. Residual GRU Bidirectional Seq2Seq-API Greedy
-52. Google NMT
-53. Dilated Seq2Seq
-54. BERT Encoder + LSTM Luong Decoder
-55. BERT Encoder + Dilated Fairseq
+Complete list (8 notebooks)
+
+1. Bidirectional RNN + CRF + Biaffine, arc accuracy 70.48%, types accuracy 65.18%, root accuracy 66.4%
+2. Bidirectional RNN + Bahdanau + CRF + Biaffine, arc accuracy 70.82%, types accuracy 65.33%, root accuracy 66.77%
+3. Bidirectional RNN + Luong + CRF + Biaffine, arc accuracy 71.22%, types accuracy 65.73%, root accuracy 67.23%
+4. BERT Base + CRF + Biaffine, arc accuracy 64.30%, types accuracy 62.89%, root accuracy 74.19%
+5. Bidirectional RNN + Biaffine Attention + Cross Entropy, arc accuracy 72.42%, types accuracy 63.53%, root accuracy 68.51%
+6. BERT Base + Biaffine Attention + Cross Entropy, arc accuracy 72.85%, types accuracy 67.11%, root accuracy 73.93%
+7. Bidirectional RNN + Stackpointer, arc accuracy 61.88%, types accuracy 48.20%, root accuracy 89.39%
+8. XLNET Base + Biaffine Attention + Cross Entropy, arc accuracy 74.41%, types accuracy 71.37%, root accuracy 73.17%
Complete list (9 notebooks)
+
+1. Bidirectional RNN + CRF, test accuracy 96%
+2. Bidirectional RNN + Luong Attention + CRF, test accuracy 93%
+3. Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 95%
+4. Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 96%
+5. Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 96%
+6. Char Ngrams + Residual Network + Bahdanau Attention + CRF, test accuracy 69%
+7. Char Ngrams + Attention is you all Need + CRF, test accuracy 90%
+8. BERT, test accuracy 99%
+9. XLNET-Base, test accuracy 99%
+
+Complete list (4 notebooks)
+
+1. LSTM RNN, test accuracy 16.13%
+2. Dilated-CNN, test accuracy 15.54%
+3. Multihead Attention, test accuracy 26.33%
+4. BERT-Base
+
+Complete list (15 notebooks)
+
+1. Character-wise RNN + LSTM
+2. Character-wise RNN + Beam search
+3. Character-wise RNN + LSTM + Embedding
+4. Word-wise RNN + LSTM
+5. Word-wise RNN + LSTM + Embedding
+6. Character-wise + Seq2Seq + GRU
+7. Word-wise + Seq2Seq + GRU
+8. Character-wise RNN + LSTM + Bahdanau Attention
+9. Character-wise RNN + LSTM + Luong Attention
+10. Word-wise + Seq2Seq + GRU + Beam
+11. Character-wise + Seq2Seq + GRU + Bahdanau Attention
+12. Word-wise + Seq2Seq + GRU + Bahdanau Attention
+13. Character-wise Dilated CNN + Beam search
+14. Transformer + Beam search
+15. Transformer XL + Beam search
+
+Complete list (1 notebooks)
+
+1. Fast-text Char N-Grams
+
+Complete list (53 notebooks)
+
+1.basic-seq2seq
+2.lstm-seq2seq
+3.gru-seq2seq
+4.basic-seq2seq-contrib-greedy
+5.lstm-seq2seq-contrib-greedy
+6.gru-seq2seq-contrib-greedy
+7.basic-birnn-seq2seq
+8.lstm-birnn-seq2seq
+9.gru-birnn-seq2seq
+10.basic-birnn-seq2seq-contrib-greedy
+11.lstm-birnn-seq2seq-contrib-greedy
+12.gru-birnn-seq2seq-contrib-greedy
+13.basic-seq2seq-luong
+14.lstm-seq2seq-luong
+15.gru-seq2seq-luong
+16.basic-seq2seq-bahdanau
+17.lstm-seq2seq-bahdanau
+18.gru-seq2seq-bahdanau
+19.basic-birnn-seq2seq-bahdanau
+20.lstm-birnn-seq2seq-bahdanau
+21.gru-birnn-seq2seq-bahdanau
+22.basic-birnn-seq2seq-luong
+23.lstm-birnn-seq2seq-luong
+24.gru-birnn-seq2seq-luong
+25.lstm-seq2seq-contrib-greedy-luong
+26.gru-seq2seq-contrib-greedy-luong
+27.lstm-seq2seq-contrib-greedy-bahdanau
+28.gru-seq2seq-contrib-greedy-bahdanau
+29.lstm-seq2seq-contrib-beam-luong
+30.gru-seq2seq-contrib-beam-luong
+31.lstm-seq2seq-contrib-beam-bahdanau
+32.gru-seq2seq-contrib-beam-bahdanau
+33.lstm-birnn-seq2seq-contrib-beam-bahdanau
+34.lstm-birnn-seq2seq-contrib-beam-luong
+35.gru-birnn-seq2seq-contrib-beam-bahdanau
+36.gru-birnn-seq2seq-contrib-beam-luong
+37.lstm-birnn-seq2seq-contrib-beam-luongmonotonic
+38.gru-birnn-seq2seq-contrib-beam-luongmonotic
+39.lstm-birnn-seq2seq-contrib-beam-bahdanaumonotonic
+40.gru-birnn-seq2seq-contrib-beam-bahdanaumonotic
+41.residual-lstm-seq2seq-greedy-luong
+42.residual-gru-seq2seq-greedy-luong
+43.residual-lstm-seq2seq-greedy-bahdanau
+44.residual-gru-seq2seq-greedy-bahdanau
+45.memory-network-lstm-decoder-greedy
+46.google-nmt
+47.transformer-encoder-transformer-decoder
+48.transformer-encoder-lstm-decoder-greedy
+49.bertmultilanguage-encoder-bertmultilanguage-decoder
+50.bertmultilanguage-encoder-lstm-decoder
+51.bertmultilanguage-encoder-transformer-decoder
+52.bertenglish-encoder-transformer-decoder
+53.transformer-t2t-2gpu
+
+Complete list (2 notebooks)
+
+1. CNN + LSTM RNN, test accuracy 100%
+2. Im2Latex, test accuracy 100%
+
+Complete list (8 notebooks)
+
1. Bidirectional RNN + CRF, test accuracy 92%
2. Bidirectional RNN + Luong Attention + CRF, test accuracy 91%
3. Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 91%
@@ -334,50 +317,80 @@ Trained on [CONLL POS](https://cogcomp.org/page/resource_view/81).
7. Char Ngrams + Attention is you all Need + CRF, test accuracy 89%
8. BERT, test accuracy 99%
-### [Entity-Tagging](entity-tagging)
+Complete list (4 notebooks)
+
+1. End-to-End Memory Network + Basic cell
+2. End-to-End Memory Network + GRU cell
+3. End-to-End Memory Network + LSTM cell
+4. Dynamic Memory
+
+Complete list (1 notebooks)
-1. Bidirectional RNN + Bahdanau Attention + CRF
-2. Bidirectional RNN + Luong Attention + CRF
-3. Residual Network + Bahdanau Attention + CRF
-4. Residual Network + Bahdanau Attention + Char Embedded + CRF
-5. Attention is all you need + CRF
+1. BERT
+
+Complete list (11 notebooks)
+
+1. Tacotron, https://github.com/Kyubyong/tacotron_asr, test accuracy 77.09%
+2. BiRNN LSTM, test accuracy 84.66%
+3. BiRNN Seq2Seq + Luong Attention + Cross Entropy, test accuracy 87.86%
+4. BiRNN Seq2Seq + Bahdanau Attention + Cross Entropy, test accuracy 89.28%
+5. BiRNN Seq2Seq + Bahdanau Attention + CTC, test accuracy 86.35%
+6. BiRNN Seq2Seq + Luong Attention + CTC, test accuracy 80.30%
+7. CNN RNN + Bahdanau Attention, test accuracy 80.23%
+8. Dilated CNN RNN, test accuracy 31.60%
+9. Wavenet, test accuracy 75.11%
+10. Deep Speech 2, test accuracy 81.40%
+11. Wav2Vec Transfer learning BiRNN LSTM, test accuracy 83.24%
+
+Complete list (4 notebooks)
+
+1. BERT-Base
+2. XLNET-Base
+3. BERT-Base Fast
+4. BERT-Base accurate
+
+Complete list (1 notebooks)
+
1. BERT,
```json
{"exact_match": 77.57805108798486, "f1": 86.18327335287402}
```
-### [Question-Answers](question-answer)
-
-Trained on [bAbI Dataset](https://research.fb.com/downloads/babi/).
-
-1. End-to-End Memory Network + Basic cell
-2. End-to-End Memory Network + GRU cell
-3. End-to-End Memory Network + LSTM cell
-4. Dynamic Memory
+Complete list (6 notebooks)
1. LSTM + Seq2Seq + Beam
2. GRU + Seq2Seq + Beam
@@ -386,66 +399,138 @@ Trained on [English Lemmatization](https://github.com/huseinzol05/NLP-Models-Ten
5. DNC + Seq2Seq + Greedy
6. BiRNN + Bahdanau + Copynet
-### [Abstractive Summarization](abstractive-summarization)
+Complete list (8 notebooks)
-1. LSTM Seq2Seq using topic modelling, test accuracy 13.22%
-2. LSTM Seq2Seq + Luong Attention using topic modelling, test accuracy 12.39%
-3. LSTM Seq2Seq + Beam Decoder using topic modelling, test accuracy 10.67%
-4. LSTM Bidirectional + Luong Attention + Beam Decoder using topic modelling, test accuracy 8.29%
-5. Pointer-Generator + Bahdanau, https://github.com/xueyouluo/my_seq2seq, test accuracy 15.51%
-6. Copynet, test accuracy 11.15%
-7. Pointer-Generator + Luong, https://github.com/xueyouluo/my_seq2seq, test accuracy 16.51%
-8. Dilated Seq2Seq, test accuracy 10.88%
-9. Dilated Seq2Seq + Self Attention, test accuracy 11.54%
-10. BERT + Dilated Fairseq, test accuracy 13.5%
-11. self-attention + Pointer-Generator, test accuracy 4.34%
-12. Dilated-Fairseq + Pointer-Generator, test accuracy 5.57%
+1. Pretrained Glove
+2. GRU VAE-seq2seq-beam TF-probability
+3. LSTM VAE-seq2seq-beam TF-probability
+4. GRU VAE-seq2seq-beam + Bahdanau Attention TF-probability
+5. VAE + Deterministic Bahdanau Attention, https://github.com/HareeshBahuleyan/tf-var-attention
+6. VAE + VAE Bahdanau Attention, https://github.com/HareeshBahuleyan/tf-var-attention
+7. BERT-Base + Nucleus Sampling
+8. XLNET-Base + Nucleus Sampling
-### [Extractive Summarization](extractive-summarization)
+Complete list (79 notebooks)
-1. CNN + LSTM RNN
+1. Basic cell RNN
+2. Basic cell RNN + Hinge
+3. Basic cell RNN + Huber
+4. Basic cell Bidirectional RNN
+5. Basic cell Bidirectional RNN + Hinge
+6. Basic cell Bidirectional RNN + Huber
+7. LSTM cell RNN
+8. LSTM cell RNN + Hinge
+9. LSTM cell RNN + Huber
+10. LSTM cell Bidirectional RNN
+11. LSTM cell Bidirectional RNN + Huber
+12. LSTM cell RNN + Dropout + L2
+13. GRU cell RNN
+14. GRU cell RNN + Hinge
+15. GRU cell RNN + Huber
+16. GRU cell Bidirectional RNN
+17. GRU cell Bidirectional RNN + Hinge
+18. GRU cell Bidirectional RNN + Huber
+19. LSTM RNN + Conv2D
+20. K-max Conv1d
+21. LSTM RNN + Conv1D + Highway
+22. LSTM RNN + Basic Attention
+23. LSTM Dilated RNN
+24. Layer-Norm LSTM cell RNN
+25. Only Attention Neural Network
+26. Multihead-Attention Neural Network
+27. Neural Turing Machine
+28. LSTM Seq2Seq
+29. LSTM Seq2Seq + Luong Attention
+30. LSTM Seq2Seq + Bahdanau Attention
+31. LSTM Seq2Seq + Beam Decoder
+32. LSTM Bidirectional Seq2Seq
+33. Pointer Net
+34. LSTM cell RNN + Bahdanau Attention
+35. LSTM cell RNN + Luong Attention
+36. LSTM cell RNN + Stack Bahdanau Luong Attention
+37. LSTM cell Bidirectional RNN + backward Bahdanau + forward Luong
+38. Bytenet
+39. Fast-slow LSTM
+40. Siamese Network
+41. LSTM Seq2Seq + tf.estimator
+42. Capsule layers + RNN LSTM
+43. Capsule layers + LSTM Seq2Seq
+44. Capsule layers + LSTM Bidirectional Seq2Seq
+45. Nested LSTM
+46. LSTM Seq2Seq + Highway
+47. Triplet loss + LSTM
+48. DNC (Differentiable Neural Computer)
+49. ConvLSTM
+50. Temporal Convd Net
+51. Batch-all Triplet-loss + LSTM
+52. Fast-text
+53. Gated Convolution Network
+54. Simple Recurrent Unit
+55. LSTM Hierarchical Attention Network
+56. Bidirectional Transformers
+57. Dynamic Memory Network
+58. Entity Network
+59. End-to-End Memory Network
+60. BOW-Chars Deep sparse Network
+61. Residual Network using Atrous CNN
+62. Residual Network using Atrous CNN + Bahdanau Attention
+63. Deep pyramid CNN
+64. Transformer-XL
+65. Transfer learning GPT-2 345M
+66. Quasi-RNN
+67. Tacotron
+68. Slice GRU
+69. Slice GRU + Bahdanau
+70. Wavenet
+71. Transfer learning BERT Base
+72. Transfer learning XL-net Large
+73. LSTM BiRNN global Max and average pooling
+74. Transfer learning BERT Base drop 6 layers
+75. Transfer learning BERT Large drop 12 layers
+76. Transfer learning XL-net Base
+77. Transfer learning ALBERT
+78. Transfer learning ELECTRA Base
+79. Transfer learning ELECTRA Large
-### [Sentence-pair](sentence-pair)
+Complete list (10 notebooks)
-Trained on [Toronto speech dataset](https://tspace.library.utoronto.ca/handle/1807/24487).
+1. BiRNN + Contrastive loss, test accuracy 73.032%
+2. BiRNN + Cross entropy, test accuracy 74.265%
+3. BiRNN + Circle loss, test accuracy 75.857%
+4. BiRNN + Proxy loss, test accuracy 48.37%
+5. BERT Base + Cross entropy, test accuracy 91.123%
+6. BERT Base + Circle loss, test accuracy 89.903%
+7. ELECTRA Base + Cross entropy, test accuracy 96.317%
+8. ELECTRA Base + Circle loss, test accuracy 95.603%
+9. XLNET Base + Cross entropy, test accuracy 93.998%
+10. XLNET Base + Circle loss, test accuracy 94.033%
-1. Tacotron, https://github.com/Kyubyong/tacotron_asr
-2. Bidirectional RNN + Greedy CTC
-3. Bidirectional RNN + Beam CTC
-4. Seq2Seq + Bahdanau Attention + Beam CTC
-5. Seq2Seq + Luong Attention + Beam CTC
-6. Bidirectional RNN + Attention + Beam CTC
-7. Wavenet
-8. CNN encoder + RNN decoder + Bahdanau Attention
-9. CNN encoder + RNN decoder + Luong Attention
-10. Dilation CNN + GRU Bidirectional
-11. Deep speech 2
-12. Pyramid Dilated CNN
+Complete list (8 notebooks)
+
1. Tacotron, https://github.com/Kyubyong/tacotron
-2. Fairseq + Dilated CNN vocoder
+2. CNN Seq2seq + Dilated CNN vocoder
3. Seq2Seq + Bahdanau Attention
4. Seq2Seq + Luong Attention
5. Dilated CNN + Monothonic Attention + Dilated CNN vocoder
@@ -453,69 +538,90 @@ Trained on [Toronto speech dataset](https://tspace.library.utoronto.ca/handle/18
7. Deep CNN + Monothonic Attention + Dilated CNN vocoder
8. Deep CNN + Self Attention + Dilated CNN vocoder
-### [Old-to-Young Vocoder](vocoder)
-
-Trained on [Toronto speech dataset](https://tspace.library.utoronto.ca/handle/1807/24487).
-
-1. Dilated CNN
-
-### [Generator](generator)
-
-Trained on [Shakespeare dataset](https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/generator/shakespeare.txt).
-
-1. Character-wise RNN + LSTM
-2. Character-wise RNN + Beam search
-3. Character-wise RNN + LSTM + Embedding
-4. Word-wise RNN + LSTM
-5. Word-wise RNN + LSTM + Embedding
-6. Character-wise + Seq2Seq + GRU
-7. Word-wise + Seq2Seq + GRU
-8. Character-wise RNN + LSTM + Bahdanau Attention
-9. Character-wise RNN + LSTM + Luong Attention
-10. Word-wise + Seq2Seq + GRU + Beam
-11. Character-wise + Seq2Seq + GRU + Bahdanau Attention
-12. Word-wise + Seq2Seq + GRU + Bahdanau Attention
-13. Character-wise Dilated CNN + Beam search
-14. Transformer + Beam search
-15. Transformer XL + Beam search
+Complete list (4 notebooks)
+
1. TAT-LSTM
2. TAV-LSTM
3. MTA-LSTM
-4. Dilated Fairseq
+4. Dilated CNN Seq2seq
-### [Language-detection](language-detection)
+Complete list (3 notebooks)
-Trained on [First Quora Dataset Release: Question Pairs](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs).
+1. LDA2Vec
+2. BERT Attention
+3. XLNET Attention
-1. BiRNN + Contrastive loss, test accuracy 76.50%
-2. Dilated CNN + Contrastive loss, test accuracy 72.98%
-3. Transformer + Contrastive loss, test accuracy 73.48%
-4. Dilated CNN + Cross entropy, test accuracy 72.27%
-5. Transformer + Cross entropy, test accuracy 71.1%
-6. Transfer learning BERT base + Cross entropy, test accuracy 90%
+Complete list (3 notebooks)
+
+1. Skip-thought Vector
+2. Residual Network using Atrous CNN
+3. Residual Network using Atrous CNN + Bahdanau Attention
+
+Complete list (11 notebooks)
+
+1. Word Vector using CBOW sample softmax
+2. Word Vector using CBOW noise contrastive estimation
+3. Word Vector using skipgram sample softmax
+4. Word Vector using skipgram noise contrastive estimation
+5. Supervised Embedded
+6. Triplet-loss + LSTM
+7. LSTM Auto-Encoder
+8. Batch-All Triplet-loss LSTM
+9. Fast-text
+10. ELMO (biLM)
+11. Triplet-loss + BERT
+
+Complete list (4 notebooks)
+
+1. Attention heatmap on Bahdanau Attention
+2. Attention heatmap on Luong Attention
+3. BERT attention, https://github.com/hsm207/bert_attn_viz
+4. XLNET attention
+
+Complete list (1 notebooks)
+
+1. Dilated CNN
+
+Complete list (8 notebooks)
+
1. Bahdanau
2. Luong
3. Hierarchical
@@ -525,12 +631,7 @@ Trained on [First Quora Dataset Release: Question Pairs](https://data.quora.com/
7. Bahdanau API
8. Luong API
-### [Miscellaneous](misc)
-
-1. Attention heatmap on Bahdanau Attention
-2. Attention heatmap on Luong Attention
-3. BERT attention, https://github.com/hsm207/bert_attn_viz
-4. XLNET attention
+\" : 1,\n",
+ " \"\" : 2,\n",
+ " \"\" : 3,\n",
+ " \"\" : 4,\n",
+ " \"\" : 5,\n",
+ " \"\" : 6,\n",
+ " \"\" : 7,\n",
+ " \"\" : 8,\n",
+ "}\n",
+ "\n",
+ "VOCAB_SIZE = 32000\n",
+ "UNK_ID = special_symbols[\"\"]\n",
+ "CLS_ID = special_symbols[\"\"]\n",
+ "SEP_ID = special_symbols[\"\"]\n",
+ "MASK_ID = special_symbols[\"\"]\n",
+ "EOD_ID = special_symbols[\"\"]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def process_corpus(corpus, until = None):\n",
+ " global word2idx, tag2idx, char2idx, word_idx, tag_idx, char_idx\n",
+ " sentences, words, depends, labels, pos, sequences = [], [], [], [], [], []\n",
+ " temp_sentence, temp_word, temp_depend, temp_label, temp_pos = [], [], [], [], []\n",
+ " segments, masks = [], []\n",
+ " first_time = True\n",
+ " for sentence in corpus:\n",
+ " try:\n",
+ " if len(sentence):\n",
+ " if sentence[0] == '#':\n",
+ " continue\n",
+ " if first_time:\n",
+ " print(sentence)\n",
+ " first_time = False\n",
+ " sentence = sentence.split('\\t')\n",
+ " if sentence[7] not in tag2idx:\n",
+ " tag2idx[sentence[7]] = tag_idx\n",
+ " tag_idx += 1\n",
+ " temp_word.append(sentence[1])\n",
+ " temp_depend.append(int(sentence[6]) + 1)\n",
+ " temp_label.append(tag2idx[sentence[7]])\n",
+ " temp_sentence.append(sentence[1])\n",
+ " temp_pos.append(sentence[3])\n",
+ " else:\n",
+ " if len(temp_sentence) < 2 or len(temp_word) != len(temp_label):\n",
+ " temp_word = []\n",
+ " temp_depend = []\n",
+ " temp_label = []\n",
+ " temp_sentence = []\n",
+ " temp_pos = []\n",
+ " continue\n",
+ " bert_tokens = []\n",
+ " labels_ = []\n",
+ " depends_ = []\n",
+ " seq_ = []\n",
+ " for no, orig_token in enumerate(temp_word):\n",
+ " t = tokenize_fn(orig_token)\n",
+ " labels_.append(temp_label[no])\n",
+ " depends_.append(temp_depend[no])\n",
+ " bert_tokens.extend(t)\n",
+ " labels_.extend([1] * (len(t) - 1))\n",
+ " depends_.extend([0] * (len(t) - 1))\n",
+ " seq_.append(no + 1)\n",
+ " bert_tokens.extend([4, 3])\n",
+ " labels_.extend([0, 0])\n",
+ " depends_.extend([0, 0])\n",
+ " segment = [0] * (len(bert_tokens) - 1) + [SEG_ID_CLS]\n",
+ " input_mask = [0] * len(segment)\n",
+ " words.append(bert_tokens)\n",
+ " depends.append(depends_)\n",
+ " labels.append(labels_)\n",
+ " sentences.append(temp_sentence)\n",
+ " pos.append(temp_pos)\n",
+ " sequences.append(seq_)\n",
+ " segments.append(segment)\n",
+ " masks.append(input_mask)\n",
+ " temp_word = []\n",
+ " temp_depend = []\n",
+ " temp_label = []\n",
+ " temp_sentence = []\n",
+ " temp_pos = []\n",
+ " except Exception as e:\n",
+ " print(e, sentence)\n",
+ " return sentences[:-1], words[:-1], depends[:-1], labels[:-1], pos[:-1], sequences[:-1], segments[:-1], masks[:-1]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1\tFrom\tfrom\tADP\tIN\t_\t3\tcase\t3:case\t_\n",
+ "invalid literal for int() with base 10: '_' ['10.1', 'has', 'have', 'VERB', 'VBZ', '_', '_', '_', '8:parataxis', 'CopyOf=-1']\n",
+ "invalid literal for int() with base 10: '_' ['21.1', 'has', 'have', 'VERB', 'VBZ', '_', '_', '_', '16:conj:and', 'CopyOf=-1']\n"
+ ]
+ }
+ ],
+ "source": [
+ "with open('en_ewt-ud-dev.conllu') as fopen:\n",
+ " dev = fopen.read().split('\\n')\n",
+ "\n",
+ "sentences_dev, words_dev, depends_dev, labels_dev, _, seq_dev, segments_dev, masks_dev = process_corpus(dev)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1\tWhat\twhat\tPRON\tWP\tPronType=Int\t0\troot\t0:root\t_\n",
+ "invalid literal for int() with base 10: '_' ['24.1', 'left', 'left', 'VERB', 'VBN', 'Tense=Past|VerbForm=Part', '_', '_', '6:parataxis', 'CopyOf=6']\n"
+ ]
+ }
+ ],
+ "source": [
+ "with open('en_ewt-ud-test.conllu') as fopen:\n",
+ " test = fopen.read().split('\\n')\n",
+ "\n",
+ "sentences_test, words_test, depends_test, labels_test, _, seq_test, segments_test, masks_test = process_corpus(test)\n",
+ "sentences_test.extend(sentences_dev)\n",
+ "words_test.extend(words_dev)\n",
+ "depends_test.extend(depends_dev)\n",
+ "labels_test.extend(labels_dev)\n",
+ "seq_test.extend(seq_dev)\n",
+ "segments_test.extend(segments_dev)\n",
+ "masks_test.extend(masks_dev)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1\tAl\tAl\tPROPN\tNNP\tNumber=Sing\t0\troot\t0:root\tSpaceAfter=No\n",
+ "invalid literal for int() with base 10: '_' ['8.1', 'reported', 'report', 'VERB', 'VBN', 'Tense=Past|VerbForm=Part|Voice=Pass', '_', '_', '5:conj:and', 'CopyOf=5']\n",
+ "invalid literal for int() with base 10: '_' ['22.1', 'used', 'use', 'VERB', 'VBN', 'Tense=Past|VerbForm=Part', '_', '_', '13:advcl:with|17:conj:and', 'CopyOf=17']\n",
+ "invalid literal for int() with base 10: '_' ['22.1', 'used', 'use', 'VERB', 'VBN', 'Tense=Past|VerbForm=Part', '_', '_', '13:advcl:with|17:conj:and', 'CopyOf=17']\n",
+ "invalid literal for int() with base 10: '_' ['11.1', 'called', 'call', 'VERB', 'VBN', 'Tense=Past|VerbForm=Part|Voice=Pass', '_', '_', '3:conj:and', 'CopyOf=3']\n",
+ "invalid literal for int() with base 10: '_' ['14.1', 'is', 'be', 'VERB', 'VBZ', '_', '_', '_', '1:conj:and', 'CopyOf=1']\n",
+ "invalid literal for int() with base 10: '_' ['20.1', 'reflect', 'reflect', 'VERB', 'VBP', 'Mood=Ind|Tense=Pres|VerbForm=Fin', '_', '_', '7:acl:relcl|9:conj', 'CopyOf=9']\n",
+ "invalid literal for int() with base 10: '_' ['21.1', 'recruited', 'recruit', 'VERB', 'VBD', 'Mood=Ind|Tense=Past|VerbForm=Fin', '_', '_', '9:conj:and', 'CopyOf=9']\n",
+ "invalid literal for int() with base 10: '_' ['9.1', 'wish', 'wish', 'VERB', 'VBP', 'Mood=Ind|Tense=Pres|VerbForm=Fin', '_', '_', '2:conj:and', 'CopyOf=2']\n",
+ "invalid literal for int() with base 10: '_' ['38.1', 'supplied', 'supply', 'VERB', 'VBN', 'Tense=Past|VerbForm=Part|Voice=Pass', '_', '_', '16:conj:and', 'CopyOf=16']\n",
+ "invalid literal for int() with base 10: '_' ['18.1', 'keep', 'keep', 'VERB', 'VB', 'Mood=Imp|VerbForm=Fin', '_', '_', '14:conj:and', 'CopyOf=14']\n",
+ "invalid literal for int() with base 10: '_' ['21.1', 'keep', 'keep', 'VERB', 'VB', 'Mood=Imp|VerbForm=Fin', '_', '_', '14:conj:and', 'CopyOf=14']\n",
+ "invalid literal for int() with base 10: '_' ['18.1', 'mean', 'mean', 'VERB', 'VB', 'VerbForm=Inf', '_', '_', '8:conj', 'CopyOf=8']\n",
+ "invalid literal for int() with base 10: '_' ['30.1', 'play', 'play', 'VERB', 'VBP', 'Mood=Ind|Tense=Pres|VerbForm=Fin', '_', '_', '18:acl:relcl|27:conj:but', 'CopyOf=27']\n",
+ "invalid literal for int() with base 10: '_' ['22.1', 'have', 'have', 'VERB', 'VBP', 'Mood=Ind|Tense=Pres|VerbForm=Fin', '_', '_', '17:conj', 'CopyOf=17']\n",
+ "invalid literal for int() with base 10: '_' ['27.1', 'have', 'have', 'VERB', 'VBP', 'Mood=Ind|Tense=Pres|VerbForm=Fin', '_', '_', '17:conj', 'CopyOf=17']\n",
+ "invalid literal for int() with base 10: '_' ['49.1', 'helped', 'help', 'VERB', 'VBD', '_', '_', '_', '38:conj:but', 'CopyOf=38']\n",
+ "invalid literal for int() with base 10: '_' ['7.1', 'found', 'find', 'VERB', 'VBD', 'Mood=Ind|Tense=Past|VerbForm=Fin', '_', '_', '3:conj', 'CopyOf=3']\n",
+ "invalid literal for int() with base 10: '_' ['10.1', 'excited', 'excited', 'ADJ', 'JJ', 'Degree=Pos', '_', '_', '4:advcl', 'CopyOf=4']\n",
+ "invalid literal for int() with base 10: '_' ['15.1', \"'s\", 'be', 'VERB', 'VBZ', '_', '_', '_', '2:conj:and', 'CopyOf=2']\n",
+ "invalid literal for int() with base 10: '_' ['25.1', 'took', 'take', 'VERB', 'VBD', 'Mood=Ind|Tense=Past|VerbForm=Fin', '_', '_', '17:conj:and', 'CopyOf=17']\n",
+ "invalid literal for int() with base 10: '_' ['10.1', 'loss', 'lose', 'VERB', 'VBD', 'Mood=Ind|Tense=Past|VerbForm=Fin', '_', '_', '3:conj:and', 'CopyOf=3']\n",
+ "invalid literal for int() with base 10: '_' ['11.1', 'leave', 'leave', 'VERB', 'VB', 'VerbForm=Inf', '_', '_', '7:parataxis', 'CopyOf=7']\n",
+ "invalid literal for int() with base 10: '_' ['24.1', 'charge', 'charge', 'VERB', 'VBP', 'Mood=Ind|Tense=Pres|VerbForm=Fin', '_', '_', '16:conj:and', 'CopyOf=16']\n"
+ ]
+ }
+ ],
+ "source": [
+ "with open('en_ewt-ud-train.conllu') as fopen:\n",
+ " train = fopen.read().split('\\n')\n",
+ "\n",
+ "sentences_train, words_train, depends_train, labels_train, _, _, segments_train, masks_train = process_corpus(train)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(12000, 3824)"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "len(sentences_train), len(sentences_test)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "idx2tag = {v:k for k, v in tag2idx.items()}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train_X = words_train\n",
+ "train_Y = labels_train\n",
+ "train_depends = depends_train\n",
+ "\n",
+ "test_X = words_test\n",
+ "test_Y = labels_test\n",
+ "test_depends = depends_test"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/home/husein/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
+ "/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.6) or chardet (3.0.4) doesn't match a supported version!\n",
+ " RequestsDependencyWarning)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING:tensorflow:From /home/husein/testing/model_utils.py:295: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
+ "\n",
+ "WARNING:tensorflow:From /home/husein/testing/xlnet.py:70: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/home/husein/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
+ "/home/husein/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
+ " np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
+ ]
+ }
+ ],
+ "source": [
+ "import xlnet\n",
+ "import model_utils\n",
+ "import tensorflow as tf\n",
+ "import numpy as np\n",
+ "\n",
+ "kwargs = dict(\n",
+ " is_training=True,\n",
+ " use_tpu=False,\n",
+ " use_bfloat16=False,\n",
+ " dropout=0.1,\n",
+ " dropatt=0.1,\n",
+ " init='normal',\n",
+ " init_range=0.1,\n",
+ " init_std=0.05,\n",
+ " clamp_len=-1)\n",
+ "\n",
+ "xlnet_parameters = xlnet.RunConfig(**kwargs)\n",
+ "xlnet_config = xlnet.XLNetConfig(json_path='xlnet_cased_L-12_H-768_A-12/xlnet_config.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "5625 562\n"
+ ]
+ }
+ ],
+ "source": [
+ "epoch = 15\n",
+ "batch_size = 32\n",
+ "warmup_proportion = 0.1\n",
+ "num_train_steps = int(len(train_X) / batch_size * epoch)\n",
+ "num_warmup_steps = int(num_train_steps * warmup_proportion)\n",
+ "print(num_train_steps, num_warmup_steps)\n",
+ "\n",
+ "training_parameters = dict(\n",
+ " decay_method = 'poly',\n",
+ " train_steps = num_train_steps,\n",
+ " learning_rate = 2e-5,\n",
+ " warmup_steps = num_warmup_steps,\n",
+ " min_lr_ratio = 0.0,\n",
+ " weight_decay = 0.00,\n",
+ " adam_epsilon = 1e-8,\n",
+ " num_core_per_host = 1,\n",
+ " lr_layer_decay_rate = 1,\n",
+ " use_tpu=False,\n",
+ " use_bfloat16=False,\n",
+ " dropout=0.0,\n",
+ " dropatt=0.0,\n",
+ " init='normal',\n",
+ " init_range=0.1,\n",
+ " init_std=0.02,\n",
+ " clip = 1.0,\n",
+ " clamp_len=-1,)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class Parameter:\n",
+ " def __init__(self, decay_method, warmup_steps, weight_decay, adam_epsilon, \n",
+ " num_core_per_host, lr_layer_decay_rate, use_tpu, learning_rate, train_steps,\n",
+ " min_lr_ratio, clip, **kwargs):\n",
+ " self.decay_method = decay_method\n",
+ " self.warmup_steps = warmup_steps\n",
+ " self.weight_decay = weight_decay\n",
+ " self.adam_epsilon = adam_epsilon\n",
+ " self.num_core_per_host = num_core_per_host\n",
+ " self.lr_layer_decay_rate = lr_layer_decay_rate\n",
+ " self.use_tpu = use_tpu\n",
+ " self.learning_rate = learning_rate\n",
+ " self.train_steps = train_steps\n",
+ " self.min_lr_ratio = min_lr_ratio\n",
+ " self.clip = clip\n",
+ " \n",
+ "training_parameters = Parameter(**training_parameters)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class BiAAttention:\n",
+ " def __init__(self, input_size_encoder, input_size_decoder, num_labels):\n",
+ " self.input_size_encoder = input_size_encoder\n",
+ " self.input_size_decoder = input_size_decoder\n",
+ " self.num_labels = num_labels\n",
+ " \n",
+ " self.W_d = tf.get_variable(\"W_d\", shape=[self.num_labels, self.input_size_decoder],\n",
+ " initializer=tf.contrib.layers.xavier_initializer())\n",
+ " self.W_e = tf.get_variable(\"W_e\", shape=[self.num_labels, self.input_size_encoder],\n",
+ " initializer=tf.contrib.layers.xavier_initializer())\n",
+ " self.U = tf.get_variable(\"U\", shape=[self.num_labels, self.input_size_decoder, self.input_size_encoder],\n",
+ " initializer=tf.contrib.layers.xavier_initializer())\n",
+ " \n",
+ " def forward(self, input_d, input_e, mask_d=None, mask_e=None):\n",
+ " batch = tf.shape(input_d)[0]\n",
+ " length_decoder = tf.shape(input_d)[1]\n",
+ " length_encoder = tf.shape(input_e)[1]\n",
+ " out_d = tf.expand_dims(tf.matmul(self.W_d, tf.transpose(input_d, [0, 2, 1])), 3)\n",
+ " out_e = tf.expand_dims(tf.matmul(self.W_e, tf.transpose(input_e, [0, 2, 1])), 2)\n",
+ " output = tf.matmul(tf.expand_dims(input_d, 1), self.U)\n",
+ " output = tf.matmul(output, tf.transpose(tf.expand_dims(input_e, 1), [0, 1, 3, 2]))\n",
+ " \n",
+ " output = output + out_d + out_e\n",
+ " \n",
+ " if mask_d is not None:\n",
+ " d = tf.expand_dims(tf.expand_dims(mask_d, 1), 3)\n",
+ " e = tf.expand_dims(tf.expand_dims(mask_e, 1), 2)\n",
+ " output = output * d * e\n",
+ " \n",
+ " return output\n",
+ " \n",
+ "class BiLinear:\n",
+ " def __init__(self, left_features, right_features, out_features):\n",
+ " self.left_features = left_features\n",
+ " self.right_features = right_features\n",
+ " self.out_features = out_features\n",
+ " \n",
+ " self.U = tf.get_variable(\"U-bi\", shape=[out_features, left_features, right_features],\n",
+ " initializer=tf.contrib.layers.xavier_initializer())\n",
+ " self.W_l = tf.get_variable(\"Wl\", shape=[out_features, left_features],\n",
+ " initializer=tf.contrib.layers.xavier_initializer())\n",
+ " self.W_r = tf.get_variable(\"Wr\", shape=[out_features, right_features],\n",
+ " initializer=tf.contrib.layers.xavier_initializer())\n",
+ " \n",
+ " def forward(self, input_left, input_right):\n",
+ " left_size = tf.shape(input_left)\n",
+ " output_shape = tf.concat([left_size[:-1], [self.out_features]], axis = 0)\n",
+ " batch = tf.cast(tf.reduce_prod(left_size[:-1]), tf.int32)\n",
+ " input_left = tf.reshape(input_left, (batch, self.left_features))\n",
+ " input_right = tf.reshape(input_right, (batch, self.right_features))\n",
+ " tiled = tf.tile(tf.expand_dims(input_left, axis = 0), (self.out_features,1,1))\n",
+ " output = tf.transpose(tf.reduce_sum(tf.matmul(tiled, self.U), axis = 2))\n",
+ " output = output + tf.matmul(input_left, tf.transpose(self.W_l))\\\n",
+ " + tf.matmul(input_right, tf.transpose(self.W_r))\n",
+ " \n",
+ " return tf.reshape(output, output_shape)\n",
+ "\n",
+ "class Attention:\n",
+ " def __init__(self, word_dim, num_words, char_dim, num_chars, num_filters, kernel_size,\n",
+ " hidden_size, encoder_layers, num_labels, arc_space, type_space):\n",
+ " \n",
+ " def cells(size, reuse=False):\n",
+ " return tf.nn.rnn_cell.LSTMCell(size,\n",
+ " initializer=tf.orthogonal_initializer(),reuse=reuse)\n",
+ " \n",
+ " self.word_embedd = tf.Variable(tf.random_uniform([num_words, word_dim], -1, 1))\n",
+ " self.char_embedd = tf.Variable(tf.random_uniform([num_chars, char_dim], -1, 1))\n",
+ " self.conv1d = tf.layers.Conv1D(num_filters, kernel_size, 1, padding='VALID')\n",
+ " self.num_labels = num_labels\n",
+ " self.encoder = tf.nn.rnn_cell.MultiRNNCell([cells(hidden_size) for _ in range(encoder_layers)])\n",
+ "\n",
+ " \n",
+ " \n",
+ " def encode(self, input_word, input_char):\n",
+ " word = tf.nn.embedding_lookup(self.word_embedd, input_word)\n",
+ " char = tf.nn.embedding_lookup(self.char_embedd, input_char)\n",
+ " b = tf.shape(char)[0]\n",
+ " wl = tf.shape(char)[1]\n",
+ " cl = tf.shape(char)[2]\n",
+ " d = char.shape[3]\n",
+ " char = tf.reshape(char, [b * wl, cl, d])\n",
+ " char = tf.reduce_max(self.conv1d(char), axis = 1)\n",
+ " char = tf.nn.tanh(char)\n",
+ " d = char.shape[-1]\n",
+ " char = tf.reshape(char, [b, wl, d])\n",
+ " \n",
+ " src_encoding = tf.concat([word, char], axis=2)\n",
+ " output, hn = tf.nn.dynamic_rnn(self.encoder, src_encoding, dtype = tf.float32,\n",
+ " scope = 'encoder')\n",
+ " arc_h = tf.nn.elu(self.arc_h(output))\n",
+ " arc_c = tf.nn.elu(self.arc_c(output))\n",
+ " \n",
+ " type_h = tf.nn.elu(self.type_h(output))\n",
+ " type_c = tf.nn.elu(self.type_c(output))\n",
+ " \n",
+ " return (arc_h, arc_c), (type_h, type_c), hn\n",
+ " \n",
+ " def forward(self, input_word, input_char, mask):\n",
+ " arcs, types, _ = self.encode(input_word, input_char)\n",
+ " \n",
+ " out_arc = tf.squeeze(self.attention.forward(arcs[0], arcs[1], mask_d=mask, mask_e=mask), axis = 1)\n",
+ " return out_arc, types, mask\n",
+ " \n",
+ " def loss(self, input_word, input_char, mask, heads, types):\n",
+ " out_arc, out_type, _ = self.forward(input_word, input_char, mask)\n",
+ " type_h, type_c = out_type\n",
+ " batch = tf.shape(out_arc)[0]\n",
+ " max_len = tf.shape(out_arc)[1]\n",
+ " batch_index = tf.range(0, batch)\n",
+ " t = tf.transpose(heads)\n",
+ " broadcasted = tf.broadcast_to(batch_index, tf.shape(t))\n",
+ " concatenated = tf.transpose(tf.concat([tf.expand_dims(broadcasted, axis = 0), \n",
+ " tf.expand_dims(t, axis = 0)], axis = 0))\n",
+ " type_h = tf.gather_nd(type_h, concatenated)\n",
+ " out_type = self.bilinear.forward(type_h, type_c)\n",
+ " minus_inf = -1e8\n",
+ " minus_mask = (1 - mask) * minus_inf\n",
+ " out_arc = out_arc + tf.expand_dims(minus_mask, axis = 2) + tf.expand_dims(minus_mask, axis = 1)\n",
+ " loss_arc = tf.nn.log_softmax(out_arc, dim=1)\n",
+ " loss_type = tf.nn.log_softmax(out_type, dim=2)\n",
+ " loss_arc = loss_arc * tf.expand_dims(mask, axis = 2) * tf.expand_dims(mask, axis = 1)\n",
+ " loss_type = loss_type * tf.expand_dims(mask, axis = 2)\n",
+ " num = tf.reduce_sum(mask) - tf.cast(batch, tf.float32)\n",
+ " child_index = tf.tile(tf.expand_dims(tf.range(0, max_len), 1), [1, batch])\n",
+ " t = tf.transpose(heads)\n",
+ " broadcasted = tf.broadcast_to(batch_index, tf.shape(t))\n",
+ " concatenated = tf.transpose(tf.concat([tf.expand_dims(broadcasted, axis = 0),\n",
+ " tf.expand_dims(t, axis = 0),\n",
+ " tf.expand_dims(child_index, axis = 0)], axis = 0))\n",
+ " loss_arc = tf.gather_nd(loss_arc, concatenated)\n",
+ " loss_arc = tf.transpose(loss_arc, [1, 0])\n",
+ " \n",
+ " t = tf.transpose(types)\n",
+ " broadcasted = tf.broadcast_to(batch_index, tf.shape(t))\n",
+ " concatenated = tf.transpose(tf.concat([tf.expand_dims(broadcasted, axis = 0),\n",
+ " tf.expand_dims(child_index, axis = 0),\n",
+ " tf.expand_dims(t, axis = 0)], axis = 0))\n",
+ " loss_type = tf.gather_nd(loss_type, concatenated)\n",
+ " loss_type = tf.transpose(loss_type, [1, 0])\n",
+ " return tf.reduce_sum(-loss_arc) / num, tf.reduce_sum(-loss_type) / num\n",
+ " \n",
+ " def decode(self, input_word, input_char, mask, leading_symbolic=0):\n",
+ " out_arc, out_type, _ = self.forward(input_word, input_char, mask)\n",
+ " batch = tf.shape(out_arc)[0]\n",
+ " max_len = tf.shape(out_arc)[1]\n",
+ " sec_max_len = tf.shape(out_arc)[2]\n",
+ " out_arc = out_arc + tf.linalg.diag(tf.fill([max_len], -np.inf))\n",
+ " minus_mask = tf.expand_dims(tf.cast(1 - mask, tf.bool), axis = 2)\n",
+ " minus_mask = tf.tile(minus_mask, [1, 1, sec_max_len])\n",
+ " out_arc = tf.where(minus_mask, tf.fill(tf.shape(out_arc), -np.inf), out_arc)\n",
+ " heads = tf.argmax(out_arc, axis = 1)\n",
+ " type_h, type_c = out_type\n",
+ " batch = tf.shape(type_h)[0]\n",
+ " max_len = tf.shape(type_h)[1]\n",
+ " batch_index = tf.range(0, batch)\n",
+ " t = tf.cast(tf.transpose(heads), tf.int32)\n",
+ " broadcasted = tf.broadcast_to(batch_index, tf.shape(t))\n",
+ " concatenated = tf.transpose(tf.concat([tf.expand_dims(broadcasted, axis = 0), \n",
+ " tf.expand_dims(t, axis = 0)], axis = 0))\n",
+ " type_h = tf.gather_nd(type_h, concatenated)\n",
+ " out_type = self.bilinear.forward(type_h, type_c)\n",
+ " out_type = out_type[:, :, leading_symbolic:]\n",
+ " types = tf.argmax(out_type, axis = 2)\n",
+ " return heads, types\n",
+ " \n",
+ "class Model:\n",
+ " def __init__(\n",
+ " self,\n",
+ " learning_rate,\n",
+ " hidden_size_word,\n",
+ " cov = 0.0):\n",
+ " \n",
+ " self.words = tf.placeholder(tf.int32, (None, None))\n",
+ " self.segment_ids = tf.placeholder(tf.int32, [None, None])\n",
+ " self.input_masks = tf.placeholder(tf.float32, [None, None])\n",
+ " self.heads = tf.placeholder(tf.int32, (None, None))\n",
+ " self.types = tf.placeholder(tf.int32, (None, None))\n",
+ " self.mask = tf.cast(tf.math.not_equal(self.words, 0), tf.float32)\n",
+ " self.maxlen = tf.shape(self.words)[1]\n",
+ " self.lengths = tf.count_nonzero(self.words, 1)\n",
+ " mask = self.mask\n",
+ " heads = self.heads\n",
+ " types = self.types\n",
+ " \n",
+ " self.arc_h = tf.layers.Dense(hidden_size_word)\n",
+ " self.arc_c = tf.layers.Dense(hidden_size_word)\n",
+ " self.attention = BiAAttention(hidden_size_word, hidden_size_word, 1)\n",
+ "\n",
+ " self.type_h = tf.layers.Dense(hidden_size_word)\n",
+ " self.type_c = tf.layers.Dense(hidden_size_word)\n",
+ " self.bilinear = BiLinear(hidden_size_word, hidden_size_word, len(tag2idx))\n",
+ " \n",
+ " xlnet_model = xlnet.XLNetModel(\n",
+ " xlnet_config=xlnet_config,\n",
+ " run_config=xlnet_parameters,\n",
+ " input_ids=tf.transpose(self.words, [1, 0]),\n",
+ " seg_ids=tf.transpose(self.segment_ids, [1, 0]),\n",
+ " input_mask=tf.transpose(self.input_masks, [1, 0]))\n",
+ " output_layer = xlnet_model.get_sequence_output()\n",
+ " output_layer = tf.transpose(output_layer, [1, 0, 2])\n",
+ " \n",
+ " arc_h = tf.nn.elu(self.arc_h(output_layer))\n",
+ " arc_c = tf.nn.elu(self.arc_c(output_layer))\n",
+ " \n",
+ " type_h = tf.nn.elu(self.type_h(output_layer))\n",
+ " type_c = tf.nn.elu(self.type_c(output_layer))\n",
+ " \n",
+ " out_arc = tf.squeeze(self.attention.forward(arc_h, arc_h, mask_d=self.mask, \n",
+ " mask_e=self.mask), axis = 1)\n",
+ " \n",
+ " batch = tf.shape(out_arc)[0]\n",
+ " max_len = tf.shape(out_arc)[1]\n",
+ " sec_max_len = tf.shape(out_arc)[2]\n",
+ " batch_index = tf.range(0, batch)\n",
+ " \n",
+ " decode_arc = out_arc + tf.linalg.diag(tf.fill([max_len], -np.inf))\n",
+ " minus_mask = tf.expand_dims(tf.cast(1 - mask, tf.bool), axis = 2)\n",
+ " minus_mask = tf.tile(minus_mask, [1, 1, sec_max_len])\n",
+ " decode_arc = tf.where(minus_mask, tf.fill(tf.shape(decode_arc), -np.inf), decode_arc)\n",
+ " self.heads_seq = tf.argmax(decode_arc, axis = 1)\n",
+ " \n",
+ " t = tf.cast(tf.transpose(self.heads_seq), tf.int32)\n",
+ " broadcasted = tf.broadcast_to(batch_index, tf.shape(t))\n",
+ " concatenated = tf.transpose(tf.concat([tf.expand_dims(broadcasted, axis = 0), \n",
+ " tf.expand_dims(t, axis = 0)], axis = 0))\n",
+ " type_h = tf.gather_nd(type_h, concatenated)\n",
+ " out_type = self.bilinear.forward(type_h, type_c)\n",
+ " self.tags_seq = tf.argmax(out_type, axis = 2)\n",
+ " \n",
+ " batch = tf.shape(out_arc)[0]\n",
+ " max_len = tf.shape(out_arc)[1]\n",
+ " batch_index = tf.range(0, batch)\n",
+ " t = tf.transpose(heads)\n",
+ " broadcasted = tf.broadcast_to(batch_index, tf.shape(t))\n",
+ " concatenated = tf.transpose(tf.concat([tf.expand_dims(broadcasted, axis = 0), \n",
+ " tf.expand_dims(t, axis = 0)], axis = 0))\n",
+ " type_h = tf.gather_nd(type_h, concatenated)\n",
+ " out_type = self.bilinear.forward(type_h, type_c)\n",
+ " minus_inf = -1e8\n",
+ " minus_mask = (1 - mask) * minus_inf\n",
+ " out_arc = out_arc + tf.expand_dims(minus_mask, axis = 2) + tf.expand_dims(minus_mask, axis = 1)\n",
+ " loss_arc = tf.nn.log_softmax(out_arc, dim=1)\n",
+ " loss_type = tf.nn.log_softmax(out_type, dim=2)\n",
+ " loss_arc = loss_arc * tf.expand_dims(mask, axis = 2) * tf.expand_dims(mask, axis = 1)\n",
+ " loss_type = loss_type * tf.expand_dims(mask, axis = 2)\n",
+ " num = tf.reduce_sum(mask) - tf.cast(batch, tf.float32)\n",
+ " child_index = tf.tile(tf.expand_dims(tf.range(0, max_len), 1), [1, batch])\n",
+ " t = tf.transpose(heads)\n",
+ " broadcasted = tf.broadcast_to(batch_index, tf.shape(t))\n",
+ " concatenated = tf.transpose(tf.concat([tf.expand_dims(broadcasted, axis = 0),\n",
+ " tf.expand_dims(t, axis = 0),\n",
+ " tf.expand_dims(child_index, axis = 0)], axis = 0))\n",
+ " loss_arc = tf.gather_nd(loss_arc, concatenated)\n",
+ " loss_arc = tf.transpose(loss_arc, [1, 0])\n",
+ " \n",
+ " t = tf.transpose(types)\n",
+ " broadcasted = tf.broadcast_to(batch_index, tf.shape(t))\n",
+ " concatenated = tf.transpose(tf.concat([tf.expand_dims(broadcasted, axis = 0),\n",
+ " tf.expand_dims(child_index, axis = 0),\n",
+ " tf.expand_dims(t, axis = 0)], axis = 0))\n",
+ " loss_type = tf.gather_nd(loss_type, concatenated)\n",
+ " loss_type = tf.transpose(loss_type, [1, 0])\n",
+ " self.cost = (tf.reduce_sum(-loss_arc) / num) + (tf.reduce_sum(-loss_type) / num)\n",
+ " self.optimizer = tf.train.AdamOptimizer(\n",
+ " learning_rate = learning_rate\n",
+ " ).minimize(self.cost)\n",
+ " \n",
+ " mask = tf.sequence_mask(self.lengths, maxlen = self.maxlen)\n",
+ " \n",
+ " self.prediction = tf.boolean_mask(self.tags_seq, mask)\n",
+ " mask_label = tf.boolean_mask(self.types, mask)\n",
+ " correct_pred = tf.equal(tf.cast(self.prediction, tf.int32), mask_label)\n",
+ " correct_index = tf.cast(correct_pred, tf.float32)\n",
+ " self.accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
+ " \n",
+ " self.prediction = tf.cast(tf.boolean_mask(self.heads_seq, mask), tf.int32)\n",
+ " mask_label = tf.boolean_mask(self.heads, mask)\n",
+ " correct_pred = tf.equal(self.prediction, mask_label)\n",
+ " correct_index = tf.cast(correct_pred, tf.float32)\n",
+ " self.accuracy_depends = tf.reduce_mean(tf.cast(correct_pred, tf.float32))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING:tensorflow:From /home/husein/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py:507: calling count_nonzero (from tensorflow.python.ops.math_ops) with axis is deprecated and will be removed in a future version.\n",
+ "Instructions for updating:\n",
+ "reduction_indices is deprecated, use axis instead\n",
+ "WARNING:tensorflow:\n",
+ "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
+ "For more information, please see:\n",
+ " * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
+ " * https://github.com/tensorflow/addons\n",
+ " * https://github.com/tensorflow/io (for I/O related ops)\n",
+ "If you depend on functionality not listed there, please file an issue.\n",
+ "\n",
+ "WARNING:tensorflow:From /home/husein/testing/xlnet.py:253: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
+ "\n",
+ "WARNING:tensorflow:From /home/husein/testing/xlnet.py:253: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.\n",
+ "\n",
+ "WARNING:tensorflow:From /home/husein/testing/modeling.py:686: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
+ "\n",
+ "INFO:tensorflow:memory input None\n",
+ "INFO:tensorflow:Use float type \n",
+ "WARNING:tensorflow:From /home/husein/testing/modeling.py:693: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
+ "\n",
+ "WARNING:tensorflow:From /home/husein/testing/modeling.py:797: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
+ "Instructions for updating:\n",
+ "Use keras.layers.dropout instead.\n",
+ "WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING:tensorflow:From /home/husein/testing/modeling.py:99: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
+ "Instructions for updating:\n",
+ "Use keras.layers.dense instead.\n",
+ "WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting >: AssertionError: Bad argument number for Name: 3, expecting 4\n",
+ "WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting