Fix tokenizer bug (#893)

* fix unified transformer dtype problem

* fix win dtype bug

* Fix plato-2 and plato-mini dtype bug

* Fix plato-2 tokenization

* Refine some doc

* Add general k support for topk sampling

* fix seed

* minor fix

* Fix unitransformer readme

* topk kernel optimization

* add unimo model and fix generate api

* add 3 datasets for unimo-text

* fix tokenizer bug

Co-authored-by: Jiaqi Liu <[email protected]>
Co-authored-by: liu zhengxi <[email protected]>

Author: 3 people

paddlenlp/transformers/unimo/tokenizer.py

@@ -75,7 +75,7 @@ class UNIMOTokenizer(PretrainedTokenizer):
             "unimo-text-1.0":
             "https://paddlenlp.bj.bcebos.com/models/transformers/unimo/unimo-text-1.0-vocab.txt",
             "unimo-text-1.0-large":
-            "https://paddlenlp.bj.bcebos.com/models/transformers/unimo/unimo-text-1.0-vocab-large.txt",
+            "https://paddlenlp.bj.bcebos.com/models/transformers/unimo/unimo-text-1.0-large-vocab.txt",
         }
     }
     pretrained_init_configuration = {