-
Notifications
You must be signed in to change notification settings - Fork 104
Description
I am getting output like following. Space between words is missing. If i use hugging face python tokenizer its give good output.
Onceuponatime,therewasamagicalkingdomwherethesunshonebrightandthebirdssangsweetmelodies.Inthiskingdom,therelivedawiseoldwizardwhohadthepowertograntwishes.Helivedinabeautifulcastlesurroundedbyalushforest,andhewaslovedbyallwhoknewhim. Oneday,ayounggirlnamedLilycametothewizard'scastle,seekingawish.Shehadheardofthewizard'smagicalpowersandwantedtomakeherdreamscometrue.Thewizardlistenedtoherwishand,withasmile,grantedit. Lilywasoverjoyedandthankedthewizardforhiskindness.Thewizardsmiledandsaid,"Youarewelcome,mydear.Iamheretohelpthosewhocometomewithkindheartsandgoodintentions." Fromthatdayon,Lilyvisitedthewizardeveryday,andhegrantedhermanymorewishes.Shebecameafrequentvisitortothecastle,andthewizardandshebecamegoodfriends.