What is the best way to add new words to model?

[info2000]

I'm using the es_core_news_lg model to do textcat task
On the eval results, I got this
ℹ 500000 vectors (500000 unique keys, 300 dimensions) ⚠ **70394 words in training data without vectors (3%)** 10 most common words without vectors: ' ' (19156), ' ' (5907), ' ' (778), 'Covid-19' (505), ' ' (331), ' ' (244), 'Wwwhatsnew.com' (240), 'Invítanos' (180), 'leyéndonos!La' (180), 'Covid' (172)

What is the best way to vectorize new words?

Thanks

[ljvmiranda921]

Hi @info2000 ,

What is the best way to vectorize new words?

To do this you need to train the vectors. However at this stage, a 3% out-of-vocab training data is still low. You might want to check first why there are many whitespace tokens in your dataset (note that the top 3 common "words" are all whitespaces). Perhaps clean the data a bit more and see how it works :)