Add preprocess script to use words as tokens with typo and rare word reduction#132

Open

dgcrouse wants to merge 12 commits intojcjohnson:masterfrom

dgcrouse:preprocessWords

Commits on Aug 12, 2016

Commits on Aug 18, 2016

Major rewrite to preprocessWords.py, fixed lots of errors, simplified and made more robust.
dgcrouse
committed

Commits on Aug 19, 2016

Fixed multiple issues pertaining to wildcards, e.g. overtraining, also fixed Unicode support
dgcrouse
committed
Added more flexibility for wildcards. Fixed bug where wildcards were capped rather than floored.
dgcrouse
committed

Commits on Aug 23, 2016

Commits on Jun 3, 2017

Merge branch 'New_Preprocess' into preprocessWords
dgcrouse
authored