Optimize statistical unigram tokenizer `decode_forward` #63

Open

aria42 wants to merge 1 commit intoJuliaText:masterfrom

aria42:faster-decode-forward

Commits on Dec 30, 2021

truncate decode_forward pass to skip spans longer than any word in SentencePieceModel vocab
aria42
committed