A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeni. Unlike traditional BPE, this tokeni selects merge candidates based on a normalised likelihood score, improving tokenisation quality by prioritising statistically significant merges.
WillKirkmanM/wordpiece
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|