Tokenizer 1.32.0

guillaumekln released this 25 Jul 09:56

· 64 commits to master since this release

4807909

New features

Add property pyonmttok.Vocab.counters to retrieve the number of occurrences of each token

Fixes and improvements

Update pybind11 to 2.10.0
Update cxxopts to 3.0.0

Assets 2