This repository was archived by the owner on Sep 10, 2025. It is now read-only.

Tokenizers tokenizer#1261

Merged

Jack-Khuu merged 6 commits intopytorch:mainfrom

gabe-l-hart:TokenizersTokenizer-1251

Nov 5, 2024

Commits on Oct 30, 2024

feat(tokenizer): Add an abstract base class for additional tokenizer support
gabe-l-hart
committed
feat(tokenizers): Add a python impl of the Tokenizer interface using tokenizers
gabe-l-hart
committed
feat(builder): Add support for using the TokenizersTokenizer in builder
gabe-l-hart
committed
feat(tokenizers): Add and plumb the option to use the "tokenizers" tokenizer
gabe-l-hart
committed
fix(tokenizers): Fix how bos/eos tokens are parsed from tokenizers (lib)
gabe-l-hart
committed
fix(hf_tokenizer): Rename to HFTokenizer and corresponding flags
gabe-l-hart
committed