v0.4.3 · benbrandt text-splitter · Discussion #38 · GitHub

benbrandt
Aug 10, 2023
Maintainer

What's Changed

Support impl ChunkSizer for &Tokenizer and &CoreBPE, allowing for generating chunks based off of a reference to a tokenizer as well, instead of requiring ownership. by @benbrandt in Support references to tokenizers #37

Full Changelog: v0.4.2...v0.4.3

This discussion was created from the release v0.4.3.

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment