Skip to content
Discussion options

You must be logged in to vote

The main Tokenizer used in the pipeline doesn't track this information at all (it would affect the performance), so there's no way to access/store this.

Tokenizer.explain is a much slower implementation of the same algorithm just for debugging purposes.

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by irowberry
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat / tokenizer Feature: Tokenizer
2 participants