Skip to content
Discussion options

You must be logged in to vote

This is intended behavior, see #10455. The English tokenizer treats "id" as a typo for "I'd" by default, though you can change it.

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by polm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lang / en English language data and models feat / tokenizer Feature: Tokenizer
2 participants
Converted from issue

This discussion was converted from issue #11148 on July 19, 2022 03:47.