Tokenizer somehow splits the word "id" into 2 sepperate tokens #10570
Answered
by
adrianeboyd
CraiyDev
asked this question in
Help: Other Questions
-
If you simply call |
Beta Was this translation helpful? Give feedback.
Answered by
adrianeboyd
Mar 29, 2022
Replies: 1 comment
-
By default the English tokenizer settings treat "id" as a variant of "I'd". Here's a comment about how to modify this: #10455 (comment) |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
adrianeboyd
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
By default the English tokenizer settings treat "id" as a variant of "I'd". Here's a comment about how to modify this: #10455 (comment)