You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Tokenizer][Fix] Fix SegFault when analyzing tokenizers without tokenizer.json (#2532)
Previously the tokenizer would segfault when analyzing a tokenizer
that did not have a tokenizer.json file.
This is due to `TokenizerInfo()` is called previously, which creates
a null object. This PR fixes this problem.
0 commit comments