Skip to content

Commit 50a1a7c

Browse files
authored
[Tokenizer][Fix] Fix SegFault when analyzing tokenizers without tokenizer.json (#2532)
Previously the tokenizer would segfault when analyzing a tokenizer that did not have a tokenizer.json file. This is due to `TokenizerInfo()` is called previously, which creates a null object. This PR fixes this problem.
1 parent 206db55 commit 50a1a7c

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

cpp/tokenizers/tokenizers.cc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ TokenizerInfo Tokenizer::DetectTokenizerInfo(const String& path_str) {
153153
if (!std::filesystem::exists(path)) {
154154
LOG(WARNING) << "Tokenizer info is not detected as tokenizer.json is not found. The default "
155155
<< "tokenizer info will be used.";
156-
return TokenizerInfo();
156+
return TokenizerInfo(make_object<TokenizerInfoNode>());
157157
}
158158

159159
std::string tokenizer_json = LoadBytesFromFile(path.string());

0 commit comments

Comments
 (0)