We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent 4167468 commit 7227015Copy full SHA for 7227015
src/tiktoken.cpp
@@ -69,7 +69,7 @@ static Result<std::pair<std::string, uint64_t>> _parse(
69
// Tiktoken format
70
// https://github.com/openai/tiktoken/blob/main/tiktoken/load.py#L140 <base64
71
// encoded token str> <rank>
72
- auto pos = line.find(" ");
+ auto pos = line.find(' ');
73
TK_CHECK_OR_RETURN_ERROR(
74
pos != std::string::npos,
75
ParseFailure,
0 commit comments