Skip to content

Commit fe8d4df

Browse files
committed
Correctly identify LF token for GPT-2 style BPE tokenizer
1 parent 6171c9d commit fe8d4df

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/llama-vocab.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1687,7 +1687,7 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) {
16871687
GGML_ASSERT(!ids.empty() && "model vocab missing newline token");
16881688
linefeed_id = ids[0];
16891689
} else {
1690-
const std::vector<int> ids = tokenize("\xC4\x8A", false); // U+010A
1690+
const std::vector<int> ids = tokenize("\n", false);
16911691

16921692
//GGML_ASSERT(!ids.empty() && "model vocab missing newline token");
16931693
if (ids.empty()) {

0 commit comments

Comments
 (0)