Skip to content

Commit b17e981

Browse files
authored
revert vocab_size() change [no ci]
1 parent f2d876a commit b17e981

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

convert_hf_to_gguf.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3668,7 +3668,7 @@ def _xlmroberta_set_vocab(self) -> None:
36683668
toktypes: list[int] = [SentencePieceTokenTypes.UNUSED] * vocab_size
36693669

36703670
if isinstance(tokenizer, SentencePieceProcessor):
3671-
for token_id in range(vocab_size):
3671+
for token_id in range(tokenizer.vocab_size()):
36723672
piece = tokenizer.IdToPiece(token_id)
36733673
text = piece.encode("utf-8")
36743674
score = tokenizer.GetScore(token_id)

0 commit comments

Comments
 (0)