Skip to content

Commit d7f340b

Browse files
authored
set bos to cls if missing [no ci]
ggml-ci
1 parent a854897 commit d7f340b

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

gguf-py/gguf/vocab.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,8 @@ def _try_load_from_tokenizer_json(self, path: Path) -> bool:
160160
special_cls = (tokenizer_config or {}).get('cls_token')
161161
special_eos = (tokenizer_config or {}).get('eos_token')
162162
special_sep = (tokenizer_config or {}).get('sep_token')
163+
if not special_bos and special_cls and tokenizer_config:
164+
tokenizer_config['bos_token'] = special_bos = special_cls
163165
if not special_eos and special_sep and tokenizer_config:
164166
tokenizer_config['eos_token'] = special_eos = special_sep
165167
post_processor = tokenizer.get('post_processor', {})

0 commit comments

Comments
 (0)