Bug: cannot find tokenizer merges in model file

[nd791899]

What happened?

When I use transformers==4.45.1 and convert llama.cpp to the file used by ollama, there is no error, but when I load the model with ollama, the error ollama cannot find tokenizer merges in model file appears

Name and Version

所有版本

What operating system are you seeing the problem on?

No response

Relevant log output

No response

[patrick60507]

same problem

[nd791899]

gguf-py/gguf/vocab.py
def add_to_gguf(self, gw: GGUFWriter, quiet: bool = False) -> None:
report：
Adding merges requested but no merges found, output may be non-functional.

and in _try_load_from_tokenizer_json function：

def _try_load_from_tokenizer_json(self, path: Path) -> bool:
    tokenizer_file = path / 'tokenizer.json'
    if tokenizer_file.is_file():
        with open(tokenizer_file, encoding = 'utf-8') as f:
        if self.load_merges:
            merges = tokenizer.get('model', {}).get('merges')
            if isinstance(merges, list) and merges and isinstance(merges[0], str):
                self.merges = merges
        added_tokens = tokenizer.get('added_tokens', {})

isinstance(merges[0], str)
but transformers==4.45.1 generate tokenizer.json ，in tokenizer.json merges is list。

Can be compatible?

[Vaibhavs10]

Hey hey, I'm VB from the open source team at Hugging Face. I can confirm that this is due to an update we've made to tokenizers - we persists merges as a list vs strings.

Everything should work on transformers 4.44.0 however from 4.45.0 onward it won't work and we'd need to add support for it.

For reference this is the tokenizers PR that introduced it: huggingface/tokenizers#909

[danielhanchen]

There are some temporary fixes which downgrade transformers to 4.44.2 for Unsloth members here: unslothai/unsloth#1065 and unslothai/unsloth#1062

[ggerganov]

Tagging @compilade for any insights how to best resolve this.