KeyError: ('torch', 'BoolStorage') when converting NSQL model

[alishobeiri]

Hi all,

Forgive me if this is a stupid question as I am still learning things with llama.cpp.

I am trying to convert https://huggingface.co/NumbersStation/nsql-350M to llama.cpp using the convert.py script.

When I do so, I get the following error:

Loading model file nsql-350M/pytorch_model.bin
Traceback (most recent call last):
  File "/content/llama.cpp/convert.py", line 1326, in <module>
    main()
  File "/content/llama.cpp/convert.py", line 1306, in main
    model_plus = load_some_model(args.model)
  File "/content/llama.cpp/convert.py", line 1221, in load_some_model
    models_plus.append(lazy_load_file(path))
  File "/content/llama.cpp/convert.py", line 1012, in lazy_load_file
    return lazy_load_torch_file(fp, path)
  File "/content/llama.cpp/convert.py", line 883, in lazy_load_torch_file
    model = unpickler.load()
  File "/content/llama.cpp/convert.py", line 872, in find_class
    return self.CLASSES[(module, name)]
KeyError: ('torch', 'BoolStorage')

Do you have any tips on what I can do to get around this? Can I change the module type from BoolStorage to IntStorage or similar if it is already pretrained?

Thanks in advance.

[klosax]

Do you have any tips on what I can do to get around this?

You cant since the model you are trying to use is not based on the LLaMA architecture.

[alishobeiri]

Thank you, is there a way I can make the conversion myself or is it a big lift?

[klosax]

It is not the conversion to ggml file format that is the main problem. You need inference code written for each architecture. llama.cpp is currently only supporting the LLaMA architecture. Inference code written for some other architectures can be found at https://github.com/ggerganov/ggml/tree/master/examples but none of them supports the architecture used by the model you are trying to use.