how to convert qlora trained model into a GGUF model?

[Kainat-R]

I am having issues with converting qlora trained model(4bit) into a GGUF model using this script (https://github.com/ggerganov/llama.cpp/blob/master/examples/make-ggml.py) mainly because qlora trained model does not have a config.json. it has a adapter_config.json.

on running this command
!python3 llama.cpp/examples/make-ggml.py resultsalpaca/checkpoint200 --model_type llama --outname alpaca-GGUF --quants F32
this is the error
Traceback (most recent call last): File "llama.cpp/examples/make-ggml.py", line 98, in <module> main(args.model, args.model_type, args.outname, args.outdir, args.quants, args.keep_fp16) File "llama.cpp/examples/make-ggml.py", line 60, in main raise Exception(f"Could not find config.json in {model}") Exception: Could not find config.json in resultsalpaca/checkpoint200

i manually loaded the config.json of the llama 13b model which i'm using as a base model into the folder then the below error came up.
Building llama.cpp make: *** No rule to make target 'quantize'. Stop. Traceback (most recent call last): File "llama.cpp/examples/make-ggml.py", line 98, in <module> main(args.model, args.model_type, args.outname, args.outdir, args.quants, args.keep_fp16) File "llama.cpp/examples/make-ggml.py", line 65, in main subprocess.run(f"cd .. && make quantize", shell=True, check=True) File "/usr/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'cd .. && make quantize' returned non-zero exit status 2.

is there a way to do it without config.json file? if not then how to correctly generate config.json file of a qlora trained model?

[Green-Sky]

you are dealing with a lora, which is an adapter for a model. if you want to use the lora, first convert it using convert-lora-to-ggml.py. then you can load the model and the lora. (it requires the base model).
you can also merge the lora into the base model using the export-lora program.

[puyuanOT]

Thank you for the answer. I have a follow-up question. After converting LoRA weights to ggml using convert-lora-to-ggml.py, which script should be used to load both the base model and the lora_ggml?

[UmutAlihan]

I am also looking forward for the answer to this followup question :)