run_localGPT.py runs but very slow - How to run it faster?

[ahmetax]

I have NVIDIA GeForce GTX 1060, 6GB. My OS is Ubuntu 22.04. Python 3.10.11. Ram 32GB.
ingest.py runs with no problems.
When I use default values of the installation in run_localGPT.py, I get memory error after submitting a question.
Then, after an internet search, I added 3 more parameters to the method AutoModelForCasualLM.from_pretrained(
...,
offload_folder="offload",
offload_state_dict=True,
max_memory={0:"2GB","cpu":"24GB"}
) in run_localGPT.py file.
Also, before running the script, I give a console command:
export PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:256
Now, run_localGPT.py can create answers to my questions.
Average execution times are as follow:
Model preparation ~ 400-450 seconds
Answering ~ 80-100 seconds
Are these values normal?
My graphics card has 6 GB memory, but I can use only 2GB of it in max_memory parameter. Is it possible to increase that value?
My model_id is "TheBloke/vicuna-7B-1.1-HF".
What can I do in order to get better results?

[KonradHoeffner]

Your GPU is probably not used at all, which would explain the slow speed in answering.

You are using a 7 billion parameter model without quantization, which means that with 16 bit weights ( = 2 byte), your model is 14 GB in size.

As your GPU only has 6 GB it will probably not be useful for any reasonable model.

For example, I have a 3070 with 8 GB and even with the 2-bit quantized version (which probably has a very low quality) of a 7 billion parameter model I run out of GPU RAM due to cuBLAS requiring extra space.

[ahmetax]

Thanks. I could run the script limiting to use only 2GB of GPU ram a few times. But it was not stable. Then I forgave trying to use GPU.