32GB RAM and 3080 8GB but still not enough?

[curvedJavelin]

I'm currently testing this out. Thank you for putting this out!

I was wondering if a 3080 8GB and 32GB RAM won't be enough to run this?
During ingest.py run, it finishes with no error.

But whenever running run_localGPT.py, I'm getting a bunch of DefaultCPUAllocator: not enough memory: errors.
And some Unable to load weights error.

Would this be due to low VRAM? If so, can I use a different model like this https://huggingface.co/TheBloke/vicuna-7B-GPTQ-4bit-128g?

And just in case, as mentioned here:

#20

You will need at least 11GB of RAM to run this. Both Instructor Embeddings and Vicuna-7B are running on the GPU. You can try the 4-Bit quantized version of Vicuna with it. That will probably work.

Does it need 11GB when running ingest.py or for also run_localGPT.py? And also, is that VRAM or RAM?

Thanks!

[bbartling]

Have you ever figured this out? What hardware does one need?

I dont have a GPU machine and curious to experiment on CPU. I have 16.0 GB (15.8 GB usable) Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz 2.30 GHz