Do we need any specific GPU architecture?

[8bitaby]

While there are a wide variety of GPUs available is there any specific GPU architecture required to run localGPT?
I see CUDA here so can it run GPUs other than NVIDIA? If not how can we make a platform independent LLM system?

[alonet23]

I am using RTX2070 with 8gb VRAM but always end up error saying

RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 8.00 GiB total capacity; 6.53 GiB already allocated; 0 bytes free; 6.58 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

any suggestions?

[PromtEngineer]

You will need at least 11GB of RAM to run this. Both Instructor Embeddings and Vicuna-7B are running on the GPU. You can try the 4-Bit quantized version of Vicuna with it. That will probably work.

[ranger2408]

how do i do that? I only have 8gigs also. The problem also exists when creating the database, can you put this requirement on the readme so i dont have to waste my time?