not enough space in the scratch memory pool

[ahmetax]

I have a 6GB Graphics card. I am using:
EMBEDDING_MODEL_NAME = "hkunlp/instructor-large"
MODEL_ID = "TheBloke/orca_mini_3B-GGML"
MODEL_BASENAME = "orca-mini-3b.ggmlv3.q4_0.bin"

The first query is answered without a problem, but at the second query, I get similar errors as following:

Enter a query: what is the power of the congress
Llama.generate: prefix-match hit
ggml_new_tensor_impl: not enough space in the scratch memory pool (needed 271565568, available 268435456)
Segmentation fault

How can I prevent this error?

[PromtEngineer]

One recommendation that I will have is to change the embedding model. For starters, try the all-MiniLM-L6-v2. The instructorEmbedding model also runs on GPU but the all-MiniLM-L6-v2 embedding doesn't need GPU (will have an impact on performance). This will reduce your GPU vRAM usages.

The issue you are facing is coming from llamacpp and seems to be a common (here, and here).

[ahmetax]

Thanks. But I want to use GPU, because when I use GPU, I get answers 60 times faster than CPU.

[jnfarooq]

The LLM will still use GPU for generation, its simply changing the embedding model. This will improve your GPUT utilization.

[ahmetax]

Thanks PromptEngineer and jnfarooq. When I used all-MiniLM-L6-v2 as the embedding model, my problem is resolved.