Tried running it on version koboldcpp-1.114.1 with low ram

[xelercen]

Hardware:
Motherboard: LGA 1200. Chipset: H510. CPU: Pentium (2 cores, 4 threads).
RAM: 4 GB DDR4.
Video Card: Tesla P40 (24 GB VRAM), 2 cards. 24 x 2 VRAM = 48 GB VRAM.
OS: Windows 10 Ent (latest service pack)

Additional Information:
I tried running it on version koboldcpp-1.114.1, model qwen3.6-27B.
I tried using CUDA and Vulkan, but neither option allowed the video card to run the model.
koboldcpp-1.114.1 crashes when loading the model, displaying something like a memory allocation error in the console.
I notice that the number of allocated bytes in the console error when the program crashes is approximately equal to the size of the model on the disk!
Got it!

Let's try to fix it:
Create a swap partition on the SSD. Then enable the "mlock" option in koboldcpp to prevent data from being loaded into memory.

"It works, but it's broken," and only on the koboldcpp-oldpc.exe version. Run it via the Vulkan API.
The funny thing is, the graphics card isn't loaded, and the graphics card's memory is completely empty... but it works, as far as I understand, all the calculations are done on the CPU.

Let's try to fix it:
Change several parameters and get an error about not being able to allocate 1 GB of memory for some Vulkan buffer. I notice that the ZGPU program sees Vulkan, but not its version? That's strange, I think the version on this driver should be 1.3.
Great, then let's try running it via CUDA.

ZGPU detects the CUDA version (I think it's the framework version, driver version 12.2, and the hardware version for the P40 card is 6.0). It crashes when running koboldcpp.exe and koboldcpp-oldpc.exe.

Another funny thing was that the card's OpenGL version was 1.1, although it should have been 4.6, I think, but I fixed that. OpenGL appeared when turning on the virtual screen.

I thought there might be a driver issue. I tried 9 different drivers, all of which gave similar results (it doesn't work).

1. Is it possible to run such a neural network using only the video card's VRAM for storing weights and 4 GB of RAM + SAWP SSD 32 GB? So, should the VRAM be loaded into the card's memory in chunks?

2. Koboldcpp.exe version 1.114.1 doesn't work on the CPU (without graphics cards), and koboldcpp-oldpc.exe automatically sets "Very Old CPU FailSafe" when loading from a file and only works in that file... Is a Pentium on LGA1200 an old processor, considering its 2020 technology? Why then does koboldcpp.exe work in simple CPU mode on my second motherboard with a Xeon E7-88XXv4 from 2016?

3. Is there any way to check the problem? I've spent 30 hours and still can't get it to work, tried different gguf models. I think there could be more than one issue, for example, the fact that the processor is a Pentium (even though it's new and supports AVX and AVX2), the CUDA driver version doesn't match the library version in the koboldcpp software, there's not enough RAM.

4. Is it possible to run through Vulkan with full loading of the neural network and KV cache into the card memory?

5. Is it possible to run through CUDA with full loading of the neural network and KV cache into the card memory?

6. Which should be upgraded first, the processor or the RAM?

[LostRuins]

Your GPU is fine, but your CPU and RAM are gonna make this really hard. The Motherboard: LGA 1200. Chipset: H510. CPU: Pentium (2 cores, 4 threads). does not have the AVX instruction, which means only the oldpc builds (with the Older PC options) will work.

I would strongly suggest upgrading your RAM and CPU, if you must choose, upgrade the RAM first but both really need upgrading. The Xeon E7-88XXv4 is a much better CPU so I would use that if possible.

That said, if you still want to test, let's try loading a tiny model first.

1. Download https://huggingface.co/unsloth/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_0.gguf
2. Download the https://github.com/LostRuins/koboldcpp/releases/download/v1.117.1/koboldcpp-oldpc.exe and select the "Older PC Vulkan" option. Load in the above model, do not change anything else from the defaults.
3. Launch. It should offload all layers into your VRAM. Check if it works.

Does this work?

[wbruna]

Create a swap partition on the SSD. Then enable the "mlock" option in koboldcpp to prevent data from being loaded into memory.

That option does (or attempts to do) the opposite of what you are trying to do. From the help:

  --usemlock, --mlock   Enables mlock, preventing the RAM used to load the
                        model from being paged out. Not usually recommended.

Note "preventing from being paged out" means the same as "preventing from being swapped out".

I'd recommend --usemmap instead, so the model can be loaded directly from the page cache.