Mac Metal support

[slavag]

Hi,
llama.cpp and llama-cpp-python are supporting Mac metal for sometime already, is there any plans to add support to metal to the lollms ?

Thanks

[LostRuins]

Mac metal is supported. Compile it with LLAMA_METAL=1 flag

[slavag]

Thanks, this is what I actually did but I don't see it using metal:

 koboldcpp git:(concedo) ././koboldcpp.py --model /Users/slava/Documents/Development/private/AI/h2ogpt/models/WizardLM-13B-1.0.ggmlv3.q5_1.bin  --unbantokens 
Welcome to KoboldCpp - Version 1.34
Warning: OpenBLAS library file not found. Non-BLAS library will be used.
Initializing dynamic library: koboldcpp.so
==========
Loading model: /Users/slava/Documents/Development/private/AI/h2ogpt/models/WizardLM-13B-1.0.ggmlv3.q5_1.bin 
[Threads: 4, BlasThreads: 4, SmartContext: False]

---
Identified as LLAMA model: (ver 5)
Attempting to Load...
---
System Info: AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | 
llama.cpp: loading model from /Users/slava/Documents/Development/private/AI/h2ogpt/models/WizardLM-13B-1.0.ggmlv3.q5_1.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32001
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 9 (mostly Q5_1)
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =    0.09 MB
llama_model_load_internal: mem required  = 11359.06 MB (+ 1608.00 MB per state)
llama_new_context_with_model: kv self size  = 1600.00 MB
Load Model OK: True
Embedded Kobold Lite loaded.
Starting Kobold HTTP Server on port 5001
Please connect to custom endpoint at http://localhost:5001

Please advise.

[LostRuins]

try using the --gpulayers flag to select some layers to offload.

You can also refer to this: #216

[slavag]

Thanks, working