Docker image not starting the server loop (since b4234)

[medihack]

I am using the llama.cpp:server Docker image as specified in the examples in a Docker compose file. This always worked great, but since b4234 the server and container directly stops without starting the server loop.
Do I have to start the container now somehow else? I am grateful for any help!

Here is the service in my compose file (using a MacBook Pro M4):

  llamacpp_cpu:
    image: ghcr.io/ggerganov/llama.cpp:server-b4234
    ports:
      - "${LLAMACPP_DEV_PORT:-8080}:8080"
    environment:
      LLAMA_ARG_MODEL_URL: https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_ENDPOINT_METRICS: 1
      LLAMA_ARG_PORT: 8080

And here the log output of the container:

build: 4234 (3420909d) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for aarch64-linux-gnu
system info: n_threads = 16, n_threads_batch = 16, total_threads = 16

system_info: n_threads = 16 (n_threads_batch = 16) / 16 | 

main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 15
main: loading model
srv    load_model: loading model '/root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf'
common_download_file: no previous model file found /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf
curl_perform_with_retry: Trying to download from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf (attempt 1 of 3)...
common_download_file: trying to download model from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf to /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf (server_etag:"216b187b137f65f185b86716e34b5d7b-7", server_last_modified:Thu, 31 Oct 2024 20:38:59 GMT)...
curl_perform_with_retry: Trying to download from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf (attempt 1 of 3)...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1144  100  1144    0     0   8799      0 --:--:-- --:--:-- --:--:--  8799
100  100M  100  100M    0     0  10.9M      0  0:00:09  0:00:09 --:--:-- 11.4M
common_download_file: file metadata saved: /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf.json
llama_model_loader: loaded meta data with 33 key-value pairs and 272 tensors from /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = SmolLM2 135M Instruct
llama_model_loader: - kv   3:                       general.organization str              = HuggingFaceTB
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = SmolLM2
llama_model_loader: - kv   6:                         general.size_label str              = 135M
llama_model_loader: - kv   7:                          llama.block_count u32              = 30
llama_model_loader: - kv   8:                       llama.context_length u32              = 8192
llama_model_loader: - kv   9:                     llama.embedding_length u32              = 576
llama_model_loader: - kv  10:                  llama.feed_forward_length u32              = 1536
llama_model_loader: - kv  11:                 llama.attention.head_count u32              = 9
llama_model_loader: - kv  12:              llama.attention.head_count_kv u32              = 3
llama_model_loader: - kv  13:                       llama.rope.freq_base f32              = 100000.000000
llama_model_loader: - kv  14:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  15:                 llama.attention.key_length u32              = 64
llama_model_loader: - kv  16:               llama.attention.value_length u32              = 64
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 49152
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 64
llama_model_loader: - kv  20:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  21:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = smollm
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,49152]   = ["<|endoftext|>", "<|im_start|>", "<|...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,49152]   = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  26:                      tokenizer.ggml.merges arr[str,48900]   = ["Ġ t", "Ġ a", "i n", "h e", "Ġ Ġ...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  29:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  31:                    tokenizer.chat_template str              = {% for message in messages %}{% if lo...
llama_model_loader: - kv  32:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   61 tensors
llama_model_loader: - type q5_0:  166 tensors
llama_model_loader: - type q8_0:   15 tensors
llama_model_loader: - type q4_K:   16 tensors
llama_model_loader: - type q6_K:   14 tensors
llm_load_vocab: special tokens cache size = 17
llm_load_vocab: token to piece cache size = 0.3170 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 49152
llm_load_print_meta: n_merges         = 48900
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 576
llm_load_print_meta: n_layer          = 30
llm_load_print_meta: n_head           = 9
llm_load_print_meta: n_head_kv        = 3
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 64
llm_load_print_meta: n_embd_head_v    = 64
llm_load_print_meta: n_gqa            = 3
llm_load_print_meta: n_embd_k_gqa     = 192
llm_load_print_meta: n_embd_v_gqa     = 192
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 1536
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 100000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 134.52 M
llm_load_print_meta: model size       = 98.87 MiB (6.17 BPW) 
llm_load_print_meta: general.name     = SmolLM2 135M Instruct
llm_load_print_meta: BOS token        = 1 '<|im_start|>'
llm_load_print_meta: EOS token        = 2 '<|im_end|>'
llm_load_print_meta: EOT token        = 0 '<|endoftext|>'
llm_load_print_meta: UNK token        = 0 '<|endoftext|>'
llm_load_print_meta: PAD token        = 0 '<|endoftext|>'
llm_load_print_meta: LF token         = 143 'Ä'
llm_load_print_meta: EOG token        = 0 '<|endoftext|>'
llm_load_print_meta: EOG token        = 2 '<|im_end|>'
llm_load_print_meta: max token length = 162

Then it stops here.

For comparison, here is the log output of b4231 which correctly starts the server loop:

build: 4231 (43957ef2) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for aarch64-linux-gnu
system info: n_threads = 16, n_threads_batch = 16, total_threads = 16

system_info: n_threads = 16 (n_threads_batch = 16) / 16 | CPU : NEON = 1 | ARM_FMA = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | 

main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 15
main: loading model
srv    load_model: loading model '/root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf'
common_download_file: no previous model file found /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf
curl_perform_with_retry: Trying to download from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf (attempt 1 of 3)...
common_download_file: trying to download model from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf to /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf (server_etag:"216b187b137f65f185b86716e34b5d7b-7", server_last_modified:Thu, 31 Oct 2024 20:38:59 GMT)...
curl_perform_with_retry: Trying to download from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf (attempt 1 of 3)...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1144  100  1144    0     0   9945      0 --:--:-- --:--:-- --:--:--  9945
100  100M  100  100M    0     0  10.6M      0  0:00:09  0:00:09 --:--:-- 11.0M
common_download_file: file metadata saved: /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf.json
llama_model_loader: loaded meta data with 33 key-value pairs and 272 tensors from /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = SmolLM2 135M Instruct
llama_model_loader: - kv   3:                       general.organization str              = HuggingFaceTB
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = SmolLM2
llama_model_loader: - kv   6:                         general.size_label str              = 135M
llama_model_loader: - kv   7:                          llama.block_count u32              = 30
llama_model_loader: - kv   8:                       llama.context_length u32              = 8192
llama_model_loader: - kv   9:                     llama.embedding_length u32              = 576
llama_model_loader: - kv  10:                  llama.feed_forward_length u32              = 1536
llama_model_loader: - kv  11:                 llama.attention.head_count u32              = 9
llama_model_loader: - kv  12:              llama.attention.head_count_kv u32              = 3
llama_model_loader: - kv  13:                       llama.rope.freq_base f32              = 100000.000000
llama_model_loader: - kv  14:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  15:                 llama.attention.key_length u32              = 64
llama_model_loader: - kv  16:               llama.attention.value_length u32              = 64
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 49152
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 64
llama_model_loader: - kv  20:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  21:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = smollm
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,49152]   = ["<|endoftext|>", "<|im_start|>", "<|...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,49152]   = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  26:                      tokenizer.ggml.merges arr[str,48900]   = ["Ġ t", "Ġ a", "i n", "h e", "Ġ Ġ...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  29:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  31:                    tokenizer.chat_template str              = {% for message in messages %}{% if lo...
llama_model_loader: - kv  32:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   61 tensors
llama_model_loader: - type q5_0:  166 tensors
llama_model_loader: - type q8_0:   15 tensors
llama_model_loader: - type q4_K:   16 tensors
llama_model_loader: - type q6_K:   14 tensors
llm_load_vocab: special tokens cache size = 17
llm_load_vocab: token to piece cache size = 0.3170 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 49152
llm_load_print_meta: n_merges         = 48900
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 576
llm_load_print_meta: n_layer          = 30
llm_load_print_meta: n_head           = 9
llm_load_print_meta: n_head_kv        = 3
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 64
llm_load_print_meta: n_embd_head_v    = 64
llm_load_print_meta: n_gqa            = 3
llm_load_print_meta: n_embd_k_gqa     = 192
llm_load_print_meta: n_embd_v_gqa     = 192
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 1536
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 100000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 134.52 M
llm_load_print_meta: model size       = 98.87 MiB (6.17 BPW) 
llm_load_print_meta: general.name     = SmolLM2 135M Instruct
llm_load_print_meta: BOS token        = 1 '<|im_start|>'
llm_load_print_meta: EOS token        = 2 '<|im_end|>'
llm_load_print_meta: EOT token        = 0 '<|endoftext|>'
llm_load_print_meta: UNK token        = 0 '<|endoftext|>'
llm_load_print_meta: PAD token        = 0 '<|endoftext|>'
llm_load_print_meta: LF token         = 143 'Ä'
llm_load_print_meta: EOG token        = 0 '<|endoftext|>'
llm_load_print_meta: EOG token        = 2 '<|im_end|>'
llm_load_print_meta: max token length = 162
llm_load_tensors:   CPU_Mapped model buffer size =    98.87 MiB
........................................................................
llama_new_context_with_model: n_seq_max     = 2
llama_new_context_with_model: n_ctx         = 4096
llama_new_context_with_model: n_ctx_per_seq = 2048
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 100000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_kv_cache_init:        CPU KV buffer size =    90.00 MiB
llama_new_context_with_model: KV self size  =   90.00 MiB, K (f16):   45.00 MiB, V (f16):   45.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.38 MiB
llama_new_context_with_model:        CPU compute buffer size =    98.25 MiB
llama_new_context_with_model: graph nodes  = 966
llama_new_context_with_model: graph splits = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 2
slot         init: id  0 | task -1 | new slot n_ctx_slot = 2048
slot         init: id  1 | task -1 | new slot n_ctx_slot = 2048
main: model loaded
main: chat template, built_in: 1, chat_example: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
request: GET /health 127.0.0.1 200
build: 4231 (43957ef2) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for aarch64-linux-gnu
system info: n_threads = 16, n_threads_batch = 16, total_threads = 16

system_info: n_threads = 16 (n_threads_batch = 16) / 16 | CPU : NEON = 1 | ARM_FMA = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | 

main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 15
main: loading model
srv    load_model: loading model '/root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf'
common_download_file: no previous model file found /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf
curl_perform_with_retry: Trying to download from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf (attempt 1 of 3)...
common_download_file: trying to download model from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf to /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf (server_etag:"216b187b137f65f185b86716e34b5d7b-7", server_last_modified:Thu, 31 Oct 2024 20:38:59 GMT)...
curl_perform_with_retry: Trying to download from https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q4_K_M.gguf (attempt 1 of 3)...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1144  100  1144    0     0   9945      0 --:--:-- --:--:-- --:--:--  9945
100  100M  100  100M    0     0  10.6M      0  0:00:09  0:00:09 --:--:-- 11.0M
common_download_file: file metadata saved: /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf.json
llama_model_loader: loaded meta data with 33 key-value pairs and 272 tensors from /root/.cache/llama.cpp/SmolLM2-135M-Instruct-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = SmolLM2 135M Instruct
llama_model_loader: - kv   3:                       general.organization str              = HuggingFaceTB
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = SmolLM2
llama_model_loader: - kv   6:                         general.size_label str              = 135M
llama_model_loader: - kv   7:                          llama.block_count u32              = 30
llama_model_loader: - kv   8:                       llama.context_length u32              = 8192
llama_model_loader: - kv   9:                     llama.embedding_length u32              = 576
llama_model_loader: - kv  10:                  llama.feed_forward_length u32              = 1536
llama_model_loader: - kv  11:                 llama.attention.head_count u32              = 9
llama_model_loader: - kv  12:              llama.attention.head_count_kv u32              = 3
llama_model_loader: - kv  13:                       llama.rope.freq_base f32              = 100000.000000
llama_model_loader: - kv  14:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  15:                 llama.attention.key_length u32              = 64
llama_model_loader: - kv  16:               llama.attention.value_length u32              = 64
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:                           llama.vocab_size u32              = 49152
llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 64
llama_model_loader: - kv  20:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  21:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = smollm
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,49152]   = ["<|endoftext|>", "<|im_start|>", "<|...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,49152]   = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  26:                      tokenizer.ggml.merges arr[str,48900]   = ["Ġ t", "Ġ a", "i n", "h e", "Ġ Ġ...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  29:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  31:                    tokenizer.chat_template str              = {% for message in messages %}{% if lo...
llama_model_loader: - kv  32:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   61 tensors
llama_model_loader: - type q5_0:  166 tensors
llama_model_loader: - type q8_0:   15 tensors
llama_model_loader: - type q4_K:   16 tensors
llama_model_loader: - type q6_K:   14 tensors
llm_load_vocab: special tokens cache size = 17
llm_load_vocab: token to piece cache size = 0.3170 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 49152
llm_load_print_meta: n_merges         = 48900
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 576
llm_load_print_meta: n_layer          = 30
llm_load_print_meta: n_head           = 9
llm_load_print_meta: n_head_kv        = 3
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 64
llm_load_print_meta: n_embd_head_v    = 64
llm_load_print_meta: n_gqa            = 3
llm_load_print_meta: n_embd_k_gqa     = 192
llm_load_print_meta: n_embd_v_gqa     = 192
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 1536
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 100000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 134.52 M
llm_load_print_meta: model size       = 98.87 MiB (6.17 BPW) 
llm_load_print_meta: general.name     = SmolLM2 135M Instruct
llm_load_print_meta: BOS token        = 1 '<|im_start|>'
llm_load_print_meta: EOS token        = 2 '<|im_end|>'
llm_load_print_meta: EOT token        = 0 '<|endoftext|>'
llm_load_print_meta: UNK token        = 0 '<|endoftext|>'
llm_load_print_meta: PAD token        = 0 '<|endoftext|>'
llm_load_print_meta: LF token         = 143 'Ä'
llm_load_print_meta: EOG token        = 0 '<|endoftext|>'
llm_load_print_meta: EOG token        = 2 '<|im_end|>'
llm_load_print_meta: max token length = 162
llm_load_tensors:   CPU_Mapped model buffer size =    98.87 MiB
........................................................................
llama_new_context_with_model: n_seq_max     = 2
llama_new_context_with_model: n_ctx         = 4096
llama_new_context_with_model: n_ctx_per_seq = 2048
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 100000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_kv_cache_init:        CPU KV buffer size =    90.00 MiB
llama_new_context_with_model: KV self size  =   90.00 MiB, K (f16):   45.00 MiB, V (f16):   45.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.38 MiB
llama_new_context_with_model:        CPU compute buffer size =    98.25 MiB
llama_new_context_with_model: graph nodes  = 966
llama_new_context_with_model: graph splits = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 2
slot         init: id  0 | task -1 | new slot n_ctx_slot = 2048
slot         init: id  1 | task -1 | new slot n_ctx_slot = 2048
main: model loaded
main: chat template, built_in: 1, chat_example: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
request: GET /health 127.0.0.1 200

[slaren]

The CPU docker images currently do not work on ARM, because they use functionality that at the moment is only available on x86. You should be able to build an ARM image by changing the build command in cpu.Dockerfile to something like this:

cmake -S . -B build -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=<arm architecture to build for> -DLLAMA_CURL=ON -DCMAKE_BUILD_TYPE=Release

[medihack]

Thanks for the reply. But really?! The README says Apple silicon is a first-class citizen and the images worked fine until b4231. Then I consider this a bug.

[slaren]

There was a fix merged today, so images should work again. However you will likely get better performance if you build it yourself. The docker images also do not support Metal.

[medihack]

Thanks a lot for the clarification and the fix! We only use the image in development with SmolLM. So, performance does not matter that much in this use case. Thanks again!

[jacebenson]

This thread helped me. I'm not sure if my problem was the same but I when I was calling docker to run llamacpp server I kept getting a arch error that the arch didn't match. I'll open a discussion tonight with details. However, going back to -b4231 worked.

Doesn't work: docker run -d --name llamacpp -v $INSTALL_DIR/models:/root/.cache/llama.cpp -p $PORT:8000 ghcr.io/ggerganov/llama.cpp:server --hf-repo $HF_REPO --hf-file $HF_FILE --ctx-size $CTX_SIZE --port $PORT --api-key $API_KEY --host 0.0.0.0

This gives the error, "The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested"

Works: docker run -d --name llamacpp -v $INSTALL_DIR/models:/root/.cache/llama.cpp -p $PORT:8000 ghcr.io/ggerganov/llama.cpp:server-b4231 --hf-repo $HF_REPO --hf-file $HF_FILE --ctx-size $CTX_SIZE --port $PORT --api-key $API_KEY --host 0.0.0.0