Eval bug: b5922 causes gibberish on context shift #14759
Description
Name and Version
version: 5922 (01612b7)
built with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.3.0
Operating systems
Mac
GGML backends
Metal
Hardware
MacBook Pro M3 36GB RAM
Models
lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf - https://huggingface.co/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF
Problem description & steps to reproduce
When I run llama-server with:
./build/bin/llama-server -m /Users/matt/.cache/lm-studio/models/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf -ngl 99 -c 512 -b 512 --temp 0 --seed 0 -n 500 --keep 454
And send curl command (goal of this curl command is to set n_keep to original prompt have the engine continuously shift while telling a story. First tokens of prompt are gibberish + "Ignore that. Tell me a story about math" to try and expose issues potential issues with n_pos tracking/shifting):
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"model": "qwen",
"messages": [
{
"role": "system",
"content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."
},
{
"role": "user",
"content": "adfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjklldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfjkldaadfadfjkldaadfjkldaadfjkldaadfjkldaadfjklldaadfjkldaadfadfjkldaadfjkldaadfjkld\n\nIgnore that. Tell me a story about math"
}
],
"max_tokens": 500,
"temperature": 0,
"seed": 0
}'
I get gibberish:
{"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","content":"In the mystical land of Zephyria, there lived a young mathematician named Zephyr. Zephyr was known for his love of numbers and his ability to solve complex problems with ease. One day, Zephyr received a challenge from his teacher, a wise old sage named Luminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminuminumin"}}],"created":1752849262,"model":"qwen","system_fingerprint":"b5922-01612b74","object":"chat.completion","usage":{"completion_tokens":500,"prompt_tokens":452,"total_tokens":952},"id":"chatcmpl-lutgjBwvkp8nYfQzVKPyUMw3BpeArCqf","timings":{"prompt_n":452,"prompt_ms":125.558,"prompt_per_token_ms":0.277783185840708,"prompt_per_second":3599.9299128689527,"predicted_n":500,"predicted_ms":432.805,"predicted_per_token_ms":0.86561,"predicted_per_second":1155.2546758933006}}
when context shift starts:
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
If I move one commit back to b5921, the same server command and request gives me coherent output when shifting:
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"In the mystical land of Zephyria, there lived a young mathematician named Zephyr. Zephyr was known for his love of numbers and his ability to solve complex problems with ease. One day, Zephyr received a challenge from his teacher, a wise old sage named Luminar.\n\nLuminar asked Zephyr to solve a math problem that was too difficult for him to solve alone. Zephyr was a bit nervous, but he knew that he had to do it. Luminar explained that the problem was about finding the area of a rectangle, which is a fundamental concept in mathematics.\n\nThe problem was as follows: \"Find the area of a rectangle with a length of 10 units and a width of 5 units.\"\n\nTo solve this, we can use the formula for the area of a rectangle: Area = length × width.\n\nPlugging in the values, we get:\n\nArea = 10 × 5 = 50 square units.\n\nSo, the area of the rectangle is 50 square units."}}],"created":1752849509,"model":"qwen","system_fingerprint":"b5921-086cf81e","object":"chat.completion","usage":{"completion_tokens":216,"prompt_tokens":452,"total_tokens":668},"id":"chatcmpl-RlJyy8WmaUB8O3V2v3KhiYSqdL8UFOS2","timings":{"prompt_n":452,"prompt_ms":121.433,"prompt_per_token_ms":0.2686570796460177,"prompt_per_second":3722.2171897260214,"predicted_n":216,"predicted_ms":1354.96,"predicted_per_token_ms":6.272962962962963,"predicted_per_second":159.4143000531381}}
First Bad Commit
Relevant log output
### Working (`b5921`)
./build/bin/llama-server -m /Users/matt/.cache/lm-studio/models/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF/Qwen2.5-0.5B-Instruct-Q4_K_
M.gguf -ngl 99 -c 512 -b 512 --temp 0 --seed 0 -n 500 --keep 454
build: 5921 (086cf81e) with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.3.0
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12
system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | ACCELERATE = 1 | REPACK = 1 |
main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 11
main: loading model
srv load_model: loading model '/Users/matt/.cache/lm-studio/models/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf'
llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 290 tensors from /Users/matt/.cache/lm-studio/models/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Qwen2.5 0.5B Instruct
llama_model_loader: - kv 3: general.finetune str = Instruct
llama_model_loader: - kv 4: general.basename str = Qwen2.5
llama_model_loader: - kv 5: general.size_label str = 0.5B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-0...
llama_model_loader: - kv 8: general.base_model.count u32 = 1
llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 0.5B
llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen
llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-0.5B
llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"]
llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"]
llama_model_loader: - kv 14: qwen2.block_count u32 = 24
llama_model_loader: - kv 15: qwen2.context_length u32 = 32768
llama_model_loader: - kv 16: qwen2.embedding_length u32 = 896
llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 4864
llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 14
llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 2
llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 22: general.file_type u32 = 15
llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645
llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
llama_model_loader: - kv 33: general.quantization_version u32 = 2
llama_model_loader: - type f32: 121 tensors
llama_model_loader: - type q5_0: 132 tensors
llama_model_loader: - type q8_0: 13 tensors
llama_model_loader: - type q4_K: 12 tensors
llama_model_loader: - type q6_K: 12 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 373.71 MiB (6.35 BPW)
load: special tokens cache size = 22
load: token to piece cache size = 0.9310 MB
print_info: arch = qwen2
print_info: vocab_only = 0
print_info: n_ctx_train = 32768
print_info: n_embd = 896
print_info: n_layer = 24
print_info: n_head = 14
print_info: n_head_kv = 2
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 64
print_info: n_embd_head_v = 64
print_info: n_gqa = 7
print_info: n_embd_k_gqa = 128
print_info: n_embd_v_gqa = 128
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 4864
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = -1
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 32768
print_info: rope_finetuned = unknown
print_info: model type = 1B
print_info: model params = 494.03 M
print_info: general.name = Qwen2.5 0.5B Instruct
print_info: vocab type = BPE
print_info: n_vocab = 151936
print_info: n_merges = 151387
print_info: BOS token = 151643 '<|endoftext|>'
print_info: EOS token = 151645 '<|im_end|>'
print_info: EOT token = 151645 '<|im_end|>'
print_info: PAD token = 151643 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<|endoftext|>'
print_info: EOG token = 151645 '<|im_end|>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors: Metal_Mapped model buffer size = 373.71 MiB
load_tensors: CPU_Mapped model buffer size = 137.94 MiB
.................................................
llama_context: constructing llama_context
llama_context: non-unified KV cache requires ggml_set_rows() - forcing unified KV cache
llama_context: n_seq_max = 1
llama_context: n_ctx = 512
llama_context: n_ctx_per_seq = 512
llama_context: n_batch = 512
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = 0
llama_context: kv_unified = true
llama_context: freq_base = 1000000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (512) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Pro
ggml_metal_init: picking default device: Apple M3 Pro
ggml_metal_load_library: using embedded metal library
ggml_metal_init: GPU name: Apple M3 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001)
ggml_metal_init: simdgroup reduction = true
ggml_metal_init: simdgroup matrix mul. = true
ggml_metal_init: has residency sets = false
ggml_metal_init: has bfloat = true
ggml_metal_init: use bfloat = false
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 28991.03 MB
ggml_metal_init: skipping kernel_get_rows_bf16 (not supported)
ggml_metal_init: skipping kernel_set_rows_bf16 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32_c4 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32_1row (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32_l4 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_bf16 (not supported)
ggml_metal_init: skipping kernel_mul_mv_id_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mm_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_bf16_f16 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h64 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h80 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h96 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h112 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h192 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_hk192_hv128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h256 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_hk576_hv512 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h64 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h96 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h192 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_hk192_hv128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h256 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_hk576_hv512 (not supported)
ggml_metal_init: skipping kernel_cpy_f32_bf16 (not supported)
ggml_metal_init: skipping kernel_cpy_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_cpy_bf16_bf16 (not supported)
llama_context: CPU output buffer size = 0.58 MiB
llama_kv_cache_unified: Metal KV buffer size = 6.00 MiB
llama_kv_cache_unified: size = 6.00 MiB ( 512 cells, 24 layers, 1/ 1 seqs), K (f16): 3.00 MiB, V (f16): 3.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context: Metal compute buffer size = 298.50 MiB
llama_context: CPU compute buffer size = 2.76 MiB
llama_context: graph nodes = 966
llama_context: graph splits = 2
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 512
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv init: initializing slots, n_slots = 1
slot init: id 0 | task -1 | new slot n_ctx_slot = 512
main: model loaded
main: chat template, chat_template: {%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role }}
{%- if message.content %}
{{- '\n' + message.content }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{{- tool_call.arguments | tojson }}
{{- '}\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}
, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:8080 - starting the main loop
srv update_slots: all slots are idle
srv params_from_: Chat format: Content-only
slot launch_slot_: id 0 | task 0 | processing task
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 512, n_keep = 454, n_prompt_tokens = 452
slot update_slots: id 0 | task 0 | kv cache rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 452, n_tokens = 452, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_past = 452, n_tokens = 452
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot release: id 0 | task 0 | stop processing: n_past = 499, truncated = 1
slot print_timing: id 0 | task 0 |
prompt eval time = 121.43 ms / 452 tokens ( 0.27 ms per token, 3722.22 tokens per second)
eval time = 1354.96 ms / 216 tokens ( 6.27 ms per token, 159.41 tokens per second)
total time = 1476.39 ms / 668 tokens
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
### Broken (`b5922`)
./build/bin/llama-server -m /Users/matt/.cache/lm-studio/models/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf -ngl 99 -c 512 -b 512 --temp 0 --seed 0 -n 500 --keep 454
build: 5922 (01612b74) with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.3.0
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12
system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | ACCELERATE = 1 | REPACK = 1 |
main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 11
main: loading model
srv load_model: loading model '/Users/matt/.cache/lm-studio/models/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf'
llama_model_load_from_file_impl: using device Metal (Apple M3 Pro) - 27647 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 290 tensors from /Users/matt/.cache/lm-studio/models/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Qwen2.5 0.5B Instruct
llama_model_loader: - kv 3: general.finetune str = Instruct
llama_model_loader: - kv 4: general.basename str = Qwen2.5
llama_model_loader: - kv 5: general.size_label str = 0.5B
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-0...
llama_model_loader: - kv 8: general.base_model.count u32 = 1
llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 0.5B
llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen
llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-0.5B
llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"]
llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"]
llama_model_loader: - kv 14: qwen2.block_count u32 = 24
llama_model_loader: - kv 15: qwen2.context_length u32 = 32768
llama_model_loader: - kv 16: qwen2.embedding_length u32 = 896
llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 4864
llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 14
llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 2
llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 22: general.file_type u32 = 15
llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645
llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643
llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
llama_model_loader: - kv 33: general.quantization_version u32 = 2
llama_model_loader: - type f32: 121 tensors
llama_model_loader: - type q5_0: 132 tensors
llama_model_loader: - type q8_0: 13 tensors
llama_model_loader: - type q4_K: 12 tensors
llama_model_loader: - type q6_K: 12 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 373.71 MiB (6.35 BPW)
load: special tokens cache size = 22
load: token to piece cache size = 0.9310 MB
print_info: arch = qwen2
print_info: vocab_only = 0
print_info: n_ctx_train = 32768
print_info: n_embd = 896
print_info: n_layer = 24
print_info: n_head = 14
print_info: n_head_kv = 2
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 64
print_info: n_embd_head_v = 64
print_info: n_gqa = 7
print_info: n_embd_k_gqa = 128
print_info: n_embd_v_gqa = 128
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 4864
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = -1
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 32768
print_info: rope_finetuned = unknown
print_info: model type = 1B
print_info: model params = 494.03 M
print_info: general.name = Qwen2.5 0.5B Instruct
print_info: vocab type = BPE
print_info: n_vocab = 151936
print_info: n_merges = 151387
print_info: BOS token = 151643 '<|endoftext|>'
print_info: EOS token = 151645 '<|im_end|>'
print_info: EOT token = 151645 '<|im_end|>'
print_info: PAD token = 151643 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151659 '<|fim_prefix|>'
print_info: FIM SUF token = 151661 '<|fim_suffix|>'
print_info: FIM MID token = 151660 '<|fim_middle|>'
print_info: FIM PAD token = 151662 '<|fim_pad|>'
print_info: FIM REP token = 151663 '<|repo_name|>'
print_info: FIM SEP token = 151664 '<|file_sep|>'
print_info: EOG token = 151643 '<|endoftext|>'
print_info: EOG token = 151645 '<|im_end|>'
print_info: EOG token = 151662 '<|fim_pad|>'
print_info: EOG token = 151663 '<|repo_name|>'
print_info: EOG token = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors: Metal_Mapped model buffer size = 373.71 MiB
load_tensors: CPU_Mapped model buffer size = 137.94 MiB
.................................................
llama_context: constructing llama_context
llama_context: non-unified KV cache requires ggml_set_rows() - forcing unified KV cache
llama_context: n_seq_max = 1
llama_context: n_ctx = 512
llama_context: n_ctx_per_seq = 512
llama_context: n_batch = 512
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = 0
llama_context: kv_unified = true
llama_context: freq_base = 1000000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (512) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Pro
ggml_metal_init: picking default device: Apple M3 Pro
ggml_metal_load_library: using embedded metal library
ggml_metal_init: GPU name: Apple M3 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001)
ggml_metal_init: simdgroup reduction = true
ggml_metal_init: simdgroup matrix mul. = true
ggml_metal_init: has residency sets = false
ggml_metal_init: has bfloat = true
ggml_metal_init: use bfloat = false
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 28991.03 MB
ggml_metal_init: skipping kernel_get_rows_bf16 (not supported)
ggml_metal_init: skipping kernel_set_rows_bf16 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32_c4 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32_1row (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_f32_l4 (not supported)
ggml_metal_init: skipping kernel_mul_mv_bf16_bf16 (not supported)
ggml_metal_init: skipping kernel_mul_mv_id_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mm_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_bf16_f16 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h64 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h80 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h96 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h112 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h192 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_hk192_hv128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_h256 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_bf16_hk576_hv512 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h64 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h96 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h192 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_hk192_hv128 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_h256 (not supported)
ggml_metal_init: skipping kernel_flash_attn_ext_vec_bf16_hk576_hv512 (not supported)
ggml_metal_init: skipping kernel_cpy_f32_bf16 (not supported)
ggml_metal_init: skipping kernel_cpy_bf16_f32 (not supported)
ggml_metal_init: skipping kernel_cpy_bf16_bf16 (not supported)
llama_context: CPU output buffer size = 0.58 MiB
llama_kv_cache_unified: Metal KV buffer size = 6.00 MiB
llama_kv_cache_unified: size = 6.00 MiB ( 512 cells, 24 layers, 1/ 1 seqs), K (f16): 3.00 MiB, V (f16): 3.00 MiB
llama_kv_cache_unified: LLAMA_SET_ROWS=0, using old ggml_cpy() method for backwards compatibility
llama_context: Metal compute buffer size = 298.50 MiB
llama_context: CPU compute buffer size = 2.76 MiB
llama_context: graph nodes = 966
llama_context: graph splits = 2
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 512
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv init: initializing slots, n_slots = 1
slot init: id 0 | task -1 | new slot n_ctx_slot = 512
main: model loaded
main: chat template, chat_template: {%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role }}
{%- if message.content %}
{{- '\n' + message.content }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{{- tool_call.arguments | tojson }}
{{- '}\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}
, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:8080 - starting the main loop
srv update_slots: all slots are idle
srv params_from_: Chat format: Content-only
slot launch_slot_: id 0 | task 0 | processing task
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 512, n_keep = 454, n_prompt_tokens = 452
slot update_slots: id 0 | task 0 | kv cache rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 452, n_tokens = 452, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_past = 452, n_tokens = 452
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot update_slots: id 0 | task 0 | slot context shift, n_keep = 454, n_left = 57, n_discard = 28
slot release: id 0 | task 0 | stop processing: n_past = 503, truncated = 1
slot print_timing: id 0 | task 0 |
prompt eval time = 128.77 ms / 452 tokens ( 0.28 ms per token, 3510.22 tokens per second)
eval time = 442.56 ms / 500 tokens ( 0.89 ms per token, 1129.79 tokens per second)
total time = 571.33 ms / 952 tokens
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200