Eval bug: model producing gibberish for Orion14b-chat #12411
Description
Name and Version
llama-cli --version
version: 5170 (658987c)
built with MSVC 19.43.34808.0 for x64
Operating systems
Windows
GGML backends
CPU
Hardware
Intel(R) Core(TM) Ultra 7 155H
Models
mmnga/OrionStarAI-Orion-14B-Chat-RAG-gguf
Problem description & steps to reproduce
llama-cli.exe -m "C:\xverse-13B\orionstar-14b-chat-rag.gguf" -i --jinja
First Bad Commit
Description
I am encountering an issue while using the latest version of llama.cpp ([b5170]) for inference with a gguf model, where the model generates nonsensical outputs.
Please fix this issue, Thanks a lot!
Additional Context
Previous Version > https://github.com/ggml-org/llama.cpp/releases/tag/b1990
In the previous version ([llama-b1990]), there were no issues with inference producing nonsensical outputs. Therefore, I suspect that some changes introduced in the latest version may have caused this abnormal output from the model. I would appreciate it if you could investigate this issue.
Thank you for your attention to this matter!
Relevant log output
C:\Users\aspwo\Downloads\llama-b5170-bin-win-openblas-x64>llama-cli.exe -m "C:\xverse-13B\orionstar-14b-chat-rag.gguf" -i --jinja
build: 5170 (658987cf) with MSVC 19.43.34808.0 for x64
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 22 key-value pairs and 444 tensors from C:\xverse-13B\orionstar-14b-chat-rag.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = orion
llama_model_loader: - kv 1: general.file_type u32 = 15
llama_model_loader: - kv 2: general.name str = Orion-14B-Chat-RAG
llama_model_loader: - kv 3: orion.tensor_data_layout str = Meta AI original pth
llama_model_loader: - kv 4: orion.context_length u32 = 4096
llama_model_loader: - kv 5: orion.embedding_length u32 = 5120
llama_model_loader: - kv 6: orion.block_count u32 = 40
llama_model_loader: - kv 7: orion.feed_forward_length u32 = 15360
llama_model_loader: - kv 8: orion.attention.head_count u32 = 40
llama_model_loader: - kv 9: orion.attention.head_count_kv u32 = 40
llama_model_loader: - kv 10: orion.attention.layer_norm_epsilon f32 = 0.000010
llama_model_loader: - kv 11: tokenizer.ggml.model str = llama
llama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,84608] = ["<unk>", "<s>", "</s>", " ", "▁▁...
llama_model_loader: - kv 13: tokenizer.ggml.scores arr[f32,84608] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,84608] = [2, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 15: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 17: tokenizer.ggml.padding_token_id u32 = 0
llama_model_loader: - kv 18: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 19: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 20: general.quantization_version u32 = 2
llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}{% if lo...
llama_model_loader: - type f32: 162 tensors
llama_model_loader: - type q4_K: 241 tensors
llama_model_loader: - type q6_K: 41 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 8.21 GiB (4.86 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 4
load: token to piece cache size = 0.4578 MB
print_info: arch = orion
print_info: vocab_only = 0
print_info: n_ctx_train = 4096
print_info: n_embd = 5120
print_info: n_layer = 40
print_info: n_head = 40
print_info: n_head_kv = 40
print_info: n_rot = 128
print_info: n_swa = 0
print_info: n_swa_pattern = 1
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 1
print_info: n_embd_k_gqa = 5120
print_info: n_embd_v_gqa = 5120
print_info: f_norm_eps = 1.0e-05
print_info: f_norm_rms_eps = 0.0e+00
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 15360
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = linear
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 4096
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 14B
print_info: model params = 14.50 B
print_info: general.name = Orion-14B-Chat-RAG
print_info: vocab type = SPM
print_info: n_vocab = 84608
print_info: n_merges = 0
print_info: BOS token = 1 '<s>'
print_info: EOS token = 2 '</s>'
print_info: UNK token = 0 '<unk>'
print_info: PAD token = 0 '<unk>'
print_info: LF token = 64 '<0x0A>'
print_info: EOG token = 2 '</s>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 0 repeating layers to GPU
load_tensors: offloaded 0/41 layers to GPU
load_tensors: CPU_AARCH64 model buffer size = 6187.50 MiB
load_tensors: CPU_Mapped model buffer size = 8402.56 MiB
...............................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = 0
llama_context: freq_base = 10000.0
llama_context: freq_scale = 1
llama_context: CPU output buffer size = 0.32 MiB
init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 40, can_shift = 1
init: CPU KV buffer size = 3200.00 MiB
llama_context: KV self size = 3200.00 MiB, K (f16): 1600.00 MiB, V (f16): 1600.00 MiB
llama_context: CPU compute buffer size = 368.01 MiB
llama_context: graph nodes = 1447
llama_context: graph splits = 162 (with bs=512), 1 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 16
main: chat template is available, enabling conversation mode (disable it with -no-cnv)
main: chat template example:
Human: You are a helpful assistant
Hello
Assistant: </s>Hi there</s>Human: How are you?
Assistant:
system_info: n_threads = 16 (n_threads_batch = 16) / 22 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
main: interactive mode on.
sampler seed: 1928939138
sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 0
== Running in interactive mode. ==
- Press Ctrl+C to interject at any time.
- Press Return to return control to the AI.
- To return control without starting a new line, end your input with '/'.
- If you want to submit another line, end your input with '\'.
- Not using system message. To change it, set a different value via -sys PROMPT
> Hello!
Hello! I'm Assistant, a large language model trained by OpenAI. I'm here to help you with any questions you have. Is there anything specific you'd like to know?
> What's AI?
😊 How can I help you with any questions or anything you need any questions or anything I can help you with any questions or anything you need any questions I'm here. How can I can help you can answer your questions I am here. I am a large AI. I can help you. What you are you can answer.I can answer you. you, chat. I can help me.
> What ?
. your or you. answer. chat. you:. or. or any? to, in. you I or you can. answer, chat or,
>