Misc. bug: [llama.android] Model keeps replying and cannot be stopped normally until it exceeds the context

[codezjx]

Name and Version

The latest b4491 version with llama.android example

Operating systems

Other? (Please let us know in description)

Which llama.cpp modules do you know to be affected?

Other (Please specify in the next section)

Command line

No response

Problem description & steps to reproduce

1. Run llama.android example
2. Loaded with SmolLM2 model
3. Send with chat template (User message: Tell a joke), the template are generated by the common_chat_apply_template() method

Code location: llama.cpp/examples/llama.android/app/src/main/java/com/example/llama/MainViewModel.kt

val smollm2msg = "<|im_start|>system\n" +
    "You are a helpful AI assistant<|im_end|>\n" +
    "<|im_start|>user\n" +
    "$text<|im_end|>\n" +
    "<|im_start|>assistant\n"
viewModelScope.launch {
    llamaAndroid.send(smollm2msg)
        .catch {
            Log.e(tag, "send() failed", it)
            messages += it.message!!
        }
        .collect { messages = messages.dropLast(1) + (messages.last() + it) }
}

4. Issue reproduce, the model keeps replying and cannot be stopped normally until it exceeds the context, and it will contain tokens such as <|im_start|> and <|im_end|>

Comparison test:

This problem cannot be reproduced using the command line program llama-cli with the same LLM model

./llama-cli -m models/smollm2-360m-instruct-q8_0.gguf -p "You are a helpful assistant" -cnv

First Bad Commit

Since the establishment of llama.android

Relevant log output

No response

[wqerrewetw]

https://github.com/ggerganov/llama.cpp/blob/681149ced2582f48c24792403df1307fed5eb951/examples/llama.android/llama/src/main/cpp/llama-android.cpp#L359

you can change the line from

common_tokenize(context, text, 1);

to

common_tokenize(context, text, true, true);

[codezjx]

@wqerrewetw Yes, I also found this code when comparing llama-cli code yesterday. The reason is that the chat template format message is used, but it is not set to parse the special token, resulting in the end method llama_vocab_is_eog() not being triggered.

I have submitted a pull request to fix this issue: #11270, add field formatChat to control whether to parse special tokens when send message. The user can set formatChat according to their actual situation.

// Explicitly set formatChat = true
llamaAndroid.send(msg, true)