[Feature] User-configurable stop sequences

[cebtenzzre]

Feature Request

This is part of the solution to dealing with broken models that emit stop tokens that are not specified in their HF configuration (#2167) and is also a commonly used feature in other similar apps.

---

From an earlier related draft by @cebtenzzre

• some models output the wrong EOS token, so this is important
• special tokens show up as blank in output because we use llama_token_to_piece with special=False, so they aren't even considered for our current hardcoded reverse prompts

Related things to fix:

• Don't loop forever when the model generates a special token other than EOS (High CPU usage in chat with Hermes 2 Pro Mistral 7B after generation has finished #2167)
• Print special token content
• Honor both EOS tokens specified in the HF model's generation_config.json (requires llama.cpp modification)

[benja0x40]

This would make GPT4All more robust and would give users more control on stop conditions.

[ThiloteE]

Related upstream issues / PRs:

• llama : keep track of all EOG tokens in the vocab ggml-org/llama.cpp#9609
• Bug: Qwen2.5-Coder variants do not properly stop in FIM mode ggml-org/llama.cpp#9606

[brankoradovanovic-mcom]

+1 for this one. Today I've tried a Qwen2.5 family model for the first time, and it's a pain: the model emits <|endoftext|> and then continues generating stuff unrelated to the prompt.

When using Python bindings, one can handle this with a custom callback, e.g. I use this catch-all handler:

def stop_on_token_callback(token_id, token_string):
    stop_on_token_callback.response += token_string
   
    print(token_string, end='', flush=True)          

    for stop_string in ['</s>', '###', '\n\n\n', '<|im_end|>', '<|endoftext|>']:
        if stop_on_token_callback.response.endswith(stop_string):
            return False

    return True

Crude but effective, and easy to implement.