Skip to content

Cleanup chat template API#650

Merged
MarcusDunn merged 2 commits intoutilityai:mainfrom
vlovich:fix-chat-template
Feb 14, 2025
Merged

Cleanup chat template API#650
MarcusDunn merged 2 commits intoutilityai:mainfrom
vlovich:fix-chat-template

Conversation

@vlovich
Copy link
Copy Markdown
Contributor

@vlovich vlovich commented Feb 13, 2025

  1. Make the template not an optional for apply_chat_template. This ensures you don't accidentally use the chatml template.
  2. Improve performance for the expected case of using get_chat_template by returning a new LlamaChatTemplate struct that internally stores the string as a CString. Unless you try to explicitly create a copy or print, there's no extra copy into a Rust string that's created. Similarly, get_chat_template -> apply_chat_template no longer copies the template string.
  3. Improve documentation including documentating what the add_ass parameter does and suggestions on what values you probably want to use. Additionally, I've made get_chat_template and apply_chat_template docs refer to one another to make it easier to discover how to use this.

This resolves #640 and #641

@MarcusDunn hope you don't mind, but instead of separate &str vs &CStr, I've instead created a new LlamaChatTemplate struct that internally stores the CString with convenience methods to easily retrieve &str/String from it or to construct it from a &str. I found that to be a bit cleaner.

1. Make the template not an optional for apply_chat_template. This
   ensures you don't accidentally use the chatml template.
2. Improve performance for the expected case of using get_chat_template
   by returning a new LlamaChatTemplate struct that internally stores
   the string as a CString. Unless you try to explicitly create a copy or
   print, there's no extra copy into a Rust string that's created.
   Similarly, get_chat_template -> apply_chat_template no longer copies
   the template string.
3. Improve documentation including documentating what the add_ass
   parameter does and suggestions on what values you probably want to
   use. Additionally, I've made get_chat_template and
   apply_chat_template docs refer to one another to make it easier to
   discover how to use this.
@MarcusDunn MarcusDunn merged commit 00f4163 into utilityai:main Feb 14, 2025
2 of 5 checks passed
@vlovich vlovich deleted the fix-chat-template branch February 14, 2025 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apply chat template defaults are confusing & under documented

2 participants