Tool call support (generic + native for Llama, Functionary, Hermes, Mistral, Firefunction, DeepSeek) w/ lazy grammars#9639

Merged

ochafik merged 375 commits intoggml-org:masterfrom

ochafik:tool-call

Jan 30, 2025

This pull request is big! We're only showing the most recent 250 commits

Commits on Oct 27, 2024

nits
ochafik
committed

Commits on Oct 28, 2024

`tool-call`: slow tool call integration tests
ochafik
committed
space nits
ochafik
committed
`tool_call`: test no tool call on a real model + rename scenarios
ochafik
committed
`tool-call`: script to prefetch models used in server tests
ochafik
committed
Update tool_call.feature
ochafik
committed
`tool-call`: add tests: tool_call=none, parallel_tool_calls=true
ochafik
committed
`tool-call`: remove duplicate script to fetch templates
ochafik
committed
`agent`: simplify syntax (default tools to local w/ default port)
ochafik
committed
`tool-call`: use Q4_K_M models
ochafik
committed
`tool-call`: update scripts/fetch_server_test_models.py
ochafik
committed

Commits on Oct 29, 2024

`tool-call`: test Hermes-3-Llama-3.1-8B
Olivier Chafik
committed
`tool-call`: use functionary-small-v3.2-Q8_0.gguf in test (Q4_K_M too dumb for function call)
Olivier Chafik
committed
`tool-call`: force printing of lazy grammar trigger tokens to regularize function call parsing
Olivier Chafik
committed
nits
Olivier Chafik
committed

Commits on Oct 30, 2024

Commits on Oct 31, 2024

Merge remote-tracking branch 'origin/master' into tool-call
ochafik
committed
Update llama-sampling.cpp
ochafik
committed
`tool-call`: greedy sampling in server tests + tweak prompt
ochafik
committed
`tool-call`: nemo tweak (accept raw sql again)
ochafik
committed
Update tool_call.feature
ochafik
committed
`tool-call`: behaviour-based detection of template features
ochafik
committed
`tool-call`: code_interpreter & system + tool call support for all jinja templates!
ochafik
committed
`tool-call`: don't use -fa w/ Mistral-Nemo (hard crashes?)
ochafik
committed
`tool-call`: add LLAMA_UPDATE_GOLDENS env for test-chat-template
ochafik
committed
`tool-call`: functionary-small-v3.2 test now green
ochafik
committed
Update README.md
ochafik
committed
nits
ochafik
committed
Update README.md
ochafik
committed
`tool-call`: fix qwen template test
ochafik
committed
`agent`: add missing tool name in response!
ochafik
committed

Commits on Nov 9, 2024

Commits on Dec 5, 2024

agent: add --think "tool", default to local tools endpoint, support --temperature, fix --seed
ochafik
committed

Commits on Dec 6, 2024

Merge remote-tracking branch 'origin/master' into tool-call
ochafik
committed
agent: more robust squid config
ochafik
committed
agent: update readme
ochafik
committed
minja: remove tests (now in https://github.com/google/minja)
ochafik
committed
Update README.md
ochafik
committed

Commits on Dec 7, 2024

Commits on Dec 10, 2024

tool-calls: migrate tests to pytest
ochafik
committed

Commits on Dec 14, 2024

Merge remote-tracking branch 'origin/master' into tool-call
ochafik
committed
tool-calls: shorter name: grammar_triggers
ochafik
committed
Merge remote-tracking branch 'origin/master' into tool-call
ochafik
committed

Commits on Dec 15, 2024

tool-call: stabilize server tests
ochafik
committed

Commits on Dec 26, 2024

Commits on Dec 27, 2024

rm trailing spaces
ochafik
committed
Update fetch_server_test_models.py
ochafik
committed
Fix tool-call server tests
ochafik
committed
Simplify tool call grammars when there's only 1 tool
ochafik
committed

Commits on Dec 30, 2024

Commits on Jan 13, 2025

Merge remote-tracking branch 'origin/master' into jinja
ochafik
committed
Wire LLM_KV_TOKENIZER_CHAT_TEMPLATE_N in llama_model_chat_template
ochafik
committed
Refactor test-chat-template
ochafik
committed
Test templates w/ minja
ochafik
committed
Fix deprecation
ochafik
committed
Add --jinja to llama-run
ochafik
committed
Merge remote-tracking branch 'origin/master' into jinja
ochafik
committed
Update common_chat_format_example to use minja template wrapper
ochafik
committed
Test chat_template in e2e test
ochafik
committed
Update utils.py
ochafik
committed
Update test_chat_completion.py
ochafik
committed
Update run.cpp
ochafik
committed

Commits on Jan 14, 2025

Update arg.cpp
ochafik
authored
Merge branch 'jinja' into tool-call
ochafik
committed
Fix merge
ochafik
committed
Update test-chat-template.cpp
Olivier Chafik
committed
Merge remote-tracking branch 'origin/master' into tool-call
Olivier Chafik
committed
Update test-chat-template.cpp
Olivier Chafik
committed
Merge remote-tracking branch 'origin/master' into jinja
Olivier Chafik
committed

Commits on Jan 18, 2025

Refactor common_chat_* functions to accept minja template + use_jinja option
ochafik
committed
Refactor common_chat_* functions to accept minja template + use_jinja option
ochafik
committed
Merge remote-tracking branch 'origin/master' into jinja
ochafik
committed
Attempt to fix linkage of LLAMA_CHATML_TEMPLATE
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed
Revert LLAMA_CHATML_TEMPLATE refactor
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed
Fix fetch_server_test_models.py (avoid conv trap)
ochafik
committed
tools: greedy sampling in tests
ochafik
committed
tools: run tool call slow tests when SLOW_TESTS=1 (+ prefetch models)
ochafik
committed
Normalize newlines in test-chat-templates for windows tests
ochafik
committed
Forward decl minja::chat_template to avoid eager json dep
ochafik
committed
Flush stdout in chat template before potential crash
ochafik
committed
Fix copy elision warning
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed
Rm unused optional include
ochafik
committed
Add missing optional include to server.cpp
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed
Disable jinja test that has a cryptic windows failure
ochafik
committed
minja: fix vigogne (https://github.com/google/minja/pull/22)
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed

Commits on Jan 19, 2025

agent: add --greedy, --top-p, --top-k options
ochafik
committed

Commits on Jan 20, 2025

Apply suggestions from code review

authored
Finish suggested renamings
ochafik
committed
Move chat_templates inside server_context + remove mutex
ochafik
committed
Update --chat-template-file w/ recent change to --chat-template
ochafik
committed
Refactor chat template validation
ochafik
committed
Merge remote-tracking branch 'origin/master' into jinja
ochafik
committed
Guard against missing eos/bos tokens (null token otherwise throws in llama_vocab::impl::token_get_attr)
ochafik
committed
Warn against missing eos / bos tokens when jinja template references them
ochafik
committed
rename: common_chat_template[s]
ochafik
committed
reinstate assert on chat_templates.template_default
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed
apply renames from jinja branch
ochafik
committed
Update minja to https://github.com/google/minja/commit/b8437df626ac6cd0ce3b333b3c74ed1129c19f25
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed

Commits on Jan 21, 2025

fix std imports for gcc build
ochafik
committed
Update minja to https://github.com/google/minja/pull/25
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed
Update minja from https://github.com/google/minja/pull/27
ochafik
committed
Merge branch 'jinja' into tool-call
ochafik
committed
Merge remote-tracking branch 'origin/master' into tool-call
Olivier Chafik
committed
rm tests/test-minja from makefile
Olivier Chafik
committed
Remove examples/agent (moved to https://gist.github.com/ochafik/9246d289b7d38d49e1ee2755698d6c79)
Olivier Chafik
committed
Delete update_jinja_goldens.py
Olivier Chafik
committed

Commits on Jan 22, 2025

Push laziness down to grammar impl
Olivier Chafik
committed
minimize diffs
Olivier Chafik
committed
common_tool_call rename
Olivier Chafik
committed
shrink diff in json conversion code
Olivier Chafik
committed
Refactor string helpers into common
Olivier Chafik
committed
follow enum naming style for tool call styles
Olivier Chafik
committed
Factor string_join, string_split, string_repeat into common
Olivier Chafik
committed
json: refactor to surface a versatile builder
Olivier Chafik
committed
drop unused fs_list_files
Olivier Chafik
committed
Merge branch 'string_utils' into tool-call
Olivier Chafik
committed
Update common.cpp
Olivier Chafik
committed
Merge branch 'string_utils' into tool-call
Olivier Chafik
committed
drop llama_sampler_accept_str
Olivier Chafik
committed
more cleanups
Olivier Chafik
committed
Merge remote-tracking branch 'origin/master' into tool-call
Olivier Chafik
committed
merge common_tool_calls into common_chat_msg
Olivier Chafik
committed
smaller diff
Olivier Chafik
committed
nits
Olivier Chafik
committed
Update tool-call.cpp
Olivier Chafik
committed
Greedy sampling in tool call tests
Olivier Chafik
committed
Update test_chat_completion.py
Olivier Chafik
committed
Sync minja after https://github.com/google/minja/pull/29
Olivier Chafik
committed
Merge remote-tracking branch 'origin/master' into tool-call
Olivier Chafik
committed
fix common_chat_msg invocations
Olivier Chafik
committed
fix msg init warning
Olivier Chafik
committed
Update chat-template.hpp
Olivier Chafik
committed
Add grammar options + rename builder to common_grammar_builder
Olivier Chafik
committed
Update real tool call tests (use less models)
Olivier Chafik
committed
Fix lazy trigger handling
Olivier Chafik
committed

Commits on Jan 24, 2025

WIP chat handlers
Olivier Chafik
committed

Commits on Jan 25, 2025

tool-call: allow special tokens that are grammar triggers
Olivier Chafik
committed
Update test_chat_completion.py
Olivier Chafik
committed
jinja: don't add bos when jinja enabled
Olivier Chafik
committed
Update test_chat_completion.py
Olivier Chafik
committed

Commits on Jan 26, 2025

nit: trailing spaces
ochafik
committed
Merge branch 'tool-call' into tool-call-handler
ochafik
committed
sync: minja
ochafik
committed
reshuffle chat handlers
ochafik
committed
tool-call: fix functionary v3.1 required test
ochafik
committed

Commits on Jan 27, 2025

nits
ochafik
committed
tool-call: fix special handling of special trigger tokens (Nemo)
ochafik
committed
tool-call: remove nonsensical code_interpreter code
ochafik
committed
jinja: only add special tokens if template doesn't seem to handle them
ochafik
committed
tool-call: add weather tool e2e tests
ochafik
committed
tool-call: fix lazy grammar & mixed content + tool calls parsing
ochafik
committed
tool-call: compact json output to cap # tokens generated
ochafik
committed
Update test_chat_completion.py
ochafik
committed
Prepare DeepSeek-R1-Distill-Llama-8B support
ochafik
committed
DeepSeek-R1: implement grammar constraints
ochafik
committed
fix test-chat-handler grammar tests
ochafik
committed
Rehabilitate test_format_detection
ochafik
committed
updated tool call example to be less ambiguous (deepseek likes to rant about hello world)
ochafik
committed
Pass grammar laziness all the way down to sampler (need to print special trigger tokens e.g. for Nemo even w/ tool_choice=required)
ochafik
committed
Split e2e test_tool_call from test_chat_completion
ochafik
committed
comment out broken tests in test_tool_call.py
ochafik
committed
Update test-chat-handler.cpp
ochafik
committed

Commits on Jan 28, 2025

Fix Llama 3.1 (incl. constrained builtin tools e.g. `<|python_tag|>foo.call(arg=vallue)`)
ochafik
committed
Allow tool use + streaming
Olivier Chafik
committed
Cleanup dead code in llama_3_1 tool call code
Olivier Chafik
committed
Tool-call: do last partial parse upon limit stop
Olivier Chafik
committed
Update test-chat-handler.cpp
ochafik
committed
build: Add missing optional include for gcc
ochafik
committed
Disable slow tests where appropriate, + nits
ochafik
committed
Revert "Allow tool use + streaming"
ochafik
committed
Simplify parser defs (incremental parsing for streaming will need more thinking)
ochafik
committed
Add missing link dep for windows build
ochafik
committed
beef up test-chat-handler w/ delta expectations
ochafik
committed
Disable test-chat-handler on win32 like the other grammar-related tests
ochafik
committed
minja: sync on https://github.com/google/minja/pull/33
ochafik
committed
sync: minja
ochafik
committed

Commits on Jan 29, 2025

Fix firefunction w/ jinja: requires two variables, use the chat handlers everywhere templates are used
ochafik
committed
Revert breaking minja change
ochafik
committed
Text fireworks v2 template
ochafik
committed
nits
ochafik
committed
refactor test-chat-handler
ochafik
committed
rm dead code + nits
Olivier Chafik
committed
Split bulk of tool call tests to slow lane
Olivier Chafik
committed
Merge branch 'tool-call' of github.com:ochafik/llama.cpp into tool-call
ochafik
committed
rm unused templates, rename one
ochafik
committed
Update test_tool_call.py
ochafik
committed
tool-calls: disable crashing tests
ochafik
committed
nits
ochafik
committed
Merge remote-tracking branch 'origin/master' into tool-call
ochafik
committed
Create meta-llama-Llama-3.1-8B-Instruct.jinja
ochafik
committed
Move templates/ under models/
ochafik
committed
Unify llama 3.x chat handling again (allow `{"type": "function", "name": ...` prefix)
ochafik
committed
sync: minja
ochafik
committed
Rename: common/chat.*, common_chat_{inputs -> params}
ochafik
committed
Finish renaming of chat inputs vs. params [skip ci]
ochafik
committed
nits
ochafik
committed
Remove server tests LLAMA_CACHE override (tests are serial, and the cache is easier to prefill w/ scripts/fetch_server_test_models.py)
ochafik
committed
Add cli mode to test-chat to generate template summaries markdown
ochafik
committed
Somehow /* bad inside block comments, ok fine.
ochafik
committed
Add tool call to hot topics
ochafik
committed
Partial revert of LLAMA_CACHE=tmp (unless set explicitly in env)
ochafik
committed

Commits on Jan 30, 2025

Avoid passing tools twice in generic handler (now that minja passes them automatically when needed)
ochafik
committed
Unify content + message in server_task_result_cmpl_final (+ avoid string copy)
ochafik
committed
llama 3.1: allow `{name:` & `{function:` syntax even w/ builtin tools (70B model just likes that!)
ochafik
committed
Update tests readme + add raw output to verbose log
ochafik
committed
split chat handler vs. parser around enum again
ochafik
committed
nits
ochafik
committed
debug logs are back
ochafik
committed
rm unused llama_param
ochafik
committed
llama 3.2 1b now fails the weather tool call?
ochafik
committed
increase http timeout to 12
ochafik
committed
Merge remote-tracking branch 'origin/master' into tool-call
Olivier Chafik
committed
code style changes on test
ngxson
committed
simplify handle_apply_template
ngxson
committed
Fix debug + verbose
Olivier Chafik
committed
Update test_chat_completion.py
Olivier Chafik
committed
Update test_chat_completion.py
Olivier Chafik
committed
Update scripts/fetch_server_test_models.py to new compact hf_repo syntax + switch Hermes models
Olivier Chafik
committed
nit: fix py import
Olivier Chafik
committed
deprecate llama_sampler_init_grammar -> llama_sampler_grammar_init
Olivier Chafik
committed
add llama_sampler_init_grammar_lazy instead of renaming the non-lazy
Olivier Chafik
committed
Format test-chat.cpp
Olivier Chafik
committed
log prompt + nits
Olivier Chafik
committed
test: leave model_hf_file blank
ngxson
committed
force printing </tool_call> on hermes 2 model if/as it's a special token
Olivier Chafik
committed
try and avoid weird server test failure (spillage / parallelism between completion & tool call tests?)
Olivier Chafik
committed
Disable chat_completion tests of non-tool jinja mode
Olivier Chafik
committed
Fix typo
Olivier Chafik
committed