Skip to content

Commit 0a8026e

Browse files
aldehirpwilkin
aldehir
and
pwilkin
authored
common : introduce composable PEG parser combinators for chat parsing (#17136)
* common : implement parser combinators to simplify chat parsing * add virtual destructor to parser_base * fix memory leak from circular references of rules * implement gbnf grammar building * remove unused private variable * create a base visitor and implement id assignment as a visitor * fix const ref for grammar builder * clean up types, friend classes, and class declarations * remove builder usage from until_parser * Use a counter class to help assign rule ids * cache everything * add short description for each parser * create a type for the root parser * implement repetition parser * Make optional, one_or_more, and zero_or_more subclasses of repetition * improve context constructor * improve until parsing and add benchmarks * remove cached() pattern, cache in parser_base with specialized parsing functions for each parser * improve json parsing performance to better match legacy parsing * fix const auto * it for windows * move id assignment to classes instead of using a visitor * create named rules in the command r7b example * use '.' for any in GBNF * fix parens around choices in gbnf grammar * add convenience operators to turn strings to literals * add free-form operators for const char * to simplify defining literals * simplify test case parser * implement semantic actions * remove groups in favor of actions and a scratchpad * add built in actions for common operations * add actions to command r7b example * use std::default_searcher for platforms that don't have bm * improve parser_type handling and add cast helper * add partial result type to better control when to run actions * fix bug in until() * run actions on partial results by default * use common_chat_msg for result * add qwen3 example wip * trash partial idea and simplify * move action arguments to a struct * implement aho-corasick matcher for until_parser and to build exclusion grammars * use std::string for input, since std::string_view is incompatible with std::regex * Refactor tests * improve qwen3 example * implement sax-style parsing and refactor * fix json string in test * rename classes to use common_chat_ prefix * remove is_ suffix from functions * rename from id_counter to just counter * Final refactored tests * Fix executable name and editorconfig-checker * Third time's the charm... * add trigger parser to begin lazy grammar rule generation * working lazy grammar * refactor json rules now that we check for reachability * reduce pointer usage * print out grammars in example * rename to chat-peg-parser* and common_chat_peg_parser* * Revert unrelated changes * New macros for CMakeLists to enable multi-file compilations * starting unicode support * add unicode support to char_parser * use unparsed args as additional sources * Refactor tests to new harness * Fix CMakeLists * fix rate calculation * add unicode tests * fix trailing whitespace and line endings skip-checks: true * Helpers + rewrite qwen3 with helpers * Fix whitespace * extract unicode functions to separate file * refactor parse unicode function * fix compiler error * improve construction of sequence/choice parsers * be less clever * add make_parser helper function * expand usage of make_parser, alias common_chat_msg_peg_parser_builder to builder in source * lower bench iterations * add unicode support to until_parser * add unicode support to json_string_parser * clean up unicode tests * reduce unicode details to match src/unicode.cpp * simplify even further * remove unused functions * fix type * reformat char class parsing * clean up json string parser * clean up + fix diagnostics * reorder includes * compact builder functions * replace action_parser with capture_parser, rename env to semantics * rename env to semantics * clean up common_chat_parse_context * move type() to below constant * use default constructor for common_chat_peg_parser * make all operators functions for consistency * fix compilation errors in test-optional.cpp * simplify result values * rename json_string_unquoted to json_string_content * Move helper to separate class, add separate explicit and helper classes * Whitespace * Change + to append() * Reformat * Add extra helpers, tests and Minimax example * Add some extra optional debugging prints + real example of how to use them * fix bug in repetitions when min_count = 0 reports failures * dump rule in debug * fix token accumulation and assert parsing never fails * indent debug by depth * use LOG_* in tests so logs sync up with test logs * - Add selective testing - Refactor all messaging to use LOG_ERR - Fix lack of argument / tool name capturing - Temporary fix for double event capture * refactor rule() and introduce ref() * clean up visitor * clean up indirection in root parser w.r.t rules * store shared ptr directly in parser classes * replace aho-corasick automation with a simple trie * Reset prev for qwen3 helper example variant * refactor to use value semantics with std::variant/std::visit * simplify trie_matcher result * fix linting issues * add annotations to rules * revert test workaround * implement serializing the parser * remove redundant parsers * remove tests * gbnf generation fixes * remove LOG_* use in tests * update gbnf tests to test entire grammar * clean up gbnf generation and fix a few bugs * fix typo in test output * remove implicit conversion rules * improve test output * rename trie_matcher to trie * simplify trie to just know if a node is the end of a word * remove common_chat_ prefix and ensure a common_peg_ prefix to all types * rename chat-peg-parser -> peg-parser * promote chat-peg-parser-helper to chat-peg-parser * checkpoint * use a static_assert to ensure we handle every branch * inline trivial peg parser builders * use json strings for now * implement basic and native chat peg parser builders/extractors * resolve refs to their rules * remove packrat caching (for now) * update tests * compare parsers with incremental input * benchmark both complete and incremental parsing * add raw string generation from json schema * add support for string schemas in gbnf generation * fix qwen example to include \n * tidy up example * rename extractor to mapper * rename ast_arena to ast * place basic tests into one * use gbnf_format_literal from json-schema-to-grammar * integrate parser with common/chat and server * clean up schema and serialization * add json-schema raw string tests * clean up json creation and remove capture parser * trim spaces from reasoning and content * clean up redundant rules and comments * rename input_is_complete to is_partial to match rest of project * simplify json rules * remove extraneous file * remove comment * implement += and |= operators * add comments to qwen3 implementation * reorder arguments to common_chat_peg_parse * remove commented outdated tests * add explicit copy constructor * fix operators and constness * wip: update test-chat for qwen3-coder * bring json parser closer to json-schema-to-grammar rules * trim trailing space for most things * fix qwen3 coder rules w.r.t. trailing spaces * group rules * do not trim trailing space from string args * tweak spacing of qwen3 grammar * update qwen3-coder tests * qwen3-coder small fixes * place parser in common_chat_syntax to simplify invocation * use std::set to collect rules to keep order predictable for tests * initialize parser to make certain platforms happy * revert back to std::unordered_set, sort rule names at the end instead * uncomment rest of chat tests * define explicit default constructor * improve arena init and server integration * fix chat test * add json_member() * add a comprehensive native example * clean up example qwen test and add response_format example to native test * make build_peg_parser accept std::function instead of template * change peg parser parameters into const ref * push tool call on tool open for constructed parser * add parsing documentation * clean up some comments * add json schema support to qwen3-coder * add id initializer in tests * remove grammar debug line from qwen3-coder * refactor qwen3-coder to use sequence over operators * only call common_chat_peg_parse if appropriate format * simplify qwen3-coder space handling * revert qwen3-coder implementation * revert json-schema-to-grammar changes * remove unnecessary forward declaration * small adjustment to until_parser * rename C/C++ files to use dashes * codeowners : add aldehir to peg-parser and related files --------- Co-authored-by: Piotr Wilkin <[email protected]>
1 parent 5ceed62 commit 0a8026e

27 files changed

+5243
-2
lines changed

‎CODEOWNERS‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,16 @@
1010
/common/arg.* @ggerganov
1111
/common/base64.hpp.* @ggerganov
1212
/common/build-info.* @ggerganov
13+
/common/chat-peg-parser.* @aldehir
1314
/common/common.* @ggerganov
1415
/common/console.* @ggerganov
1516
/common/http.* @angt
1617
/common/llguidance.* @ggerganov
1718
/common/log.* @ggerganov
19+
/common/peg-parser.* @aldehir
1820
/common/sampling.* @ggerganov
1921
/common/speculative.* @ggerganov
22+
/common/unicode.* @aldehir
2023
/convert_*.py @CISC
2124
/examples/batched.swift/ @ggerganov
2225
/examples/batched/ @ggerganov

‎common/CMakeLists.txt‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,8 @@ add_library(${TARGET} STATIC
5252
chat-parser.h
5353
chat-parser-xml-toolcall.h
5454
chat-parser-xml-toolcall.cpp
55+
chat-peg-parser.cpp
56+
chat-peg-parser.h
5557
chat.cpp
5658
chat.h
5759
common.cpp
@@ -69,12 +71,16 @@ add_library(${TARGET} STATIC
6971
log.h
7072
ngram-cache.cpp
7173
ngram-cache.h
74+
peg-parser.cpp
75+
peg-parser.h
7276
regex-partial.cpp
7377
regex-partial.h
7478
sampling.cpp
7579
sampling.h
7680
speculative.cpp
7781
speculative.h
82+
unicode.cpp
83+
unicode.h
7884
)
7985

8086
if (BUILD_SHARED_LIBS)

‎common/chat-parser.cpp‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
#include "chat-parser.h"
2+
#include "chat-peg-parser.h"
23
#include "common.h"
34
#include "log.h"
5+
#include "peg-parser.h"
46
#include "regex-partial.h"
57

68
#include <algorithm>
@@ -1483,6 +1485,11 @@ static void common_chat_parse(common_chat_msg_parser & builder) {
14831485
}
14841486

14851487
common_chat_msg common_chat_parse(const std::string & input, bool is_partial, const common_chat_syntax & syntax) {
1488+
if (syntax.format == COMMON_CHAT_FORMAT_PEG_SIMPLE ||
1489+
syntax.format == COMMON_CHAT_FORMAT_PEG_NATIVE ||
1490+
syntax.format == COMMON_CHAT_FORMAT_PEG_CONSTRUCTED) {
1491+
return common_chat_peg_parse(syntax.parser, input, is_partial, syntax);
1492+
}
14861493
common_chat_msg_parser builder(input, is_partial, syntax);
14871494
try {
14881495
common_chat_parse(builder);
@@ -1500,3 +1507,36 @@ common_chat_msg common_chat_parse(const std::string & input, bool is_partial, co
15001507
}
15011508
return msg;
15021509
}
1510+
1511+
common_chat_msg common_chat_peg_parse(const common_peg_arena & parser, const std::string & input, bool is_partial, const common_chat_syntax & syntax) {
1512+
if (parser.empty()) {
1513+
throw std::runtime_error("Failed to parse due to missing parser definition.");
1514+
}
1515+
1516+
LOG_DBG("Parsing input with format %s: %s\n", common_chat_format_name(syntax.format), input.c_str());
1517+
1518+
common_peg_parse_context ctx(input, is_partial);
1519+
auto result = parser.parse(ctx);
1520+
if (result.fail()) {
1521+
throw std::runtime_error(std::string("Failed to parse input at pos ") + std::to_string(result.end));
1522+
}
1523+
1524+
common_chat_msg msg;
1525+
msg.role = "assistant";
1526+
1527+
if (syntax.format == COMMON_CHAT_FORMAT_PEG_NATIVE) {
1528+
auto mapper = common_chat_peg_native_mapper(msg);
1529+
mapper.from_ast(ctx.ast, result);
1530+
} else if (syntax.format == COMMON_CHAT_FORMAT_PEG_CONSTRUCTED) {
1531+
auto mapper = common_chat_peg_constructed_mapper(msg);
1532+
mapper.from_ast(ctx.ast, result);
1533+
} else {
1534+
// Generic mapper
1535+
auto mapper = common_chat_peg_mapper(msg);
1536+
mapper.from_ast(ctx.ast, result);
1537+
}
1538+
if (!is_partial) {
1539+
LOG_DBG("Parsed message: %s\n", common_chat_msgs_to_json_oaicompat<json>({msg}).at(0).dump().c_str());
1540+
}
1541+
return msg;
1542+
}

‎common/chat-peg-parser.cpp‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
#include "chat-peg-parser.h"
2+
3+
#include <nlohmann/json.hpp>
4+
5+
using json = nlohmann::json;
6+
7+
static std::string_view trim_trailing_space(std::string_view sv) {
8+
while (!sv.empty() && std::isspace(static_cast<unsigned char>(sv.back()))) {
9+
sv.remove_suffix(1);
10+
}
11+
return sv;
12+
}
13+
14+
void common_chat_peg_mapper::from_ast(const common_peg_ast_arena & arena, const common_peg_parse_result & result) {
15+
arena.visit(result, [this](const common_peg_ast_node & node) {
16+
map(node);
17+
});
18+
}
19+
20+
void common_chat_peg_mapper::map(const common_peg_ast_node & node) {
21+
bool is_reasoning = node.tag == common_chat_peg_builder::REASONING;
22+
bool is_content = node.tag == common_chat_peg_builder::CONTENT;
23+
24+
if (is_reasoning) {
25+
result.reasoning_content = std::string(trim_trailing_space(node.text));
26+
}
27+
28+
if (is_content) {
29+
result.content = std::string(trim_trailing_space(node.text));
30+
}
31+
}
32+
33+
void common_chat_peg_native_mapper::map(const common_peg_ast_node & node) {
34+
common_chat_peg_mapper::map(node);
35+
36+
bool is_tool_open = node.tag == common_chat_peg_native_builder::TOOL_OPEN;
37+
bool is_tool_name = node.tag == common_chat_peg_native_builder::TOOL_NAME;
38+
bool is_tool_id = node.tag == common_chat_peg_native_builder::TOOL_ID;
39+
bool is_tool_args = node.tag == common_chat_peg_native_builder::TOOL_ARGS;
40+
41+
if (is_tool_open) {
42+
result.tool_calls.emplace_back();
43+
current_tool = &result.tool_calls.back();
44+
}
45+
46+
if (is_tool_id && current_tool) {
47+
current_tool->id = std::string(trim_trailing_space(node.text));
48+
}
49+
50+
if (is_tool_name && current_tool) {
51+
current_tool->name = std::string(trim_trailing_space(node.text));
52+
}
53+
54+
if (is_tool_args && current_tool) {
55+
current_tool->arguments = std::string(trim_trailing_space(node.text));
56+
}
57+
}
58+
59+
void common_chat_peg_constructed_mapper::map(const common_peg_ast_node & node) {
60+
common_chat_peg_mapper::map(node);
61+
62+
bool is_tool_open = node.tag == common_chat_peg_constructed_builder::TOOL_OPEN;
63+
bool is_tool_name = node.tag == common_chat_peg_constructed_builder::TOOL_NAME;
64+
bool is_tool_close = node.tag == common_chat_peg_constructed_builder::TOOL_CLOSE;
65+
bool is_arg_open = node.tag == common_chat_peg_constructed_builder::TOOL_ARG_OPEN;
66+
bool is_arg_close = node.tag == common_chat_peg_constructed_builder::TOOL_ARG_CLOSE;
67+
bool is_arg_name = node.tag == common_chat_peg_constructed_builder::TOOL_ARG_NAME;
68+
bool is_arg_string = node.tag == common_chat_peg_constructed_builder::TOOL_ARG_STRING_VALUE;
69+
bool is_arg_json = node.tag == common_chat_peg_constructed_builder::TOOL_ARG_JSON_VALUE;
70+
71+
if (is_tool_open) {
72+
result.tool_calls.emplace_back();
73+
current_tool = &result.tool_calls.back();
74+
arg_count = 0;
75+
}
76+
77+
if (is_tool_name) {
78+
current_tool->name = std::string(node.text);
79+
current_tool->arguments = "{";
80+
}
81+
82+
if (is_arg_open) {
83+
needs_closing_quote = false;
84+
}
85+
86+
if (is_arg_name && current_tool) {
87+
if (arg_count > 0) {
88+
current_tool->arguments += ",";
89+
}
90+
current_tool->arguments += json(trim_trailing_space(node.text)).dump() + ":";
91+
++arg_count;
92+
}
93+
94+
if (is_arg_string && current_tool) {
95+
// Serialize to JSON, but exclude the end quote
96+
std::string dumped = json(node.text).dump();
97+
current_tool->arguments += dumped.substr(0, dumped.size() - 1);
98+
needs_closing_quote = true;
99+
}
100+
101+
if (is_arg_close && current_tool) {
102+
if (needs_closing_quote) {
103+
current_tool->arguments += "\"";
104+
}
105+
}
106+
107+
if (is_arg_json && current_tool) {
108+
current_tool->arguments += std::string(trim_trailing_space(node.text));
109+
}
110+
111+
if (is_tool_close && current_tool) {
112+
current_tool->arguments += "}";
113+
}
114+
}

‎common/chat-peg-parser.h‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#pragma once
2+
3+
#include "chat.h"
4+
#include "peg-parser.h"
5+
6+
class common_chat_peg_builder : public common_peg_parser_builder {
7+
public:
8+
static constexpr const char * REASONING_BLOCK = "reasoning-block";
9+
static constexpr const char * REASONING = "reasoning";
10+
static constexpr const char * CONTENT = "content";
11+
12+
common_peg_parser reasoning_block(const common_peg_parser & p) { return tag(REASONING_BLOCK, p); }
13+
common_peg_parser reasoning(const common_peg_parser & p) { return tag(REASONING, p); }
14+
common_peg_parser content(const common_peg_parser & p) { return tag(CONTENT, p); }
15+
};
16+
17+
inline common_peg_arena build_chat_peg_parser(const std::function<common_peg_parser(common_chat_peg_builder & builder)> & fn) {
18+
common_chat_peg_builder builder;
19+
builder.set_root(fn(builder));
20+
return builder.build();
21+
}
22+
23+
class common_chat_peg_mapper {
24+
public:
25+
common_chat_msg & result;
26+
27+
common_chat_peg_mapper(common_chat_msg & msg) : result(msg) {}
28+
29+
virtual void from_ast(const common_peg_ast_arena & arena, const common_peg_parse_result & result);
30+
virtual void map(const common_peg_ast_node & node);
31+
};
32+
33+
class common_chat_peg_native_builder : public common_chat_peg_builder {
34+
public:
35+
static constexpr const char * TOOL = "tool";
36+
static constexpr const char * TOOL_OPEN = "tool-open";
37+
static constexpr const char * TOOL_CLOSE = "tool-close";
38+
static constexpr const char * TOOL_ID = "tool-id";
39+
static constexpr const char * TOOL_NAME = "tool-name";
40+
static constexpr const char * TOOL_ARGS = "tool-args";
41+
42+
common_peg_parser tool(const common_peg_parser & p) { return tag(TOOL, p); }
43+
common_peg_parser tool_open(const common_peg_parser & p) { return atomic(tag(TOOL_OPEN, p)); }
44+
common_peg_parser tool_close(const common_peg_parser & p) { return atomic(tag(TOOL_CLOSE, p)); }
45+
common_peg_parser tool_id(const common_peg_parser & p) { return atomic(tag(TOOL_ID, p)); }
46+
common_peg_parser tool_name(const common_peg_parser & p) { return atomic(tag(TOOL_NAME, p)); }
47+
common_peg_parser tool_args(const common_peg_parser & p) { return tag(TOOL_ARGS, p); }
48+
};
49+
50+
class common_chat_peg_native_mapper : public common_chat_peg_mapper {
51+
common_chat_tool_call * current_tool;
52+
53+
public:
54+
common_chat_peg_native_mapper(common_chat_msg & msg) : common_chat_peg_mapper(msg) {}
55+
56+
void map(const common_peg_ast_node & node) override;
57+
};
58+
59+
inline common_peg_arena build_chat_peg_native_parser(const std::function<common_peg_parser(common_chat_peg_native_builder & builder)> & fn) {
60+
common_chat_peg_native_builder builder;
61+
builder.set_root(fn(builder));
62+
return builder.build();
63+
}
64+
65+
class common_chat_peg_constructed_builder : public common_chat_peg_builder {
66+
public:
67+
static constexpr const char * TOOL = "tool";
68+
static constexpr const char * TOOL_OPEN = "tool-open";
69+
static constexpr const char * TOOL_CLOSE = "tool-close";
70+
static constexpr const char * TOOL_NAME = "tool-name";
71+
static constexpr const char * TOOL_ARG = "tool-arg";
72+
static constexpr const char * TOOL_ARG_OPEN = "tool-arg-open";
73+
static constexpr const char * TOOL_ARG_CLOSE = "tool-arg-close";
74+
static constexpr const char * TOOL_ARG_NAME = "tool-arg-name";
75+
static constexpr const char * TOOL_ARG_STRING_VALUE = "tool-arg-string-value";
76+
static constexpr const char * TOOL_ARG_JSON_VALUE = "tool-arg-json-value";
77+
78+
common_peg_parser tool(const common_peg_parser & p) { return tag(TOOL, p); }
79+
common_peg_parser tool_open(const common_peg_parser & p) { return atomic(tag(TOOL_OPEN, p)); }
80+
common_peg_parser tool_close(const common_peg_parser & p) { return atomic(tag(TOOL_CLOSE, p)); }
81+
common_peg_parser tool_name(const common_peg_parser & p) { return atomic(tag(TOOL_NAME, p)); }
82+
common_peg_parser tool_arg(const common_peg_parser & p) { return tag(TOOL_ARG, p); }
83+
common_peg_parser tool_arg_open(const common_peg_parser & p) { return atomic(tag(TOOL_ARG_OPEN, p)); }
84+
common_peg_parser tool_arg_close(const common_peg_parser & p) { return atomic(tag(TOOL_ARG_CLOSE, p)); }
85+
common_peg_parser tool_arg_name(const common_peg_parser & p) { return atomic(tag(TOOL_ARG_NAME, p)); }
86+
common_peg_parser tool_arg_string_value(const common_peg_parser & p) { return tag(TOOL_ARG_STRING_VALUE, p); }
87+
common_peg_parser tool_arg_json_value(const common_peg_parser & p) { return tag(TOOL_ARG_JSON_VALUE, p); }
88+
};
89+
90+
class common_chat_peg_constructed_mapper : public common_chat_peg_mapper {
91+
common_chat_tool_call * current_tool;
92+
int arg_count = 0;
93+
bool needs_closing_quote = false;
94+
95+
public:
96+
common_chat_peg_constructed_mapper(common_chat_msg & msg) : common_chat_peg_mapper(msg) {}
97+
98+
void map(const common_peg_ast_node & node) override;
99+
};
100+
101+
inline common_peg_arena build_chat_peg_constructed_parser(const std::function<common_peg_parser(common_chat_peg_constructed_builder & builder)> & fn) {
102+
common_chat_peg_constructed_builder builder;
103+
builder.set_root(fn(builder));
104+
return builder.build();
105+
}

‎common/chat.cpp‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -649,6 +649,9 @@ const char * common_chat_format_name(common_chat_format format) {
649649
case COMMON_CHAT_FORMAT_QWEN3_CODER_XML: return "Qwen3 Coder";
650650
case COMMON_CHAT_FORMAT_APRIEL_1_5: return "Apriel 1.5";
651651
case COMMON_CHAT_FORMAT_XIAOMI_MIMO: return "Xiaomi MiMo";
652+
case COMMON_CHAT_FORMAT_PEG_SIMPLE: return "peg-simple";
653+
case COMMON_CHAT_FORMAT_PEG_NATIVE: return "peg-native";
654+
case COMMON_CHAT_FORMAT_PEG_CONSTRUCTED: return "peg-constructed";
652655
default:
653656
throw std::runtime_error("Unknown chat format");
654657
}

‎common/chat.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
#pragma once
44

55
#include "common.h"
6+
#include "peg-parser.h"
67
#include <functional>
78
#include <chrono>
89
#include <string>
@@ -124,6 +125,11 @@ enum common_chat_format {
124125
COMMON_CHAT_FORMAT_APRIEL_1_5,
125126
COMMON_CHAT_FORMAT_XIAOMI_MIMO,
126127

128+
// These are intended to be parsed by the PEG parser
129+
COMMON_CHAT_FORMAT_PEG_SIMPLE,
130+
COMMON_CHAT_FORMAT_PEG_NATIVE,
131+
COMMON_CHAT_FORMAT_PEG_CONSTRUCTED,
132+
127133
COMMON_CHAT_FORMAT_COUNT, // Not a format, just the # formats
128134
};
129135

@@ -154,6 +160,7 @@ struct common_chat_params {
154160
std::vector<common_grammar_trigger> grammar_triggers;
155161
std::vector<std::string> preserved_tokens;
156162
std::vector<std::string> additional_stops;
163+
std::string parser;
157164
};
158165

159166
struct common_chat_syntax {
@@ -163,6 +170,7 @@ struct common_chat_syntax {
163170
bool reasoning_in_content = false;
164171
bool thinking_forced_open = false;
165172
bool parse_tool_calls = true;
173+
common_peg_arena parser = {};
166174
};
167175

168176
// Check if the template supplied via "--chat-template" is supported or not. Returns true if it's valid
@@ -206,6 +214,7 @@ const char* common_chat_format_name(common_chat_format format);
206214
const char* common_reasoning_format_name(common_reasoning_format format);
207215
common_reasoning_format common_reasoning_format_from_name(const std::string & format);
208216
common_chat_msg common_chat_parse(const std::string & input, bool is_partial, const common_chat_syntax & syntax);
217+
common_chat_msg common_chat_peg_parse(const common_peg_arena & parser, const std::string & input, bool is_partial, const common_chat_syntax & syntax);
209218

210219
common_chat_tool_choice common_chat_tool_choice_parse_oaicompat(const std::string & tool_choice);
211220

0 commit comments

Comments
 (0)