Skip to content

Add support for Qwen3-4B-Instruct and add model-template mappings #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mohammed-Faizzzz wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add support for Qwen3-4B-Instruct and add model-template mappings #85

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
795c156
Add support for Qwen3-4B-Instruct and add model-template mappings
Mohammed-Faizzzz Sep 11, 2025
33e1251
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 11, 2025
b3f4492
Add Support for Qwen3
Mohammed-Faizzzz Sep 12, 2025
c445ff1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 12, 2025
bc26dea
modify dependency to support Qwen3
Mohammed-Faizzzz Sep 16, 2025
45bfb7b
Update repo names and update dependancies in requirements.txt
Mohammed-Faizzzz Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions src/core/constant.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
"Qwen/Qwen2.5-32B-Instruct",
"Qwen/Qwen2.5-72B",
"Qwen/Qwen2.5-72B-Instruct",
"Qwen/Qwen3-4B-Instruct",
# yi 1.5
"01-ai/Yi-1.5-6B",
"01-ai/Yi-1.5-6B-Chat",
Expand Down Expand Up @@ -50,3 +51,59 @@
"microsoft/Phi-4-mini-instruct",
"microsoft/phi-4",
]

MODEL_TEMPLATE_MAP = {
# Qwen
"Qwen/Qwen2.5-0.5B": "qwen1.5",
"Qwen/Qwen2.5-0.5B-Instruct": "qwen1.5",
"Qwen/Qwen2.5-1.5B": "qwen1.5",
"Qwen/Qwen2.5-1.5B-Instruct": "qwen1.5",
"Qwen/Qwen2.5-3B": "qwen1.5",
"Qwen/Qwen2.5-3B-Instruct": "qwen1.5",
"Qwen/Qwen2.5-7B": "qwen1.5",
"Qwen/Qwen2.5-7B-Instruct": "qwen1.5",
"Qwen/Qwen2.5-14B": "qwen1.5",
"Qwen/Qwen2.5-14B-Instruct": "qwen1.5",
"Qwen/Qwen2.5-32B": "qwen1.5",
"Qwen/Qwen2.5-32B-Instruct": "qwen1.5",
"Qwen/Qwen2.5-72B": "qwen1.5",
"Qwen/Qwen2.5-72B-Instruct": "qwen1.5",
"Qwen/Qwen3-4B-Instruct": "qwen3",
# Yi
"01-ai/Yi-1.5-6B": "yi",
"01-ai/Yi-1.5-6B-Chat": "yi",
"01-ai/Yi-1.5-9B": "yi",
"01-ai/Yi-1.5-9B-Chat": "yi",
"01-ai/Yi-1.5-34B": "yi",
"01-ai/Yi-1.5-34B-Chat": "yi",
# Mistral
"mistralai/Mistral-7B-v0.3": "mistral",
"mistralai/Mistral-7B-Instruct-v0.3": "mistral",
"mistralai/Ministral-8B-Instruct-2410": "mistral",
# Mixtral
"mistralai/Mixtral-8x7B-v0.1": "mixtral",
"mistralai/Mixtral-8x7B-Instruct-v0.1": "mixtral",
# Gemma 2
"google/gemma-2-2b": "gemma",
"google/gemma-2-9b": "gemma",
"google/gemma-2-27b": "gemma",
"google/gemma-2-2b-it": "gemma",
"google/gemma-2-9b-it": "gemma",
"google/gemma-2-27b-it": "gemma",
# LLaMA 3 + 3.1
"meta-llama/Meta-Llama-3-8B": "llama3",
"meta-llama/Meta-Llama-3-8B-Instruct": "llama3",
"meta-llama/Meta-Llama-3-70B": "llama3",
"meta-llama/Meta-Llama-3-70B-Instruct": "llama3",
"meta-llama/Meta-Llama-3.1-8B": "llama3",
"meta-llama/Meta-Llama-3.1-8B-Instruct": "llama3",
"meta-llama/Meta-Llama-3.1-70B": "llama3",
"meta-llama/Meta-Llama-3.1-70B-Instruct": "llama3",
# Phi 3
"microsoft/Phi-3.5-mini-instruct": "phi3",
"microsoft/Phi-3-mini-4k-instruct": "phi3",
"microsoft/Phi-3-medium-4k-instruct": "phi3",
# Phi 4
"microsoft/Phi-4-mini-instruct": "phi4",
"microsoft/phi-4": "phi4",
}
23 changes: 23 additions & 0 deletions src/core/template.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from dataclasses import dataclass
from typing import Dict
from constant import MODEL_TEMPLATE_MAP
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Use package-relative import to avoid import errors with src/ layout.

Direct from constant import ... will fail when imported as a package. Prefer a relative import (optionally keep a fallback for ad-hoc scripts).

-from constant import MODEL_TEMPLATE_MAP
+try:
+    from .constant import MODEL_TEMPLATE_MAP  # package-relative
+except ImportError:  # pragma: no cover
+    from constant import MODEL_TEMPLATE_MAP   # fallback for direct execution
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
from constant import MODEL_TEMPLATE_MAP
try:
from .constant import MODEL_TEMPLATE_MAP # package-relative
except ImportError: # pragma: no cover
from constant import MODEL_TEMPLATE_MAP # fallback for direct execution
🤖 Prompt for AI Agents 
In src/core/template.py around line 3, the direct top-level import "from
constant import MODEL_TEMPLATE_MAP" will break when the package is imported;
change to a package-relative import and optionally provide a fallback for ad-hoc
execution: use "from .constant import MODEL_TEMPLATE_MAP" as the primary import
and wrap it in a try/except that falls back to "from constant import
MODEL_TEMPLATE_MAP" if the relative import fails, so both package usage and
standalone/script runs work.



@dataclass
Expand Down Expand Up @@ -67,6 +68,25 @@ def register_template(
stop_word="<|im_end|>",
)

register_template(
template_name="qwen3",
system_format="<|im_start|>system\n{content}<|im_end|>\n",
user_format="<|im_start|>user\n{content}<|im_end|>\n<|im_start|>assistant\n",
assistant_format="{content}<|im_end|>\n",
tool_format = (
"# Tools\n\n"
"You may call one or more functions to assist with the user query.\n\n"
"You are provided with function signatures within <tools></tools> XML tags:\n"
"<tools>\n{content}\n</tools>\n\n"
"For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n"
"<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>"
),
function_format="<tool_call>\n{content}\n</tool_call><|im_end|>\n",
observation_format="<|im_start|>user\n<tool_response>\n{content}\n</tool_response><|im_end|>\n<|im_start|>assistant\n",
system="You are a helpful assistant.",
stop_word="<|im_end|>",
)
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Fix unescaped braces in qwen3 tool_format; .format() will crash.

The JSON example in tool_format includes bare {} which collide with str.format. Escape them.

-    tool_format = (
+    tool_format = (
         "# Tools\n\n"
         "You may call one or more functions to assist with the user query.\n\n"
         "You are provided with function signatures within <tools></tools> XML tags:\n"
         "<tools>\n{content}\n</tools>\n\n"
         "For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n"
-        "<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>"
+        "<tool_call>\n{{\"name\": <function-name>, \"arguments\": <args-json-object>}}\n</tool_call>"
     ),

If you want, I can add a quick unit test that formats every template field with content="x" to catch such issues proactively.

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "---- show src/core/template.py (1-240) ----"
nl -ba src/core/template.py | sed -n '1,240p' || true

echo
echo "---- rg register_template calls ----"
rg -n "register_template\(" || true

echo
echo "---- rg def register_template ----"
rg -n "def register_template" || true

echo
echo "---- rg .format( occurrences (2 lines context) ----"
rg -n --hidden --no-ignore-vcs "\.format\(" -S -g '!**/node_modules/**' -C2 || true

echo
echo "---- rg {content} occurrences (2 lines context) ----"
rg -n --hidden --no-ignore-vcs "\{content\}" -S -C2 -g '!**/node_modules/**' || true

echo
echo "---- rg tool_format occurrences ----"
rg -n "tool_format" -S || true

Length of output: 2408

Escape braces in qwen3 tool_format — fix str.format() crash

Unescaped JSON braces in the qwen3 tool_format will be parsed by str.format() and raise a ValueError at runtime (see src/core/dataset.py:49 where .format(...) is called). Fix by doubling the literal braces in the JSON example.

File: src/core/template.py (register_template qwen3, ~lines 71-88)

-    tool_format = (
+    tool_format = (
         "# Tools\n\n"
         "You may call one or more functions to assist with the user query.\n\n"
         "You are provided with function signatures within <tools></tools> XML tags:\n"
         "<tools>\n{content}\n</tools>\n\n"
         "For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n"
-        "<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>"
+        "<tool_call>\n{{\"name\": <function-name>, \"arguments\": <args-json-object>}}\n</tool_call>"
     ),
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
register_template(
template_name="qwen3",
system_format="<|im_start|>system\n{content}<|im_end|>\n",
user_format="<|im_start|>user\n{content}<|im_end|>\n<|im_start|>assistant\n",
assistant_format="{content}<|im_end|>\n",
tool_format = (
"# Tools\n\n"
"You may call one or more functions to assist with the user query.\n\n"
"You are provided with function signatures within <tools></tools> XML tags:\n"
"<tools>\n{content}\n</tools>\n\n"
"For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n"
"<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>"
),
function_format="<tool_call>\n{content}\n</tool_call><|im_end|>\n",
observation_format="<|im_start|>user\n<tool_response>\n{content}\n</tool_response><|im_end|>\n<|im_start|>assistant\n",
system="You are a helpful assistant.",
stop_word="<|im_end|>",
)
register_template(
template_name="qwen3",
system_format="<|im_start|>system\n{content}<|im_end|>\n",
user_format="<|im_start|>user\n{content}<|im_end|>\n<|im_start|>assistant\n",
assistant_format="{content}<|im_end|>\n",
tool_format = (
"# Tools\n\n"
"You may call one or more functions to assist with the user query.\n\n"
"You are provided with function signatures within <tools></tools> XML tags:\n"
"<tools>\n{content}\n</tools>\n\n"
"For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n"
"<tool_call>\n{{\"name\": <function-name>, \"arguments\": <args-json-object>}}\n</tool_call>"
),
function_format="<tool_call>\n{content}\n</tool_call><|im_end|>\n",
observation_format="<|im_start|>user\n<tool_response>\n{content}\n</tool_response><|im_end|>\n<|im_start|>assistant\n",
system="You are a helpful assistant.",
stop_word="<|im_end|>",
)
🤖 Prompt for AI Agents 
In src/core/template.py around lines 71 to 88, the qwen3 template's tool_format
contains unescaped JSON braces which are consumed by str.format() and cause a
ValueError at runtime; fix by doubling all literal braces in the JSON example
inside tool_format (replace { with {{ and } with }} for the JSON snippet and any
other literal braces) so that str.format() treats them as literal characters
while leaving the format placeholders like {content} intact.


register_template(
template_name="yi",
system_format="<|im_start|>system\n{content}<|im_end|>\n",
Expand Down Expand Up @@ -182,3 +202,6 @@ def register_template(
system=None,
stop_word="<|end|>",
)

for model_name, template_name in MODEL_TEMPLATE_MAP.items():
template_dict[model_name] = template_dict[template_name]
Comment on lines +206 to +207
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Copy and validate when aliasing templates to avoid KeyError and shared-state bugs.

  • Guard against mappings to unknown template names.
  • Create a new Template instance per alias to avoid accidental cross-mutation.
-for model_name, template_name in MODEL_TEMPLATE_MAP.items():
-    template_dict[model_name] = template_dict[template_name]
+missing = []
+for model_name, template_name in MODEL_TEMPLATE_MAP.items():
+    base = template_dict.get(template_name)
+    if base is None:
+        missing.append((model_name, template_name))
+        continue
+    # copy to avoid shared object between aliases
+    template_dict[model_name] = replace(base)
+if missing:
+    raise KeyError(f"MODEL_TEMPLATE_MAP points to unknown template(s): {missing}")

Add import (outside the changed hunk) to support replace:

-from dataclasses import dataclass
+from dataclasses import dataclass, replace
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for model_name, template_name in MODEL_TEMPLATE_MAP.items():
template_dict[model_name] = template_dict[template_name]
# (update import at top of file)
from dataclasses import dataclass, replace
Suggested change
for model_name, template_name in MODEL_TEMPLATE_MAP.items():
template_dict[model_name] = template_dict[template_name]
missing = []
for model_name, template_name in MODEL_TEMPLATE_MAP.items():
base = template_dict.get(template_name)
if base is None:
missing.append((model_name, template_name))
continue
# copy to avoid shared object between aliases
template_dict[model_name] = replace(base)
if missing:
raise KeyError(f"MODEL_TEMPLATE_MAP points to unknown template(s): {missing}")
🤖 Prompt for AI Agents 
In src/core/template.py around lines 206-207, the aliasing loop blindly assigns
references causing KeyError for unknown targets and shared-state bugs; change it
to first check that template_name exists in template_dict and raise or log a
clear KeyError with context if not, then assign a fresh copy for the alias
(e.g., template_dict[model_name] = deepcopy(template_dict[template_name]] ) or
construct a new Template instance from the source so the alias does not share
mutable state; also add the needed import at the top (from copy import deepcopy)
to support the replacement.