fix: merge system messages to comply with Qwen vLLM chat template - #294
Closed
OlegSob-glitch wants to merge 1 commit into
Closed
fix: merge system messages to comply with Qwen vLLM chat template#294OlegSob-glitch wants to merge 1 commit into
OlegSob-glitch wants to merge 1 commit into
Conversation
Summary: - Fixes 'System message must be at the beginning' error when using Qwen models via vLLM - Qwen's chat template requires exactly ONE system message at position 0 of messages array Changes: 1. deeptutor/agents/chat/agentic_pipeline.py: Merge system_prompt and memory_context into single system message, filter out system messages from conversation_history 2. deeptutor/services/session/context_builder.py: Filter out system messages from DB since summary is already added as system 3. deeptutor/agents/chat/chat_agent.py: Merge system prompt and RAG context into single system message Benefits: - Fixes Qwen vLLM compatibility issue completely - Reduces token usage by eliminating duplicate system messages - Does not break any functionality - system context from DB is already in compressed_summary - Cleaner message structure with exactly one system message at position 0 Testing: - Verified with Qwen model via vLLM - no more template errors - Chat with memory files (PROFILE.md/SUMMARY.md) works correctly - Chat with conversation history works correctly - No regression with other models (GPT-4, etc.) Fixes #1
20 tasks
Contributor
Author
|
these edits are taken into account in #295 |
vaskoyudha
added a commit
to vaskoyudha/deeptutor-for-programmer-fork
that referenced
this pull request
Jul 25, 2026
This PR fixes two related issues: 1. Qwen vLLM 'System message must be at the beginning' error: - deeptutor/agents/chat/agentic_pipeline.py: Merge system_prompt and memory_context into single system message, filter out system messages from conversation_history - deeptutor/services/session/context_builder.py: Filter out system messages from DB since summary is already added as system - deeptutor/agents/chat/chat_agent.py: Merge system prompt and RAG context into single system message 2. History reference (@reference) empty context issue: - deeptutor/services/session/turn_runtime.py: Add fallback to raw record content when NotebookAnalysisAgent.analyze() returns empty string Benefits: - Fixes Qwen vLLM compatibility issue completely - Enables history reference feature to work correctly - Reduces token usage by eliminating duplicate system messages - Does not break any functionality - system context from DB is already in compressed_summary - Cleaner message structure with exactly one system message at position 0 Testing: - Verified with Qwen model via vLLM - no more template errors - Chat with memory files (PROFILE.md/SUMMARY.md) works correctly - Chat with conversation history works correctly - History reference (@reference) now works correctly - Conversation continuation works without errors - No regression with other models (GPT-4, etc.) Closes HKUDS#294
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Changes:
Benefits:
Testing:
Description
A clear and concise description of the changes.
Related Issues
Module(s) Affected
agentsapiconfigcoreknowledgeloggingservicestoolsutilsweb(Frontend)docs(Documentation)scriptstests...Checklist
pre-commit run --all-filesand fixed any issues.Additional Notes
Summary
When using Qwen models via vLLM, the chat template throws a jinja2.exceptions.TemplateError because multiple system messages are being sent to the LLM. Qwen's chat template requires exactly one system message at position 0 of the messages array.
Error Message
jinja2.exceptions.TemplateError: System message must be the beginning.
ValueError: System message must be at the beginning.
Full traceback shows the error originates from:
vllm/renderers/hf.py:502 - safe_apply_chat_template()
transformers/tokenization_utils_base.py:3063 - apply_chat_template()
Qwen's Jinja2 chat template at line 85
Reproduction Steps
Start DeepTutor with Qwen model via vLLM
Open a new chat session
Send any message (e.g., "привет" / "hello")
Observe the error in vLLM logs
Note: This happens even in a brand new chat with no conversation history, if memory files (PROFILE.md/SUMMARY.md) exist.
Root Cause Analysis
Message Construction Flow
The issue occurs in the message construction pipeline where multiple system messages are added:
┌─────────────────────────────────────────────────────────┐
│ 1. _build_messages() in agentic_pipeline.py │
│ ├─ System prompt → {"role": "system", ...} │
│ ├─ Memory context → {"role": "system", ...} ← BUG │
│ └─ Conversation history (may include system) ← BUG │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Messages sent to vLLM: │
│ [ │
│ {"role": "system", "content": "thinking prompt"}, │
│ {"role": "system", "content": "## Background Memory"},│
│ {"role": "user", "content": "привет"} │
│ ] │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Qwen template error: System must be at position 0 only │
└─────────────────────────────────────────────────────────┘