Skip to content

fix: merge system messages to comply with Qwen vLLM chat template - #294

Closed
OlegSob-glitch wants to merge 1 commit into
HKUDS:devfrom
OlegSob-glitch:fix/qwen-vllm-system-message-from-dev
Closed

fix: merge system messages to comply with Qwen vLLM chat template#294
OlegSob-glitch wants to merge 1 commit into
HKUDS:devfrom
OlegSob-glitch:fix/qwen-vllm-system-message-from-dev

Conversation

@OlegSob-glitch

Copy link
Copy Markdown
Contributor

Summary:

  • Fixes 'System message must be at the beginning' error when using Qwen models via vLLM
  • Qwen's chat template requires exactly ONE system message at position 0 of messages array

Changes:

  1. deeptutor/agents/chat/agentic_pipeline.py: Merge system_prompt and memory_context into single system message, filter out system messages from conversation_history
  2. deeptutor/services/session/context_builder.py: Filter out system messages from DB since summary is already added as system
  3. deeptutor/agents/chat/chat_agent.py: Merge system prompt and RAG context into single system message

Benefits:

  • Fixes Qwen vLLM compatibility issue completely
  • Reduces token usage by eliminating duplicate system messages
  • Does not break any functionality - system context from DB is already in compressed_summary
  • Cleaner message structure with exactly one system message at position 0

Testing:

  • Verified with Qwen model via vLLM - no more template errors
  • Chat with memory files (PROFILE.md/SUMMARY.md) works correctly
  • Chat with conversation history works correctly
  • No regression with other models (GPT-4, etc.)

Description

A clear and concise description of the changes.

Related Issues

  • Closes #...
  • Related to #...

Module(s) Affected

  • agents
  • api
  • config
  • core
  • knowledge
  • logging
  • services
  • tools
  • utils
  • web (Frontend)
  • docs (Documentation)
  • scripts
  • tests
  • Other: ...

Checklist

  • [+] I have read and followed the contribution guidelines.
  • [+] My code follows the project's coding standards.
  • [-] I have run pre-commit run --all-files and fixed any issues.
  • [-] I have added relevant tests for my changes.
  • [-] I have updated the documentation (if necessary).
  • [+] My changes do not introduce any new security vulnerabilities.

Additional Notes

Summary
When using Qwen models via vLLM, the chat template throws a jinja2.exceptions.TemplateError because multiple system messages are being sent to the LLM. Qwen's chat template requires exactly one system message at position 0 of the messages array.

Error Message
jinja2.exceptions.TemplateError: System message must be the beginning.
ValueError: System message must be at the beginning.
Full traceback shows the error originates from:

vllm/renderers/hf.py:502 - safe_apply_chat_template()
transformers/tokenization_utils_base.py:3063 - apply_chat_template()
Qwen's Jinja2 chat template at line 85
Reproduction Steps

Start DeepTutor with Qwen model via vLLM
Open a new chat session
Send any message (e.g., "привет" / "hello")
Observe the error in vLLM logs
Note: This happens even in a brand new chat with no conversation history, if memory files (PROFILE.md/SUMMARY.md) exist.

Root Cause Analysis
Message Construction Flow
The issue occurs in the message construction pipeline where multiple system messages are added:

┌─────────────────────────────────────────────────────────┐
│ 1. _build_messages() in agentic_pipeline.py │
│ ├─ System prompt → {"role": "system", ...} │
│ ├─ Memory context → {"role": "system", ...} ← BUG │
│ └─ Conversation history (may include system) ← BUG │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│ Messages sent to vLLM: │
│ [ │
│ {"role": "system", "content": "thinking prompt"}, │
│ {"role": "system", "content": "## Background Memory"},│
│ {"role": "user", "content": "привет"} │
│ ] │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│ Qwen template error: System must be at position 0 only │
└─────────────────────────────────────────────────────────┘

Summary:
- Fixes 'System message must be at the beginning' error when using Qwen models via vLLM
- Qwen's chat template requires exactly ONE system message at position 0 of messages array

Changes:
1. deeptutor/agents/chat/agentic_pipeline.py: Merge system_prompt and memory_context into single system message, filter out system messages from conversation_history
2. deeptutor/services/session/context_builder.py: Filter out system messages from DB since summary is already added as system
3. deeptutor/agents/chat/chat_agent.py: Merge system prompt and RAG context into single system message

Benefits:
- Fixes Qwen vLLM compatibility issue completely
- Reduces token usage by eliminating duplicate system messages
- Does not break any functionality - system context from DB is already in compressed_summary
- Cleaner message structure with exactly one system message at position 0

Testing:
- Verified with Qwen model via vLLM - no more template errors
- Chat with memory files (PROFILE.md/SUMMARY.md) works correctly
- Chat with conversation history works correctly
- No regression with other models (GPT-4, etc.)

Fixes #1
@OlegSob-glitch

Copy link
Copy Markdown
Contributor Author

these edits are taken into account in #295

vaskoyudha added a commit to vaskoyudha/deeptutor-for-programmer-fork that referenced this pull request Jul 25, 2026
This PR fixes two related issues:

1. Qwen vLLM 'System message must be at the beginning' error:
   - deeptutor/agents/chat/agentic_pipeline.py: Merge system_prompt and memory_context into single system message, filter out system messages from conversation_history
   - deeptutor/services/session/context_builder.py: Filter out system messages from DB since summary is already added as system
   - deeptutor/agents/chat/chat_agent.py: Merge system prompt and RAG context into single system message

2. History reference (@reference) empty context issue:
   - deeptutor/services/session/turn_runtime.py: Add fallback to raw record content when NotebookAnalysisAgent.analyze() returns empty string

Benefits:
- Fixes Qwen vLLM compatibility issue completely
- Enables history reference feature to work correctly
- Reduces token usage by eliminating duplicate system messages
- Does not break any functionality - system context from DB is already in compressed_summary
- Cleaner message structure with exactly one system message at position 0

Testing:
- Verified with Qwen model via vLLM - no more template errors
- Chat with memory files (PROFILE.md/SUMMARY.md) works correctly
- Chat with conversation history works correctly
- History reference (@reference) now works correctly
- Conversation continuation works without errors
- No regression with other models (GPT-4, etc.)

Closes HKUDS#294
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant