Support DeepSeek v3.2 encoding module

[soaringk]

DeepSeek-V3.2 introduces significant updates to its chat template compared to prior versions. The primary changes involve a revised format for tool calling and the introduction of a "thinking with tools" capability.

To assist the community in understanding and adapting to this new template, we have provided a dedicated encoding folder, which contains Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model and how to parse the model's text output.

A brief example is illustrated below:

import transformers
# encoding/encoding_dsv32.py
from encoding_dsv32 import encode_messages, parse_message_from_completion_text

tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3.2")

messages = [
    {"role": "user", "content": "hello"},
    {"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
    {"role": "user", "content": "1+1=?"}
]
encode_config = dict(thinking_mode="thinking", drop_thinking=True, add_default_bos_token=True)

# messages -> string
prompt = encode_messages(messages, **encode_config)
# Output: "<｜begin▁of▁sentence｜><｜User｜>hello<｜Assistant｜></think>Hello! I am DeepSeek.<｜end▁of▁sentence｜><｜User｜>1+1=?<｜Assistant｜><think>"

# string -> tokens
tokens = tokenizer.encode(prompt)
# Output: [0, 128803, 33310, 128804, 128799, 19923, 3, 342, 1030, 22651, 4374, 1465, 16, 1, 128803, 19, 13, 19, 127252, 128804, 128798]

Important Notes:

This release does not include a Jinja-format chat template. Please refer to the Python code mentioned above.
The output parsing function included in the code is designed to handle well-formatted strings only. It does not attempt to correct or recover from malformed output that the model might occasionally generate. It is not suitable for production use without robust error handling.
A new role named developer has been introduced in the chat template. This role is dedicated exclusively to search agent scenarios and is designated for no other tasks. The official API does not accept messages assigned to developer.

Reference: https://huggingface.co/deepseek-ai/DeepSeek-V3.2

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!

[github-actions]

internal source has been updated, please review the changes!