Json Error in evaluation process

[wkfdb]

I was running the evaluation of the LongVT-RFT model and I found that the result is extremely low:
"videomme_w_subtitle_reward_tool": {
"alias": "videomme_w_subtitle_reward_tool",
"videomme_perception_score,none": 19.185185185185187,
"videomme_perception_score_stderr,none": "N/A",
"acc_score,none": 0.18555555555555556,
"acc_score_stderr,none": 0.00748283676164936,
"format_score,none": 0.27518518518518487,
"format_score_stderr,none": 0.008474372792357103
}

By checking the reason I found that the LongVT-RFT server gets the following json error:

Error in extracting tool call from response.
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] Traceback (most recent call last):
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] File "xxxxxxxxx/envs/eval-longvt/lib/python3.11/site-packages/vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py", line 142, in extract_tool_calls
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] raw_function_calls = [
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] ^
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] File "xxxxxxxxx/envs/eval-longvt/lib/python3.11/site-packages/vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py", line 143, in
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] json.loads(match[0] if match[0] else match[1])
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] File "xxxxxxxxx/envs/eval-longvt/lib/python3.11/json/init.py", line 346, in loads
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] return _default_decoder.decode(s)
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] File "xxxxxxxxxx/envs/eval-longvt/lib/python3.11/json/decoder.py", line 337, in decode
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] obj, end = self.raw_decode(s, idx=_w(s, 0).end())
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] File "xxxxxxxxxx/envs/eval-longvt/lib/python3.11/json/decoder.py", line 353, in raw_decode
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] obj, end = self.scan_once(s, idx)
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] ^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=65904) ERROR 12-23 08:25:31 [hermes_tool_parser.py:168] json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 9 (char 8)

I didn't change any source codes and set the whole framework following the README file, any ideas on why the above JSON error occurs? I'm trying to find out.

[mwxely]

Hi @wkfdb, thanks for reporting this issue!

Analysis

The error JSONDecodeError: Unterminated string starting at: line 1 column 9 (char 8) indicates that the JSON content inside <tool_call>...</tool_call> tags is being truncated before it can be fully parsed. The error position (char 8) suggests the truncation happens very early, around {"name":.

I have reproduced the evaluation pipeline locally with the default setup, and it works correctly on my end. Here is my environment:

• vLLM version: 0.12.0
• Python: 3.12
• other dependencies should follow the official lmms-eval repo

Possible Causes

1. vLLM version mismatch: Different vLLM versions may have different hermes_tool_parser implementations
2. Missing or incorrect chat template: The --chat-template parameter is critical for Qwen2.5-VL models
3. Model checkpoint issues: The model files might be incomplete or corrupted
4. max_tokens too low: The output might be truncated due to length limits

Diagnostic Information Needed

To help pinpoint the issue, please provide:

1. vLLM version: Run python -c "import vllm; print(vllm.__version__)"
2. Your startup command: Please share your complete vLLM server startup command
3. Chat template: Are you using --chat-template ./examples/eval/tool_call_qwen2_5_vl.jinja? Is the file path correct relative to your working directory?
4. Model checkpoint: Which checkpoint are you using? Is it downloaded completely?
5. Raw model output: If possible, enable --log_samples and $LMMS_EVAL_USE_CACHE to check the inference results stored in $LMMS_EVAL_HOME/eval_cache/ and see what the model is actually generating

Note

I have pushed a fix to examples/video_tools/mcp_server.py that explicitly sets min_pixels configuration. Please stash your local changes and pull the latest changes and re-run the evaluation on your end to verify if my suggestions can locate this issue.

[wkfdb]

The result was still strange.
First of all, my vLLM version is 0.12.0, and the startup command is

vllm serve /home/notebook/nas/data/group/models/LongVT-RL \
    --chat-template ./examples/eval/tool_call_qwen2_5_vl.jinja \
    --tool-call-parser hermes \
    --enable-auto-tool-choice \
    --data-parallel-size 4 \
    --trust-remote-code

The checkpoint is downloaded from the huggingface link and It seems downliaded completely since most of the output follows the answer format.
I evaluated the LongVT-RL checkpoint and the results are as follows:
"results": {
"videomme_w_subtitle_reward_tool": {
"alias": "videomme_w_subtitle_reward_tool",
"videomme_perception_score,none": 24.51851851851852,
"videomme_perception_score_stderr,none": "N/A",
"acc_score,none": 0.3725925925925926,
"acc_score_stderr,none": 0.009306587971599015,
"format_score,none": 0.677629629629628,
"format_score_stderr,none": 0.008790329034963393
}
},
The accuracy score is 0.37 which is still strange.
There are some output examples by printing the all_response (the tag and tag):

doc_id:0: <think>The video opens with a news channel intro, leading into two anchors presenting a segment, followed by various shots of festive food and a close-up of a mug and cookie. The scene then transitions to a detailed historical discussion about Christmas trees, starting with a decorated tree, then showing an old illustration of people gathered around an early Christmas tree adorned with apples, candles, and berries. This is followed by a display of different types of ornaments, including baubles and tinsel, before showcasing a modern Christmas tree in a home setting with presents. The underpinning theme of the video is clearly the evolution and popularization of Christmas tree decorations. To specifically address the question regarding the initial decoration of the German Christmas tree with apples, candles, and berries, a close examination of the video segment detailing these early decorations is required, particularly within the window from [41.00s – 57.00s].</think>
<tool_call>crop_video {"video_path": "/mnt/videodata_nas/UnifiedVideoDatasetHub/VideoLLMDataset/BenchMark/Video-MME/data/fFjv93ACGo8.mp4", "start_time": 41.0, "end_time": 57.0}</tool_call></tool_response><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url></tool_response><think>**Evidence**: This window [41.00s – 57.00s] provides clear visual evidence. From approximately 41.00s to 45.00s, a decorated Christmas tree is shown, featuring prominent red apples, golden candles, and clusters of small red berries. An old illustration within this segment, specifically from ~46.00s to 49.00s, depicts a family gathered around a similar tree, also clearly adorned with apples, candles, and berries. The visual density and prominence in both the animated tree and the historical illustration suggest that all three items are present. However, the berries appear as small clusters on multiple branches, the apples are larger and fewer in number, and the candles are tall and stand out. While an exact count isn't possible, the *appearance* of berries is the most numerous due to their small size and clustered nature.

**Integration**: My earlier hypothesis that this segment would contain the answer was correct. The visual information from ~41.00s to 49.00s directly addresses the question. Although all three are decorated on the tree and illustrations, the berries (C) seem to have the highest combined total when considering their small, repeated presence on the branches.

**Self-reflection**: This window [41.00s – 57.00s] was well-localized and fully sufficient for answering the question. The visual content from ~41.00s to 49.00s directly shows the tree decorated with apples, candles, and berries, allowing for a reasonable estimation of their relative abundance. A tighter window could focus from approximately 41.00s to 49.00s to capture the core visual information.</think><answer>Berries</answer>

doc_id:1: <think>The video opens with a "NEW AT 6" title card, featuring two anchors at a news desk, which sets the stage for a report. Following this introduction, the content shifts to various festive scenes, including a family enjoying a holiday meal, close-ups of different Christmas ornaments, and illustrated historical depictions of early Christmas trees and decorations. Throughout these segments, text overlays provide historical context about the evolution of Christmas tree traditions and ornaments. The anchors are seen intermittently, reinforcing the educational nature of the presentation. Given the focus on explaining the history and development of Christmas decorations and tree traditions, the video's primary genre is best described as a news report that introduces the history behind Christmas decorations, making option A the most appropriate choice. The other options, such as a documentary on recipes (B), a travel vlog (C), or a crafting tutorial (D), do not align with the video's content, which primarily emphasizes historical information.</think>
<tool_call>crop_video {"video_path": "/mnt/videodata_nas/UnifiedVideoDatasetHub/VideoLLMDataset/BenchMark/Video-MME/data/fFjv93ACGo8.mp4", "start_time": 0.0, "end_time": 107.0}</tool_call></tool_response>Error executing tool crop_video: end_time (107.0s) exceeds video duration (74.31s)</tool_response><think>**Evidence**: This window [0.00, 107.00] was mis-localized as it exceeds the video's actual duration by 32.69 seconds (107.00 - 74.31 = 32.69). The video segment up to the actual end time of 74.31 seconds contains the introductory "NEW AT 6" title card, the appearance of two news anchors at a desk (suggesting a report format), and the very beginning of a family scene. However, the detailed discussion of Christmas tree history and ornaments, which is crucial for determining the genre, starts around 12 seconds into the video and continues throughout the video. The frames from approximately 12s onwards (e.g., historical illustrations and text overlays about "paradise trees") provide the specific content that would allow for a precise genre identification.

**Integration**: My earlier hypothesis that the video's primary genre is best described as a news report that introduces the history behind Christmas decorations (option A) is supported by the detailed content visible within the relevant portion of this window (e.g., the family scene at a table with food, the historical narrative about Christmas trees and ornaments). This observation confirms the initial assessment despite the initial mis-localization.

**Self-reflection**: The window [0.00, 107.00] was indeed too wide, as the actual video content ends at 74.31 seconds. The decisive cues for classifying the video's genre are contained within the first 74.31 seconds, specifically from around 12 seconds onwards. To refine future localizations, I would focus on the segment from 12.00 seconds to 74.31 seconds, as this portion provides ample evidence of the video's content and purpose.</think><answer>A news report that introduces the history behind Christmas decorations.</answer> 

doc_id:2: <think>The video opens with a news channel's "NEW AT 6" graphic before transitioning to two anchors in a studio setting discussing seasonal topics, around 10-25 seconds. Following this, the content shifts to various festive images, including a family meal, a mug with a snowflake cookie, and different styles of Christmas trees, some with historical commentary, from around 25 seconds to 70 seconds. The narrative then focuses on the history of Christmas ornaments, featuring illustrations and images of early decorations, continuing through to about 1:40. Towards the very end of the video, there is a shot of a decorated Christmas tree near a fireplace, which is where we should look for the answer to the question about the number of red socks above the fireplace. To accurately identify the number of red socks, I will crop the video to the segment that displays this specific scene for a closer inspection.</think>
<tool_call>crop_video {"video_path": "/mnt/videodata_nas/UnifiedVideoDatasetHub/VideoLLMDataset/BenchMark/Video-MME/data/fFjv93ACGo8.mp4", "start_time": 109.0, "end_time": 177.0}</tool_call></tool_response>Error executing tool crop_video: start_time (109.0s) exceeds video duration (74.31s)</tool_response><think>**Evidence**: My initial localization for this question was incorrect. The video segment from [109.00s – 177.00s] is outside the actual video's end, which concludes around 1:52 (112 seconds). Therefore, no visual content relevant to the question was present in this window. The video ends with a decorated Christmas tree near a fireplace, but it does not provide a clear view of any red socks above the fireplace. The frames available up to the video's actual conclusion show the tree and part of the fireplace, but no distinct red socks are visible in those later frames.

**Integration**: Given that the specified cropping interval is invalid and no red socks are visible in the actual video content, particularly in the final seconds where the question is focused, I must revise my earlier assumption. My previous thought process incorrectly anticipated finding the answer within this range.

**Self-reflection**: This window was indeed mis-localized. The correct segment to inspect for the presence of red socks above the fireplace would be towards the very end of the video, specifically around 1:50 to 1:52. Since this is the last segment of the video, and no red socks are observed, it indicates there is no explicit answer to the question as posed. In the absence of visible evidence, the most reasonable conclusion is that the question cannot be directly answered based on the provided video content.</think><answer>C. 2.</answer> 

doc_id:6: <think>The video presents a hand-drawn animation narrating the history of hamburgers, starting with the Mongols in the 13th century, depicted through simple drawings and cut-out food items. The narrative progresses through various cultural exchanges, showing maps and flags to represent different countries, such as Russia and Germany, appearing around 29-33 seconds. The animation continues through Europe and the United States, with additional flags illustrated or referenced around 34-40 seconds, and then details the development of the hamburger steak and its eventual transformation into the modern hamburger around 41-76 seconds. To accurately count the distinct national flags that appear in the video as part of this historical narrative, I will examine the segment that introduces these geographical and cultural elements in detail, specifically focusing on the time range [13.00s – 76.00s].</think>
<tool_call>crop_video {"video_path": "/mnt/videodata_nas/UnifiedVideoDatasetHub/VideoLLMDataset/BenchMark/Video-MME/data/HIjX8OPuf-w.mp4", "start_time": 13.0, "end_time": 76.0}</tool_call></tool_response><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url></tool_response><think>Evidence: Within the [13.00s – 76.00s] window, several national flags are clearly depicted. At approximately 29.00s, a Russian flag (red, white, and blue tricolor) appears next to a map of Russia. Shortly after, at around 30.00s, a German flag (black, red, gold) is shown alongside a map of Germany. Later, at approximately 40.00s, an American flag (red stripes and white stars on a blue canton) is displayed next to a map of the United States. Finally, at around 44.00s, a Dutch flag (red, white, blue horizontal stripes) appears near a map of the Netherlands. These are the only distinct national flags that I can confirm visually appear as part of the historical narrative in this segment.

Integration: This detailed inspection confirms my earlier hypothesis about the presence of flags representing Russia, Germany, and the United States. I have now identified a fourth distinct flag: the Netherlands', bringing the total count to four. Therefore, the previous working assumption was revised to include this additional flag.

Self-reflection: This window [13.00s – 76.00s] was well-localized and fully sufficient for identifying all the national flags in the video. The crucial information for counting the flags is concentrated within this range, particularly between 29.00s and 44.00s, making the initial localization accurate and effective for the subgoal.</think><answer>B</answer> 

It seems that sometimes it can not get valid parameter for tool calling, and the answers in the answer tag sometime only containing the choice, sometimes contains the choice and a description and sometimes only a description that cannot extract a choice answer for accuracy computing. Do you have any ideas? I assume that the prompt is different between evaluation and the training process.

[mwxely]

Hi @wkfdb ,

Thank you for the follow-up! The JSON parsing error in hermes_tool_parser.py indeed suggests that the model's tool call output may not be properly formatted, which can stem from various configuration or environment factors.

To help figure out the root cause from my side more effectively, could you please provide the following details:

1. Evaluation script - the full bash script used to launch videomme_w_subtitle_reward_tool evaluation
2. Task YAML configuration - are you using our provided configs from lmms_eval_tasks/ or custom ones?
3. Hardware setup - GPU model, number of GPUs, and any relevant infrastructure settings

As I recently proposed in Issue #11, configuration and infrastructure differences are common causes of discrepancies on model responses as well.

Looking forward to your response with more details so I can better assist you with the reproduction!

Note

I noticed that you initially mentioned evaluating the LongVT-RFT checkpoint, but in your follow-up reply, the vLLM server was started with LongVT-RL instead. This mismatch could also lead to model's unexpected behaviors.

[wkfdb]

Thanks for the quick and detail reply. Here are some main points I'd like to express:
1、I tried RFT-checkpoint first and I realize that I may incorrectly downloaded SFT checkpoint as the RFT version. So I changed to RL checkpoint which I ensure that I downloaded the correct version.
2、The reason for the json error is because the RFT(or actually the SFT) checkpoint may output unlimited img-padding token which broke the JSON format. After changing to RL checkpoint this error do not occur anymore.
3、I use 4 H20 for evaluation and the bash script and command are as follows:

# Environment variables
export OPENAI_API_BASE="http://localhost:8000/v1"
export OPENAI_MODEL_NAME="judge"
export OPENAI_BASE_URL="http://10.77.211.240:8000/v1"
export OPENAI_API_KEY="EMPTY"
export USE_LLM_JUDGE=True
export DECORD_EOF_RETRY_MAX=409600

# Optional
# export LMMS_EVAL_USE_CACHE=True
# export LMMS_EVAL_HOME="your_cache_directory"

# Arguments
CKPT_PATH=$1                # Path to model checkpoint
TASK_NAME=$2                # Evaluation task name
IS_QWEN3_VL=$3              # Whether using Qwen3-VL model (True/False)
MAX_FRAME_NUM=${4:-768}     # Number of frames (Default:768)

# Path to MCP server for tool calling
MCP_PATH="./examples/video_tools/mcp_server.py"

# Activate lmms-eval environment
# source /your/path/to/lmms-eval/.venv/bin/activate


echo "run evalation"
# Run evaluation
accelerate launch --num_processes=4 --main_process_port 12345 -m lmms_eval \
    --model async_openai \
    --model_args model_version=$CKPT_PATH,mcp_server_path=$MCP_PATH,fps=1,max_frames=$MAX_FRAME_NUM,max_pixels=50176,base_url=$OPENAI_API_BASE,api_key=$OPENAI_API_KEY,num_cpus=1,timeout=12000,is_qwen3_vl=$IS_QWEN3_VL \
    --tasks $TASK_NAME \
    --batch_size 1 \
    --output_path ./eval_logs/temp \
    --log_samples \
    --include_path ./lmms_eval_tasks

The command:

bash examples/eval/run_eval.sh /home/notebook/nas/data/group/models/LongVT-RL videomme_w_subtitle_reward_tool False 512

4、I use the yaml and utils that provided in the lmms_eval_tasks dir and I only changed the path to read the VideoMME benchmark. I have tried as much as possible to align with the official evaluation setup and the results is just not as the reports in the paper and I'm confused.
5、Thanks for your quick and patient response.

[wkfdb]

By the way, the invalid tool call parameter are mainly comes from invalid end time which is bigger than the whole video length. This error mainly occurs on the short videos.

[mwxely]

Your provided bash script looks correct to me. Just a quick note on hyperparameters: in the paper (dense frame sampling setting), we report LongVT-SFT and LongVT-RL results using MAX_FRAME_NUM=768 (aligned with the Qwen2.5-VL baseline) and LongVT-RFT with MAX_FRAME_NUM=512.

Regarding the tool call formatting issue, I would like to debug further from the MCP server end. Could you please share a complete copy of your inference results (the JSONL files) via email / Google Drive / WeChat or any other convenient approaches? This will greatly help me analyze the root cause of the performance inconsistency from your side.