Question about metrics when evaluating LVBench

[Yusux]

Thanks for your hard work.

I recently tried to reproduce the evaluation of your LongVT-RFT model on LVBench, but for some reason unknown, the results deviate from the metric reported in the paper.

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
lvbench-custom	Yaml	none	0	lvbench_score	↑	0.2479	±	0.011

Following README, I referred to lmms-eval (https://github.com/EvolvingLMMs-Lab/lmms-eval) and modified the task yaml and utils.py files (https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/lvbench) to obtain the correct local path and make max_new_tokens usable

dataset_path: lmms-lab/LVBench
dataset_kwargs:
  token: False
  cache_dir: /path/to/data/LVBench
  video: True
  # From_YouTube: True
test_split: train
task: lvbench-custom
output_type: generate_until
doc_to_visual: !function utils.lvbench_doc_to_visual
doc_to_text: !function utils.lvbench_doc_to_text
doc_to_target: "answer"
generation_kwargs:
  max_new_tokens: 4096
# The return value of process_results will be used by metrics
process_results: !function utils.lvbench_process_results
# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results
metric_list:
  - metric: lvbench_score
    aggregation: mean
    higher_is_better: true
lmms_eval_specific_kwargs:
  default:
    pre_prompt: ""
    post_prompt: "\nAnswer the question with the option letter"
metadata:
  - version: 0.0

import os
import re
from pathlib import Path

import yaml

with open(Path(__file__).parent / "lmms_eval.yaml", "r") as f:
    raw_data = f.readlines()
    safe_data = []
    for i, line in enumerate(raw_data):
        # remove function definition since yaml load cannot handle it
        if "!function" not in line:
            safe_data.append(line)
cache_name = yaml.safe_load("".join(safe_data))["dataset_kwargs"]["cache_dir"]


def lvbench_doc_to_visual(doc):
    cache_dir = cache_name
    video_path = doc["video_path"]
    assert os.path.exists(os.path.join(cache_dir, "video_chunks", video_path))
    video_path = os.path.join(cache_dir, "video_chunks", video_path)
    return [video_path]


def lvbench_doc_to_text(doc, lmms_eval_specific_kwargs=None):
    if lmms_eval_specific_kwargs is None:
        lmms_eval_specific_kwargs = {}
    if "pre_prompt" not in lmms_eval_specific_kwargs:
        lmms_eval_specific_kwargs["pre_prompt"] = ""
    if "post_prompt" not in lmms_eval_specific_kwargs:
        lmms_eval_specific_kwargs["post_prompt"] = "\nAnswer the question with the option letter"
    return lmms_eval_specific_kwargs["pre_prompt"] + doc["question"] + lmms_eval_specific_kwargs["post_prompt"]


def extract_characters_regex(s):
    s = s.strip()
    answer_prefixes = [
        "The best answer is",
        "The correct answer is",
        "The answer is",
        "The answer",
        "The best option is" "The correct option is",
        "Best answer:" "Best option:",
    ]
    for answer_prefix in answer_prefixes:
        s = s.replace(answer_prefix, "")

    if len(s.split()) > 10 and not re.search("[ABCD]", s):
        return ""

    matches = re.search(r"[ABCD]", s)
    if matches is None:
        return ""
    return matches[0]


def lvbench_process_results(doc, results):
    """
    Args:
        doc: a instance of the eval dataset
        results: [pred]
    Returns:
        a dictionary with key: metric name (in this case videomme score), value: metric value
    """
    pred = results[0]
    pred_ans = extract_characters_regex(pred)
    # gt_ans = doc["answer"].lower().strip().replace(".", "")
    gt_ans = doc["answer"]
    score = pred_ans == gt_ans

    # return {f"videomme_perception_score": data_dict for metric in matrices}
    return {f"lvbench_score": score}

The vllm version used is 0.13.0, and the command used is

CUDA_VISIBLE_DEVICES=0 vllm serve /path/to/weights/LongVT-RFT --port 28100 --tool-call-parser hermes --enable-auto-tool-choice --trust-remote-code --chat-template /path/to/LongVT/examples/eval/tool_call_qwen2_5_vl.jinja

And the script used

#!/bin/bash

export OMP_NUM_THREADS=16
export TORCHCODEC_NUM_THREADS=16

# Environment variables for LLM Judge
export OPENAI_API_BASE="http://<server_ip>:28100/v1"
export OPENAI_API_KEY="EMPTY"
export OPENAI_MODEL_NAME="/path/to/weights/LongVT-RFT"

# Cache settings
export DECORD_EOF_RETRY_MAX=409600
export LMMS_EVAL_USE_CACHE=True
export LMMS_EVAL_HOME="./cache"  # Set your cache directory

# Arguments
NUM_CPUS=1
TASK_NAME=lvbench-custom
IS_QWEN3_VL=False
MAX_FRAME_NUM=768

# Path to MCP server for tool calling
MCP_PATH="/path/to/LongVT/examples/video_tools/mcp_server.py"

# Run evaluation
accelerate launch --cpu --num_processes=1 --num_machines=1 --mixed_precision=no --dynamo_backend=no -m lmms_eval \
    --model async_openai \
    --model_args model_version=$OPENAI_MODEL_NAME,mcp_server_path=$MCP_PATH,fps=1,max_frames=$MAX_FRAME_NUM,max_pixels=50176,base_url=$OPENAI_API_BASE,api_key=$OPENAI_API_KEY,num_cpus=$NUM_CPUS,timeout=12000,is_qwen3_vl=$IS_QWEN3_VL \
    --tasks $TASK_NAME \
    --batch_size 1 \
    --output_path ./eval_logs \
    --log_samples \
    --include_path .

It may be worth noting that the cached response contains duplicate <|vision_start|> <|vision_end|> tokens. Here is the cache file https://drive.google.com/file/d/1VlwJxTBe33_2v5QGYMyzQSKUfynLSOAh.

I would be extremely grateful if you could help me solve this problem.

[mwxely]

Hi @Yusux ,

Thank you for your interest in LongVT and for providing such detailed reproduction steps! I have carefully reviewed your configuration and identified several key discrepancies that are likely causing the performance gap.

Official Environment

• vLLM version: 0.12.0
• Python: 3.12
• other dependencies should follow the official lmms-eval repo
• Hardware: 1 Node x 8 A800-SXM4-80GB GPUs

Configuration Mismatches

1. Task YAML & Utils

Your custom implementation differs significantly from our official configuration:

Aspect	Your Config	Official Config
token	False	True
doc_to_messages	Not used	Uses SYSTEM_PROMPT with reasoning tags
Metric	lvbench_score (simple match)	acc_score + format_score (via compute_score)
process_results	Simple pred_ans == gt_ans	Uses custom_rewards.lmms_lab_recipe.compute_score

Recommendation: Please use the task configurations directly from lmms_eval_tasks/lvbench/ as specified in the provided eval script (Line 53: --include_path ./lmms_eval_tasks).

2. OPENAI_MODEL_NAME

Your Config -> export OPENAI_MODEL_NAME="/path/to/weights/LongVT-RFT"

Official Config -> export OPENAI_MODEL_NAME="judge"

This environment variable is used for the LLM-as-Judge component that evaluates reasoning traces and open-ended responses, not for specifying the model weights.

3. MAX_FRAME_NUM

For the LongVT-7B-RFT checkpoint, the results reported in our paper use MAX_FRAME_NUM=512

4. Accelerate Launch Arguments

Your Config -> accelerate launch --cpu --num_processes=1 --num_machines=1 --mixed_precision=no --dynamo_backend=no

Official Config -> accelerate launch --num_processes=8 --main_process_port 12345

Key differences:

• --cpu: Forces CPU execution, which may cause numerical differences and is significantly slower
• --num_processes=1: We use 8 processes with data parallelism (--data-parallel-size 8 for vLLM)
• --mixed_precision=no: May affect floating-point precision in intermediate computations

These infrastructure differences could also contribute to the abnormal responses as shared in your provided cache file.

Recommended Steps

1. Use the official task configs from lmms_eval_tasks/lvbench/lvbench_tool.yaml without modification
2. Use the official metric computation from lmms_eval_tasks/lvbench/utils.py without modification
3. Set OPENAI_MODEL_NAME="judge" and ensure your LLM judge server is properly configured
4. Use MAX_FRAME_NUM=512 for LongVT-7B-RFT
5. Run on GPU with our recommended accelerate settings

If issues persist after aligning these configurations, I would be happy to share our inference JSONL results and statistics for your references. Please feel free to follow up!

[Yusux]

Thank you for your patient reply.

I realized I misused the official lmms-eval task and utils.py scripts, neglecting functions like lvbench_tool_doc_to_messages in the repository, assuming I had modified them to be equivalent to the prompts in the repository. This may be the cause of the problem.

As for not using LLM-as-Judge, it's because I wanted to obtain metrics aligned with the official LVBench without leveraging LLM (https://github.com/zai-org/LVBench/blob/main/scripts/test_acc.py).

I will try your suggestions as soon as possible. Thank you again for your reply.

[mwxely]

I fully understand your needs — just to clarify: as shown in the README.md screenshot above, if you want to evaluate using "direct answer" mode on LVBench, you should specify the task lvbench (highlighted in the red box). However, if you want to reproduce the results reported in our paper, note that LongVT-series models fall under the "Tool Calling" category, as their responses include both reasoning traces and tool invocations. To properly enable the agentic capabilities and use the correct evaluation prompts, you should specify lvbench_tool (highlighted in the green box) instead.

[CeeZh]

Hi,

I have the same issue when trying to reproduce LVBench results. Did you reproduce the number in the paper?

Thank you for your patient reply.

I realized I misused the official lmms-eval task and utils.py scripts, neglecting functions like lvbench_tool_doc_to_messages in the repository, assuming I had modified them to be equivalent to the prompts in the repository. This may be the cause of the problem.

As for not using LLM-as-Judge, it's because I wanted to obtain metrics aligned with the official LVBench without leveraging LLM (https://github.com/zai-org/LVBench/blob/main/scripts/test_acc.py).

I will try your suggestions as soon as possible. Thank you again for your reply.

[Yusux]

Hi, @CeeZh

Unfortunately, I'm still working on this issue. I've tried Python 3.12, vllm 0.12.0, and transformers 4.57.3, but the duplicate token problem persists. LongVT-RL's weights might be better than LongVT-RFT in solving this problem.

By the way, although the task I want to measure is lvbench, as highlighted in the red box of the image above, I've also tried using llm-as-judge, and the scores are closer to the model's reasoning process and largely consistent with those in the paper. If you want to reproduce the scores, you need to use this method.

Hi,

I have the same issue when trying to reproduce LVBench results. Did you reproduce the number in the paper?

Thank you for your patient reply.
I realized I misused the official lmms-eval task and utils.py scripts, neglecting functions like lvbench_tool_doc_to_messages in the repository, assuming I had modified them to be equivalent to the prompts in the repository. This may be the cause of the problem.
As for not using LLM-as-Judge, it's because I wanted to obtain metrics aligned with the official LVBench without leveraging LLM (https://github.com/zai-org/LVBench/blob/main/scripts/test_acc.py).
I will try your suggestions as soon as possible. Thank you again for your reply.

[CeeZh]

Hi, @CeeZh

Unfortunately, I'm still working on this issue. I've tried Python 3.12, vllm 0.12.0, and transformers 4.57.3, but the duplicate token problem persists. LongVT-RL's weights might be better than LongVT-RFT in solving this problem.

By the way, although the task I want to measure is lvbench, as highlighted in the red box of the image above, I've also tried using llm-as-judge, and the scores are closer to the model's reasoning process and largely consistent with those in the paper. If you want to reproduce the scores, you need to use this method.

Hi,
I have the same issue when trying to reproduce LVBench results. Did you reproduce the number in the paper?

Thank you for your patient reply.
I realized I misused the official lmms-eval task and utils.py scripts, neglecting functions like lvbench_tool_doc_to_messages in the repository, assuming I had modified them to be equivalent to the prompts in the repository. This may be the cause of the problem.
As for not using LLM-as-Judge, it's because I wanted to obtain metrics aligned with the official LVBench without leveraging LLM (https://github.com/zai-org/LVBench/blob/main/scripts/test_acc.py).
I will try your suggestions as soon as possible. Thank you again for your reply.

Hi @Yusux and @mwxely

Thanks for your response. I used the same library versions and also evaluated with LLM-as-judge. I tested LongVT-RL on lvbench-tool, but I still wasn’t able to reproduce the reported results.

Just to confirm, did you also evaluate using lvbench-tool, or was a different evaluation setup used?

[mwxely]

Hi, @CeeZh
Unfortunately, I'm still working on this issue. I've tried Python 3.12, vllm 0.12.0, and transformers 4.57.3, but the duplicate token problem persists. LongVT-RL's weights might be better than LongVT-RFT in solving this problem.
By the way, although the task I want to measure is lvbench, as highlighted in the red box of the image above, I've also tried using llm-as-judge, and the scores are closer to the model's reasoning process and largely consistent with those in the paper. If you want to reproduce the scores, you need to use this method.

Hi,
I have the same issue when trying to reproduce LVBench results. Did you reproduce the number in the paper?

Thank you for your patient reply.
I realized I misused the official lmms-eval task and utils.py scripts, neglecting functions like lvbench_tool_doc_to_messages in the repository, assuming I had modified them to be equivalent to the prompts in the repository. This may be the cause of the problem.
As for not using LLM-as-Judge, it's because I wanted to obtain metrics aligned with the official LVBench without leveraging LLM (https://github.com/zai-org/LVBench/blob/main/scripts/test_acc.py).
I will try your suggestions as soon as possible. Thank you again for your reply.

Hi @Yusux and @mwxely

Thanks for your response. I used the same library versions and also evaluated with LLM-as-judge. I tested LongVT-RL on lvbench-tool, but I still wasn’t able to reproduce the reported results.

Just to confirm, did you also evaluate using lvbench-tool, or was a different evaluation setup used?

Hi @CeeZh , thanks for the update — and sorry you are still running into this.

For your reference, the LongVT-RL score reported in the paper was obtained with:

• Task: lvbench_tool
• Setting: MAX_FRAME_NUM=768
• Hardware: 1 node × 8 A800-SXM4-80GB GPUs
• Parallelism: num_processes=8 (we shard the dataset into 8 splits and run them in parallel)

In our experience, the results can be sensitive to environment / hyperparameters, and LLM-as-Judge can also introduce instability.

Would you mind sharing a bit more detail (similar to what @Yusux provided), so we can narrow this down more systematically? For example:

• The exact command you ran (and any overrides)
• The exact evaluation script you used
• Which judge model and decoding settings were used (e.g., temperature / top_p / max_tokens)

Also, have you inspected the generated JSONL outputs from inference? Sometimes the score drop is explained by silent failures. In particular, could you check whether the JSONL contains any of the following:

• internal server error (HTTP 500)
• tool-calling failures (e.g., excessive unreasonable crops)
• parsing/format errors (e.g., invalid JSON with intractable predictions)

If it would help, I can also share:

• the stat aggregation file we used for the paper (.json)
• the corresponding inference outputs (.jsonl)

so you can diff/compare against your run.

I am also CC’ing my co-authors @XIAO4579 @kcz358 to see whether they could join this discussion and provide some insights regarding the reproduction.