How to reproduce the benchmark numbers in the RynnBrain technical report?

[kding1]

Thanks for sharing the great work! I followed the step-by-step instructions and use the Rynn Bench dataset with a few simple code change. However it looks like the benchmark numbers in Table 3 from TR paper cannot be reproduced for open-source models such as Qwen3-VL-8B-Instruct and Cosmos-Reason2, especially for RynnBrain-Grounding, Area and Affordance. Is there any important details missing in the instruction for hyper-parameters and/or datasets?

[ZMHH-H]

Hi, thanks for your interest in RynnBrain. For location tasks in RynnBrain-Bench, the standard evaluation protocol requires the model to output both the keyframe index <frame i> and the spatial coordinate point in a single inference. However, we found that this protocol is too difficult for some open-source models, making it hard for them to produce outputs in the appropriate format. Therefore, we simplified the evaluation protocol for open-source models by using a two-step inference process:  (1) given the frame sequence and the task instruction, the model selects the most task-relevant frame; (2) given the selected frame and the task instruction, the model outputs the coordinates. However, we strongly recommend using the standard evaluation protocol if your model is capable enough.
We provide some prompt examples for each task.

• Frame Selection: "The task is '{task}'. Analyze all images and select the single image that is most relevant for the task instruction or best contains the target object of interest. Output the ID of the selected image in the format 'Picture [ID]'. For example: Picture 3."
• Grounding: "Please provide the bounding box coordinate of the region this sentence describes: {task}. Provide the output strictly in the following format: '[x1, y1, x2, y2]'."
• Area: "The task is '{task}'. Please predict 1-4 points within the region that satisfy the task instruction. Output strictly in the following format: [(x1, y1), (x2, y2), ...]."
• Affordance: "You are a robot using the joint control. The task is '{task}'. Please predict 1-4 affordance points that indicate how to interact with the relevant object(s) to execute the task. Output strictly in the following format: [(x1, y1), (x2, y2), ...]."
• Trajectory: "You are a robot using the joint control. The task is "{task}". Please predict up to 10 key trajectory points to complete the task. Your answer should be formatted as a list of tuples, i.e. [(x1, y1), (x2, y2), ...]."

[kding1]

Thank you @ZMHH-H this is very helpful. Do you have plan to release those details soon to fully reproduce the numbers - or you expect researchers to explore the two-step inference process themselves given the prompts you shared?

[ZMHH-H]

We provide an example inference function for open-source models. As models differ in design and input, this function requires ad-hoc adaptation for each model. We encourage researchers to explore the optimal parameter choices themselves.

import re
import json
import torch
import numpy as np
from PIL import Image
from typing import List, Optional, Tuple, Union
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info, smart_resize

MIN_PIXELS = 600*28*28
MAX_PIXELS = 600*28*28
THINKING = False

def reformat_image_question(img_paths, question):
    image = Image.open(img_paths[0]).convert('RGB')
    width, height = image.width, image.height

    # calcualte the actual image input resoluiton
    input_height, input_width = smart_resize(height, width, factor=28, min_pixels=MIN_PIXELS, max_pixels=MAX_PIXELS)
    # print('input_height, input_width', input_height, input_width)
    
    def scale_coordinates(match):
        x = int(match.group(1))
        y = int(match.group(2))
        
        new_x = round(x * input_width / 1000)
        new_y = round(y * input_height / 1000)

        return f"({new_x},{new_y})"

    match = re.findall(r"<frame\s*(\d+)>", question)
    target_frames = [int(m) for m in match]

    question = re.sub(r'<frame\s*(\d+)>', r'Picture \1', question)

    question = re.sub(r'\(\s*(\d+)\s*,\s*(\d+)\s*\)', scale_coordinates, question)

    total_frames = len(img_paths)
    
    max_frames = 30
    if total_frames > max_frames:
        indices = np.round(np.linspace(0, total_frames - 1, max_frames)).astype(int)
        
    else:
        indices = np.arange(total_frames)
    
    for tf in target_frames:
        if tf not in indices:
            closest_pos = np.argmin(np.abs(indices - tf))
            indices[closest_pos] = tf
    img_paths = [img_paths[i] for i in indices]
    return img_paths, indices, question

class MLLM:
    def __init__(
        self,
        pretrained: str = "MLLM",
        device_map: Optional[any] = "auto",) -> None:

        self.pretrained = pretrained
        print('start loading model', self.pretrained)
        self.model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
            self.pretrained, torch_dtype=torch.bfloat16, device_map=device_map
        )
        print('finish loading model')

        # default processor
        self.processor = AutoProcessor.from_pretrained(self.pretrained, min_pixels=MIN_PIXELS, max_pixels=MAX_PIXELS)

    def inference_rynnbrain_cog(self, text: str, image: Union[list, str], task="general"):
        if isinstance(image, str):
            image = [image]

        image, indices, text = reformat_image_question(image, text)

        if task == "area":
            # print("Pointing task detected. Adding pointing prompt.")
            text = f"Task: {text}. Please predict 1-4 points within the region that satisfy the task instruction. Output strictly in the following format: [(x1, y1), (x2, y2), ...]."
        elif task == "affordance":
            text = f"You are a robot using the joint control. The task is '{text}'. Please predict 1-4 affordance points that indicate how to interact with the relevant object(s) to execute the task. Output strictly in the following format: [(x1, y1), (x2, y2), ...]."
        elif task == "trajectory":
            text = f"You are a robot using the joint control. The task is \"{text}\". Please predict up to 10 key trajectory points to complete the task. Your answer should be formatted as a list of tuples, i.e. [(x1, y1), (x2, y2), ...]."
        elif task == "referring":
            text = f"Please provide the bounding box coordinate of the region this sentence describes: {text}. Provide the output strictly in the following format: '[x1, y1, x2, y2]'."
        elif task == "select_frame":
            text = f"The task is \"{text}\". Analyze all images and select the single image that is most relevant for the task instruction or best contains the target object of interest. Output the ID of the selected image in the format 'Picture [ID]'. For example: Picture 3."

        # disable thinking
        if not THINKING:
            text = f"{text}. /no_think"

        content = []
        for i, path in enumerate(image):
            content.append({"type":"text","text":f"Picture {indices[i]}: "})
            content.append({"type": "image", "image": path}) # f"file://{path}"
        content.append({"type": "text", "text": f"{text}"})
        messages = [
                {"role":"user","content":content}
        ]
        inputs = self.processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt")
        inputs = inputs.to(self.model.device, dtype=torch.bfloat16)
        
        # Inference: Generation of the output
        generated_ids = self.model.generate(**inputs, max_new_tokens=2048, use_cache=True)
        generated_ids_trimmed = [out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]
        answers = self.processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)
        answers = answers[0] if answers else ""
        # print(answers)

        return answers

    def inference_rynnbrain_loc(self, text: str, image: Union[List[str], str], task: str = "general"):
        image, indices, text = reformat_image_question(image, text)
        text = text.rsplit('?', 1)[0] # discatd text after the last ?

        answer_text = self.inference_rynnbrain_cog(text, image, task="select_frame")

        match = list(re.finditer(r"Picture *(\d+)", answer_text))
        if match:
            selected_id = int(match[-1].group(1))
            selected_id = min(indices, key=lambda x: abs(x - selected_id))
        else:
            selected_id = indices[-1] # select the last frame
            print('Select the last frames: ', selected_id)
        
        selected_image = [image[list(indices).index(selected_id)]]
        
        answer_text = self.inference_rynnbrain_cog(text, selected_image, task=task)
        
        answer_last = answer_text.splitlines()[-1] if answer_text and answer_text.splitlines() else answer_text


        # calcualte the actual image input resoluiton
        width, height = Image.open(image[list(indices).index(selected_id)]).size
        input_height, input_width = smart_resize(height, width, factor=28, min_pixels=MIN_PIXELS, max_pixels=MAX_PIXELS)

        if task == "referring":
            box_pattern_float = r'\[\s*(\d+\.?\d*)\s*,\s*(\d+\.?\d*)\s*,\s*(\d+\.?\d*)\s*,\s*(\d+\.?\d*)\s*\]'
            match = re.search(box_pattern_float, answer_last)
            if match:
                x1, y1, x2, y2= match.groups()
                points = [
                        (round(float(x1)*1000/input_width), round(float(y1)*1000/input_height)), 
                        (round(float(x2)*1000/input_width), round(float(y2)*1000/input_height)),
                ]
            else:
                points = []
        elif task in ["trajectory", "affordance", "area"]:
            point_pattern_float = r'(\d+\.?\d*)\s*,\s*(\d+\.?\d*)'
            pred_points = re.findall(point_pattern_float, answer_last)
            points = [(round(float(x)*1000/input_width), round(float(y)*1000/input_height)) for x, y in pred_points] #normalize to the actual input size

        answer_text = json.dumps(
            {
                'frame_idx': int(selected_id),
                'outputs': points,
                'raw outpus': answer_text
            }
        )
        # print('final answer', answer_text)

        return answer_text

[kding1]

Thank you so much @ZMHH-H, now the localization results are fully reproduced, and got even better results after updating the scaling logic: Qwen2.5-VL outputs coordinates in pixel space of the resized input image then rescales these to 0-1000, which is not needed for Qwen3-VL and Cosmos-Reason2, since both output 0-1000 normalized.
Is there anything special to handle cognition side of benchmarks for open-source models? I see roughly 5pt gap for RynnBrain-Spatial.

[ZMHH-H]

There are no special points for the evaluation of cognition tasks. You can try adjusting the task prompt and input resolution for each model.