[Bug Report] Fix for conversation history logic (verified) & Persistent Bbox grounding issues

[gorgingbunny-stack]

I have been testing the inference_demo.py (specifically on Visual Probe tasks) and identified two major issues.

1. Logic Error in Conversation History (Fix Verified)

The Problem: The original script causes the model to get stuck in an infinite loop or fail to generate a final . This is caused by two factors in the while loop:
a. Missing Context: The model's generated response_message (Thought/Action) is not appended to the chat history.
b. Wrong Role: The code execution result is appended with "role": "assistant", causing the model to treat the observation as its own past hallucination rather than new feedback.

The Fix: I modified the code to:
a. Append the response_message to chat_message before tool execution.
b. Change the execution result message role from "assistant" to "user".

Result: After these changes, the agent correctly recognizes previous actions, stops repeating itself, and successfully generates an .

2. Persistent Visual Grounding / Bbox Accuracy Issues

The Problem: Even with the logic fix above, the model frequently fails to crop the correct Region of Interest (ROI) in the first turn. The predicted bounding boxes are often significantly offset from the target.

Debugging Attempts: I suspected maybe_resize_image was downscaling the image too aggressively.

Result: I disabled maybe_resize_image (passing the original full-res image), but the Bbox accuracy did not improve. The coordinates remain inaccurate.

Questions: Since image resizing is not the cause, could you clarify the expected input format for coordinate generation?
a. Coordinate Space: Does the model output coordinates in a normalized 0-1000 space (requiring conversion) or absolute pixel values? The current prompt/script seems to assume pixel values.
b. Prompt Engineering: Is there a specific prompt trigger (e.g., asking for image.size first) required to ground the model before it generates crop coordinates?

Looking forward to your advice on improving the grounding accuracy.

[JaaackHongggg]

Hi, if you can raise a pr about the bug fix, I will appreciate it very much.

For the incorrect crop image problem, this is normal because we do not supervise the model's cropping results during the RL process, so it is reasonable for model to produce cropping errors. maybe_resize_image is only intended to prevent the cropped images from being too small or too large, and it has nothing to do with the cropping coordinates themselves. This issue is irrelevant to coordinate normalization (ranging from 0 to 1000) or prompts; however, you can attempt to resolve this problem by optimizing the prompts.