Question regarding "Adaptive Thinking"

[RubyNg]

Hi there,

I have a question about the "adaptive thinking" capability. The paper mentions that the model should solve problems directly when tools are unnecessary. However, in my experiments, the model invokes the Code tool 100% of the time, even for very simple tasks.

I'm using the system prompt from the Appendix but removed the "Search" section. I tested it on V-Star, V-Probe, and about 150 simple QA pairs based on SA-1B. What I found is:

1. The model always chooses to write code.
2. Even when the model gets the correct answer during the "thinking" phase, it still proceeds to generate code.
3. Sometimes the code is meaningless, like just running plt.imshow(image_1) to show the image.

I'm wondering if removing the Search tool caused some bias, or if I need to adjust the prompt to make it answer directly?

Thanks!

[YuanzhengWu999]

Hello, I've also encountered the same problem. I'm using the prompt from VLMEvalkit, and I've noticed that almost every first "thinking" step produces the same answer as the final answer. Have you experienced this phenomenon as well?

[YuanzhengWu999]

Another problem is that during inference, almost all of the cropped images are incorrect; there are no correct ones. Even under these circumstances, the answer is still correct. I think this might be related to the fact that the answer appears during the first step of the thinking process.

[RubyNg]

Another problem is that during inference, almost all of the cropped images are incorrect; there are no correct ones. Even under these circumstances, the answer is still correct. I think this might be related to the fact that the answer appears during the first step of the thinking process.

Yeah, I observed the same phenomenon.
It seems the model nails the correct answer in the very first "thinking" turn. The subsequent code execution feels like it's just acting out of inertia (or just going through the motions). It doesn't actually seem to rely on the tool's output; it just blindly follows its initial thought and repeats the answer, regardless of whether the code works or makes sense.

[JaaackHongggg]

Regarding the issue of adaptive thinking, we removed the search-related content during testing (for non-search data). As you mentioned, the model basically invokes tools for the v* test set, which is largely consistent with the proportion presented in our paper. We see no issues with this result. You may try testing with other datasets.

As for the issues of meaningless code generation and the model arriving at answers in the first round, these are consequences of RL optimization. We only utilized outcome reward and did not incorporate any form of process reward, so such phenomena are entirely normal and reasonable. Given that DeepEyesV2 is merely a preliminary exploration, we believe the community could investigate ways to optimize the model's reasoning process during RL training.

Finally, regarding the issue of incorrectly cropped images, we think this is also caused by RL optimization, rather than a problem with the sandbox or the code itself.

[RubyNg]

Thanks for the detailed explanation! The point about outcome-based RL optimization makes sense. But I'm still a bit confused about MathVista. The paper mentions a 56.9% tool usage rate, but I'm still seeing 100% in my run. It often wraps simple answers in code like print(1000).

Just to make sure I didn't mess up the config, here is my setup:

1. Model: honglyhly/DeepEyesV2_7B_1031
2. Prompt: The one provided in the Appendix.
3. Counting: I count any code tag execution as a tool call.

Let me know if I missed anything!