Agent hallucinates image content

[SmartManoj]

App Version

Version: 3.20.3 (23bbad0)

API Provider

Google Vertex AI

Model Used

gemini-2.5-pro-preview-05-06

🔁 Steps to Reproduce

Prompt: describe image.png

💥 Outcome Summary

Current:

Expected:

Should read the file and describe.

Claude model uses a browser tool and describes the image due to its support computer-use?

📄 Relevant Logs or Errors (Optional)

[elianiva]

Which model are you using? Doesn't seem to happen with Sonnet 4. It correctly reads the file before coming up with answers.
It could be that the model you're using is not very good at following instructions.

[SmartManoj]

Which model are you using?

Model Used

gemini-2.5-pro-preview-05-06

---

It correctly reads the file before coming up with answers.

Nope.

detected from the name.

cannot

[elianiva]

Nope, it did read the file otherwise this UI wouldn't show up, but since it's a binary file and read_file is designed for text, it failed, but it did try to read the file, unlike your case which completely hallucinates.

[SmartManoj]

It correctly reads the file before coming up with answers.

It reads, but not correctly.

[SmartManoj]

Why restrict browsers to computer-use models only?

Roo-Code/src/core/prompts/tools/browser-action.ts

Lines 3 to 6 in 829fe8f

	export function getBrowserActionDescription(args: ToolArgs): string | undefined {
	if (!args.supportsComputerUse) {
	return undefined
	}

[elianiva]

It reads, but not correctly.

As expected, I mean, what do you expect from a tool that is supposed to read a file, but you ask it to read a binary? The point here is that it uses the tool before coming up with an answer, despite it being failed. It's still better than completely hallucinating because if the tool supports it then it will succeed.

Why restrict browsers to computer-use models only?

Simply because models that don't support computer-use cannot directly control a browser. You can get around this by using browser MCP or some sort.