Add mouse pointer visualization to screenshots in BrowserEnv #347
Conversation
- Introduced `show_mouse_pointer` parameter to control mouse pointer display. - Implemented methods to extract mouse coordinates from actions and overlay pointer on screenshots.
![korbit-ai[bot]](https://avatars.githubusercontent.com/in/322216?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by Korbit AI
Korbit automatically attempts to detect when you fix issues in new commits.
| Category | Issue | Status |
|---|---|---|
 |
Mouse pointer boundary check missing ▹ view |
Files scanned
| File Path | Reviewed |
|---|---|
| browsergym/core/src/browsergym/core/observation.py | ✅ |
| browsergym/core/src/browsergym/core/env.py | ✅ |
Explore our documentation to understand the languages and file types we support and the files we ignore.
Check out our docs on how you can make Korbit work best for you and your team.
| pointer_shape = [ | ||
| (x, y), | ||
| (x + pointer_size, y + pointer_size // 2), | ||
| (x + pointer_size // 2, y + pointer_size // 2), | ||
| (x + pointer_size // 2, y + pointer_size), | ||
| ] |
There was a problem hiding this comment.
Mouse pointer boundary check missing 
Tell me more
What is the issue?
The mouse pointer visualization could appear outside the screenshot boundaries when coordinates are near the edges, potentially causing IndexError or visual artifacts.
Why this matters
This could lead to runtime errors or partial/incorrect pointer visualization when the agent interacts with elements near screen edges.
Suggested change ∙ Feature Preview
Add boundary checks before drawing the pointer to ensure coordinates stay within image dimensions:
def add_mouse_pointer_to_screenshot(screenshot: Image.Image, action: str) -> Image.Image:
coords = extract_mouse_coords_from_action(action)
if not coords:
return screenshot
if isinstance(screenshot, np.ndarray):
pil_image = Image.fromarray(screenshot)
else:
pil_image = screenshot
width, height = pil_image.size
x, y = coords
pointer_size = 20
# Constrain pointer coordinates to image boundaries
x = max(0, min(x, width - pointer_size))
y = max(0, min(y, height - pointer_size))
overlay = pil_image.convert("RGBA").copy()
draw = ImageDraw.Draw(overlay)
pointer_shape = [
(x, y),
(x + pointer_size, y + pointer_size // 2),
(x + pointer_size // 2, y + pointer_size // 2),
(x + pointer_size // 2, y + pointer_size),
]
draw.polygon(pointer_shape, fill=(0, 0, 0, 128))
result_image = Image.alpha_composite(pil_image.convert("RGBA"), overlay)
return np.array(result_image.convert("RGB"))Provide feedback to improve future suggestions
💬 Looking for more details? Reply to this comment to chat with Korbit.
|
Deprecated in favour of #348 |
show_mouse_pointerparameter to control mouse pointer display.Description by Korbit AI
What change is being made?
Add mouse pointer visualization to screenshots in the
BrowserEnvclass to better illustrate click actions, including the implementation of new functions for coordinate extraction and mouse pointer overlay and the addition of corresponding tests.Why are these changes being made?
These changes enhance visual clarity in automation processes by allowing users to see where mouse click actions occur in screenshots, improving debug capability and comprehension of action sequences. This solution leverages the existing screenshot functionality while adding a minimal overhead through an optional boolean flag,
show_mouse_pointer, set to True by default, with comprehensive tests ensuring reliability and correctness.