Skip to content

Commit f5a4b42

Browse files
authored
feat(browserTool): Implement hover action (#2368)
* Implement hover action for the browser action tool * Update snapshots
1 parent fff8fdd commit f5a4b42

File tree

3 files changed

+17
-4
lines changed

3 files changed

+17
-4
lines changed

src/core/prompts/__tests__/__snapshots__/system.test.ts.snap

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2714,6 +2714,9 @@ Parameters:
27142714
* launch: Launch a new Puppeteer-controlled browser instance at the specified URL. This **must always be the first action**.
27152715
- Use with the \`url\` parameter to provide the URL.
27162716
- Ensure the URL is valid and includes the appropriate protocol (e.g. http://localhost:3000/page, file:///path/to/file.html, etc.)
2717+
* hover: Move the cursor to a specific x,y coordinate.
2718+
- Use with the \`coordinate\` parameter to specify the location.
2719+
- Always move to the center of an element (icon, button, link, etc.) based on coordinates derived from a screenshot.
27172720
* click: Click at a specific x,y coordinate.
27182721
- Use with the \`coordinate\` parameter to specify the location.
27192722
- Always click in the center of an element (icon, button, link, etc.) based on coordinates derived from a screenshot.
@@ -2727,7 +2730,7 @@ Parameters:
27272730
- Example: \`<action>close</action>\`
27282731
- url: (optional) Use this for providing the URL for the \`launch\` action.
27292732
* Example: <url>https://example.com</url>
2730-
- coordinate: (optional) The X and Y coordinates for the \`click\` action. Coordinates should be within the **900x600** resolution.
2733+
- coordinate: (optional) The X and Y coordinates for the \`click\` and \`hover\` actions. Coordinates should be within the **900x600** resolution.
27312734
* Example: <coordinate>450,300</coordinate>
27322735
- size: (optional) The width and height for the \`resize\` action.
27332736
* Example: <size>1280,720</size>
@@ -3629,6 +3632,9 @@ Parameters:
36293632
* launch: Launch a new Puppeteer-controlled browser instance at the specified URL. This **must always be the first action**.
36303633
- Use with the \`url\` parameter to provide the URL.
36313634
- Ensure the URL is valid and includes the appropriate protocol (e.g. http://localhost:3000/page, file:///path/to/file.html, etc.)
3635+
* hover: Move the cursor to a specific x,y coordinate.
3636+
- Use with the \`coordinate\` parameter to specify the location.
3637+
- Always move to the center of an element (icon, button, link, etc.) based on coordinates derived from a screenshot.
36323638
* click: Click at a specific x,y coordinate.
36333639
- Use with the \`coordinate\` parameter to specify the location.
36343640
- Always click in the center of an element (icon, button, link, etc.) based on coordinates derived from a screenshot.
@@ -3642,7 +3648,7 @@ Parameters:
36423648
- Example: \`<action>close</action>\`
36433649
- url: (optional) Use this for providing the URL for the \`launch\` action.
36443650
* Example: <url>https://example.com</url>
3645-
- coordinate: (optional) The X and Y coordinates for the \`click\` action. Coordinates should be within the **1280x800** resolution.
3651+
- coordinate: (optional) The X and Y coordinates for the \`click\` and \`hover\` actions. Coordinates should be within the **1280x800** resolution.
36463652
* Example: <coordinate>450,300</coordinate>
36473653
- size: (optional) The width and height for the \`resize\` action.
36483654
* Example: <size>1280,720</size>

src/core/prompts/tools/browser-action.ts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@ Parameters:
1515
* launch: Launch a new Puppeteer-controlled browser instance at the specified URL. This **must always be the first action**.
1616
- Use with the \`url\` parameter to provide the URL.
1717
- Ensure the URL is valid and includes the appropriate protocol (e.g. http://localhost:3000/page, file:///path/to/file.html, etc.)
18+
* hover: Move the cursor to a specific x,y coordinate.
19+
- Use with the \`coordinate\` parameter to specify the location.
20+
- Always move to the center of an element (icon, button, link, etc.) based on coordinates derived from a screenshot.
1821
* click: Click at a specific x,y coordinate.
1922
- Use with the \`coordinate\` parameter to specify the location.
2023
- Always click in the center of an element (icon, button, link, etc.) based on coordinates derived from a screenshot.
@@ -28,7 +31,7 @@ Parameters:
2831
- Example: \`<action>close</action>\`
2932
- url: (optional) Use this for providing the URL for the \`launch\` action.
3033
* Example: <url>https://example.com</url>
31-
- coordinate: (optional) The X and Y coordinates for the \`click\` action. Coordinates should be within the **${args.browserViewportSize}** resolution.
34+
- coordinate: (optional) The X and Y coordinates for the \`click\` and \`hover\` actions. Coordinates should be within the **${args.browserViewportSize}** resolution.
3235
* Example: <coordinate>450,300</coordinate>
3336
- size: (optional) The width and height for the \`resize\` action.
3437
* Example: <size>1280,720</size>

src/core/tools/browserActionTool.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ export async function browserActionTool(
7373
await cline.browserSession.launchBrowser()
7474
browserActionResult = await cline.browserSession.navigateToUrl(url)
7575
} else {
76-
if (action === "click") {
76+
if (action === "click" || action === "hover") {
7777
if (!coordinate) {
7878
cline.consecutiveMistakeCount++
7979
pushToolResult(await cline.sayAndCreateMissingParamError("browser_action", "coordinate"))
@@ -112,6 +112,9 @@ export async function browserActionTool(
112112
case "click":
113113
browserActionResult = await cline.browserSession.click(coordinate!)
114114
break
115+
case "hover":
116+
browserActionResult = await cline.browserSession.hover(coordinate!)
117+
break
115118
case "type":
116119
browserActionResult = await cline.browserSession.type(text!)
117120
break
@@ -133,6 +136,7 @@ export async function browserActionTool(
133136
switch (action) {
134137
case "launch":
135138
case "click":
139+
case "hover":
136140
case "type":
137141
case "scroll_down":
138142
case "scroll_up":

0 commit comments

Comments
 (0)