A tool to "Select an element in the page to inspect it"

[jjkavalam]

Is your feature request related to a problem? Please describe.

Select an element in the page to inspect it could arguably be the most used Chrome Dev Tools "tool".

This allows the (human) programmer to focus on just that part of the DOM he is interested in. On the other hand, most tools we put in the hands of AI agents today always give back the full DOM. The AI is then forced to first filter out all the irrelevant parts leading to the context window filling up extremely fast, and the agent loop slowing down a lot.

Describe the solution you'd like

If this MCP server could implement something like Select an element in the page to inspect it, we can give the AI agents the same efficient method of inspecting the DOM.

I would imagine the AI can send an annotated screenshot as input; with a suitably coloured bounding box to indicate the region of the page the AI hopes to inspect.

Describe alternatives you've considered

Alternative solutions include a tool that allows the AI to specify a CSS selector, and the tool will return the DOM rooted at the corresponding element (instead of the whole page). But this does mean the selector be identified somehow first, which may not be easy. Working with a screenshot and placing a bounding box on the region of interest could be markedly more easy in certain cases.

Additional context

The Select an element in the page to inspect it tool in Chrome Dev Tools:

[natorion]

Do you mean that you can select an element/artifact in Chrome DevTools and use it as context in your MCP client?

[jjkavalam]

Do you mean that you can select an element/artifact in Chrome DevTools and use it as context in your MCP client?

@natorion No, i didn't mean, me, as a human user selecting. I meant giving the AI agent the ability to select DOM elements by position (as opposed to by a css selector).

[natorion]

Understood, so the human would then use the selected element to continue debugging on their own in DevTools?

[jjkavalam]

Understood, so the human would then use the selected element to continue debugging on their own in DevTools?

@natorion No, the idea is, this way, the AI agent can use its vision to select the part of the DOM it is interested in.

What we have today is the AI agent first reading the DOM structure of the entire page (this can be massive and could use a lot of tokens very quickly) and then trying to figure out the subtree of interest.

But, that is not how a human would debug typically. Because the human can use his vision and the Select an element in the page to inspect it in ChromeDevTools to quickly inspect just the part of the page he is interested in. The request here is to give the AI agents the same capabilities so that they can be more effective (in autonomous operation).

[jjkavalam]

@natorion I see the confusion here. When i say, Select an element in the page to inspect it tool, i do NOT mean the AI agent interacting with the Chrome Dev Tools front end to select a DOM element for the human.

What i mean is: What if we gave the AI agent a "virtual mouse pointer", if you will, so that it can use its vision to select DOM elements.

Hope it clarifies the intent here.

[natorion]

Hmmm, in what prompts would you expect your MCP client make use of let's say select_element_visually(imageData, position) function?

[jjkavalam]

Hmmm, in what prompts would you expect your MCP client make use of let's say select_element_visually(imageData, position) function?

Consider,

The "Delete" button is not aligned correctly on http://localhost:3000/my-complex-page. Could you fix it?

The agent is likely to first attempt to "look" at the page, issuing some command that returns a dump of the page's DOM. But this could result in a lot of token usage (and potentially the context window filling up).

So, instead if i could,

The "Delete" button is not aligned correctly on http://localhost:3000/my-complex-page. Could you fix it?

The page is complex and you might want to select the button's container visually to inspect its DOM.

With this, the agent will hopefully take the cue and "look" at the page, issuing a screenshot command, leveraging its vision capability. And then following it up with "select_element_visually".

To summarise, in cases where the page is large in terms of the number of DOM nodes, vision based selection, could give the agent a method that can scale irrespective of the DOM level complexity.

[natorion]

In this case you still would need to match the (marked) screenshot to the actual page, which could very well have changed visually in the meantime (consider elements slightly shading or flickering, or a video element, or a autoscrollig carousel.

The IDs we generated or the A11y tree should be more resilient.

Assuming the visual representation of the page is relevant and take_snapshot is not enough it might be more useful to be a good MCP server and help the client to make the decision: Add another parameter to take_screenshot called "overlayUIDs", which overlays UIDs visually like a map over the screenshot. This way the MCP client has the information where stuff on the page is located. I have no clue though, if MCP clients nowadays are smart enough to figure it out and what resolution you would need to provide a good enough map.

[jjkavalam]

Actually, the overlay idea is a good alternative — it reminds me of the well-known Set-of-Marks prompting method.

Staying on topic, though: I hadn't really thought through how the client could communicate to the Chrome DevTools MCP which element it wants to select.

I'm curious how the "Select an element in the page to inspect it" feature in Chrome DevTools actually works. What's involved in translating the mouse pointer's position into the corresponding DOM element?

My thinking is that, in all scenarios where a human benefits from inspecting the DOM with that tool, an AI agent could potentially benefit as well.

[natorion]

My thinking is that, in all scenarios where a human benefits from inspecting the DOM with that tool, an AI agent could potentially benefit as well.

I think this is an example of difference in behavior between humans and AI agents. A human is a very visual entity, that will react timely to something on a screen.
A typical MCP client will use a model that is not optimized for speed and image processing and will have a harder time figuring out stuff on the screen in a timely manner. Converting visual information to structured information that is less time-dependent (the map proposal above) would actually be a better input mechanism.

This is just mental gymnastics at this point though, no clue if its actually useful.

[jjkavalam]

@natorion Really appreciate you looking into this. Am glad I got a chance to explain my thinking too.

[natorion]

In this case you still would need to match the (marked) screenshot to the actual page, which could very well have changed visually in the meantime (consider elements slightly shading or flickering, or a video element, or a autoscrollig carousel.

The IDs we generated or the A11y tree should be more resilient.

Assuming the visual representation of the page is relevant and take_snapshot is not enough it might be more useful to be a good MCP server and help the client to make the decision: Add another parameter to take_screenshot called "overlayUIDs", which overlays UIDs visually like a map over the screenshot. This way the MCP client has the information where stuff on the page is located. I have no clue though, if MCP clients nowadays are smart enough to figure it out and what resolution you would need to provide a good enough map.

Let's collect feedback if something like this would be useful for more people.

[jformio]

I would just add my use case to this ticket as well - coming more from a design -, than engineering, perspective (knowledge wise).

Using a prompts like

Check my current selection in the browser, help me replace that with my current selection in Figma

or

Help me change the background color of the current selection in the browser

would help a lot where my CSS skills fails to help me convey the exact thing I want to change.

My current workaround is to use the MCP, make the changes in the DevTools with the help of the built in AI Assistant in DevTools, then ask Cursor to inspect the changes made and implement them. But this approach is a lot of back and forth, I would just like to work 100% in Cursor and use the MCPs for it instead.