clicked to wrong location when using "Choose File" button in iOS simulator Safari browser.

[wzfukui]

Describe the bug
I'm using this mcp tool for iOS appication tesing. Mostly, it works well, Thanks!

I found some strange bug:

When I ask mcp to navigate to https://tmpfile.link to upload a file. This tool can find and click the "Choose file" button. After that, safari popup 3 buttons

• Photo Libray
• Camera or Record
• Choose File.

The MCP tool can get there buttons and knows to click the third one "Choose File", but,it finally clicked "Camera or Record" button :(

Configuration (please complete the following information):

• Agent: [Cherry Studio]
• OS: [macOS]
• Device used: [iOS Simulator]
• Device version: [18.5]
• Device model: [iPhone 16 Pro Max]

To Reproduce
Steps to reproduce the behavior:

1. Use prompt 'safari访问https://tmpfile.link ,上传一个本地的文本文件，得到临时下载文件，然后把临时地址url发给我。'
2. Then do 'wait for running'
3. The mcp server finally helps to click a wrong button and activate iOS Camear app.

Expected behavior
The mcp should help to click the right button to popup system file selection menu

Screenshots
If applicable, add screenshots to help explain your problem.

MCP running logs:

mobile-mcp : mobile_list_elements_on_screen

{
  "params": {
    "noParams": {}
  },
  "response": {
    "content": [
      {
        "type": "text",
        "text": "Found these elements on screen: [{\"type\":\"StaticText\",\"label\":\"tfLink\",\"name\":\"tfLink\",\"value\":\"tfLink\",\"identifier\":null,\"coordinates\":{\"x\":32,\"y\":78,\"width\":65,\"height\":30}},{\"type\":\"StaticText\",\"label\":\"EN\",\"name\":\"EN\",\"value\":\"EN\",\"identifier\":null,\"coordinates\":{\"x\":300,\"y\":83,\"width\":22,\"height\":20}},{\"type\":\"StaticText\",\"label\":\"中文\",\"name\":\"中文\",\"value\":\"中文\",\"identifier\":null,\"coordinates\":{\"x\":367,\"y\":83,\"width\":31,\"height\":20}},{\"type\":\"StaticText\",\"label\":\"Free Temporary File Sharing Service - Fast & Secure File Transfer\",\"name\":\"Free Temporary File Sharing Service - Fast & Secure File Transfer\",\"value\":\"Free Temporary File Sharing Service - Fast & Secure File Transfer\",\"identifier\":null,\"coordinates\":{\"x\":16,\"y\":205,\"width\":408,\"height\":148}},{\"type\":\"StaticText\",\"label\":\"Upload files (up to 100MB) and get temporary download links with global CDN acceleration\",\"name\":\"Upload files (up to 100MB) and get temporary download links with global CDN acceleration\",\"value\":\"Upload files (up to 100MB) and get temporary download links with global CDN acceleration\",\"identifier\":null,\"coordinates\":{\"x\":40,\"y\":369,\"width\":360,\"height\":45}},{\"type\":\"StaticText\",\"label\":\"✨ NEW: Generate QR codes for easy mobile downloads!\",\"name\":\"✨ NEW: Generate QR codes for easy mobile downloads!\",\"value\":\"✨ NEW: Generate QR codes for easy mobile downloads!\",\"identifier\":null,\"coordinates\":{\"x\":53,\"y\":431,\"width\":334,\"height\":41}},{\"type\":\"StaticText\",\"label\":\"Note: Anonymous uploads are automatically deleted after 7 days to protect your privacy\",\"name\":\"Note: Anonymous uploads are automatically deleted after 7 days to protect your privacy\",\"value\":\"Note: Anonymous uploads are automatically deleted after 7 days to protect your privacy\",\"identifier\":null,\"coordinates\":{\"x\":24,\"y\":487,\"width\":392,\"height\":37}},{\"type\":\"StaticText\",\"label\":\"Drag & Drop your file here or click Choose File\",\"name\":\"Drag & Drop your file here or click Choose File\",\"value\":\"Drag & Drop your file here or click Choose File\",\"identifier\":null,\"coordinates\":{\"x\":39,\"y\":637,\"width\":362,\"height\":23}},{\"type\":\"StaticText\",\"label\":\"Choose File\",\"name\":\"Choose File\",\"value\":\"Choose File\",\"identifier\":null,\"coordinates\":{\"x\":124,\"y\":689,\"width\":86,\"height\":21}},{\"type\":\"Button\",\"label\":\"QRCode\",\"name\":\"QRCode\",\"value\":null,\"identifier\":null,\"coordinates\":{\"x\":249,\"y\":679,\"width\":91,\"height\":42}},{\"type\":\"Button\",\"label\":\"Choose File\",\"name\":\"Choose File\",\"value\":null,\"identifier\":null,\"coordinates\":{\"x\":16,\"y\":676,\"width\":408,\"height\":10}},{\"type\":\"Button\",\"label\":\"页面菜单\",\"name\":\"PageFormatMenuButton\",\"value\":null,\"identifier\":\"PageFormatMenuButton\",\"coordinates\":{\"x\":36,\"y\":830,\"width\":51,\"height\":45}},{\"type\":\"TextField\",\"label\":\"地址\",\"name\":\"TabBarItemTitle\",\"value\":\"‎tmpfile.link\",\"identifier\":\"TabBarItemTitle\",\"coordinates\":{\"x\":178,\"y\":842,\"width\":84,\"height\":21}},{\"type\":\"Button\",\"label\":\"刷新\",\"name\":\"ReloadButton\",\"value\":null,\"identifier\":\"ReloadButton\",\"coordinates\":{\"x\":373,\"y\":842,\"width\":18,\"height\":21}},{\"type\":\"Button\",\"label\":\"返回\",\"name\":\"BackButton\",\"value\":null,\"identifier\":\"BackButton\",\"coordinates\":{\"x\":0,\"y\":878,\"width\":60,\"height\":44}},{\"type\":\"Button\",\"label\":\"前进\",\"name\":\"ForwardButton\",\"value\":null,\"identifier\":\"ForwardButton\",\"coordinates\":{\"x\":94,\"y\":878,\"width\":60,\"height\":44}},{\"type\":\"Button\",\"label\":\"共享\",\"name\":\"ShareButton\",\"value\":null,\"identifier\":\"ShareButton\",\"coordinates\":{\"x\":186,\"y\":878,\"width\":65,\"height\":44}},{\"type\":\"Button\",\"label\":\"显示书签\",\"name\":\"SidebarButton\",\"value\":null,\"identifier\":\"SidebarButton\",\"coordinates\":{\"x\":277,\"y\":878,\"width\":70,\"height\":44}},{\"type\":\"Button\",\"label\":\"标签页\",\"name\":\"TabOverviewButton\",\"value\":null,\"identifier\":\"TabOverviewButton\",\"coordinates\":{\"x\":372,\"y\":878,\"width\":68,\"height\":44}}]"
      }
    ]
  }
}

mobile-mcp : mobile_click_on_screen_at_coordinates

{
  "params": {
    "x": 200,
    "y": 680
  },
  "response": {
    "content": [
      {
        "type": "text",
        "text": "Clicked on screen at coordinates: 200, 680"
      }
    ]
  }
}

mobile-mcp : mobile_list_elements_on_screen

{
"params": {
"x": 42,
"y": 609
},
"response": {
"content": [
{
"type": "text",
"text": "Clicked on screen at coordinates: 42, 609"
}
]
}
}

mobile-mcp : mobile_list_elements_on_screen (We are now in Camera interface.)

{
  "params": {
    "noParams": {}
  },
  "response": {
    "content": [
      {
        "type": "text",
        "text": "Found these elements on screen: [{\"type\":\"StaticText\",\"label\":\"tfLink\",\"name\":\"tfLink\",\"value\":\"tfLink\",\"identifier\":null,\"coordinates\":{\"x\":32,\"y\":78,\"width\":65,\"height\":30}},{\"type\":\"StaticText\",\"label\":\"EN\",\"name\":\"EN\",\"value\":\"EN\",\"identifier\":null,\"coordinates\":{\"x\":300,\"y\":83,\"width\":22,\"height\":20}},{\"type\":\"StaticText\",\"label\":\"中文\",\"name\":\"中文\",\"value\":\"中文\",\"identifier\":null,\"coordinates\":{\"x\":367,\"y\":83,\"width\":31,\"height\":20}},{\"type\":\"StaticText\",\"label\":\"Free Temporary File Sharing Service - Fast & Secure File Transfer\",\"name\":\"Free Temporary File Sharing Service - Fast & Secure File Transfer\",\"value\":\"Free Temporary File Sharing Service - Fast & Secure File Transfer\",\"identifier\":null,\"coordinates\":{\"x\":16,\"y\":205,\"width\":408,\"height\":148}},{\"type\":\"StaticText\",\"label\":\"Upload files (up to 100MB) and get temporary download links with global CDN acceleration\",\"name\":\"Upload files (up to 100MB) and get temporary download links with global CDN acceleration\",\"value\":\"Upload files (up to 100MB) and get temporary download links with global CDN acceleration\",\"identifier\":null,\"coordinates\":{\"x\":40,\"y\":369,\"width\":360,\"height\":45}},{\"type\":\"StaticText\",\"label\":\"✨ NEW: Generate QR codes for easy mobile downloads!\",\"name\":\"✨ NEW: Generate QR codes for easy mobile downloads!\",\"value\":\"✨ NEW: Generate QR codes for easy mobile downloads!\",\"identifier\":null,\"coordinates\":{\"x\":53,\"y\":431,\"width\":334,\"height\":41}},{\"type\":\"StaticText\",\"label\":\"Note: Anonymous uploads are automatically deleted after 7 days to protect your privacy\",\"name\":\"Note: Anonymous uploads are automatically deleted after 7 days to protect your privacy\",\"value\":\"Note: Anonymous uploads are automatically deleted after 7 days to protect your privacy\",\"identifier\":null,\"coordinates\":{\"x\":24,\"y\":487,\"width\":392,\"height\":37}},{\"type\":\"StaticText\",\"label\":\"Drag & Drop your file here or click Choose File\",\"name\":\"Drag & Drop your file here or click Choose File\",\"value\":\"Drag & Drop your file here or click Choose File\",\"identifier\":null,\"coordinates\":{\"x\":39,\"y\":637,\"width\":362,\"height\":23}},{\"type\":\"StaticText\",\"label\":\"Choose File\",\"name\":\"Choose File\",\"value\":\"Choose File\",\"identifier\":null,\"coordinates\":{\"x\":124,\"y\":689,\"width\":86,\"height\":21}},{\"type\":\"Button\",\"label\":\"QRCode\",\"name\":\"QRCode\",\"value\":null,\"identifier\":null,\"coordinates\":{\"x\":249,\"y\":679,\"width\":91,\"height\":42}},{\"type\":\"Button\",\"label\":\"Choose File\",\"name\":\"Choose File\",\"value\":null,\"identifier\":null,\"coordinates\":{\"x\":16,\"y\":676,\"width\":408,\"height\":10}},{\"type\":\"Button\",\"label\":\"闪光灯\",\"name\":\"FlashButton\",\"value\":\"自动\",\"identifier\":\"FlashButton\",\"coordinates\":{\"x\":0,\"y\":49,\"width\":44,\"height\":42}},{\"type\":\"Image\",\"label\":null,\"name\":\"CAMFlashButton\",\"value\":null,\"identifier\":\"CAMFlashButton\",\"coordinates\":{\"x\":11,\"y\":59,\"width\":22,\"height\":22}},{\"type\":\"Button\",\"label\":\"拍照\",\"name\":\"PhotoCapture\",\"value\":null,\"identifier\":\"PhotoCapture\",\"coordinates\":{\"x\":160,\"y\":796,\"width\":120,\"height\":77}},{\"type\":\"Button\",\"label\":\"相机选取器\",\"name\":\"FrontBackFacingCameraChooser\",\"value\":\"后置\",\"identifier\":\"FrontBackFacingCameraChooser\",\"coordinates\":{\"x\":361,\"y\":797,\"width\":70,\"height\":70}},{\"type\":\"Button\",\"label\":\"取消\",\"name\":\"取消\",\"value\":null,\"identifier\":null,\"coordinates\":{\"x\":15,\"y\":815,\"width\":36,\"height\":34}},{\"type\":\"StaticText\",\"label\":\"取消\",\"name\":\"取消\",\"value\":\"取消\",\"identifier\":null,\"coordinates\":{\"x\":15,\"y\":821,\"width\":36,\"height\":22}}]"
      }
    ]
  }
}

More info:
I'm using OpenAI o1-mini model.

[wzfukui]

I found the same bug, when I ask mcp to find a friend's contact info from iOS Contacts App. It clicked to a wrong place which did not pop up the user's profile.

[gmegidish]

I'll take a look today and try to reproduce. If it uses list_elements_on_screen, then it should be able to click properly on things. Right now there's no web support, so it cannot access the DOM on the html, and can only use screenshot, and using screenshots for coordinates highly depends on the abilities of the model. Thank you for the detailed bug report!