[🐛 Bug]: [py] BiDi threads prevent interpreter shutdown

[cgoldberg]

Description

When using BiDi with remote Chrome webdriver, it is not possible to shutdown the Python interpreter.

For example, the attached code will quit the browser, but never finish executing (even if the WebSocket timeout elapses).

#16298 will kill the WebSocket threads and allow interpreter shutdown, but I'm not sure if this is a deeper issue that needs to be investigated.

Note: this works fine with remote Firefox and local Chrome sessions.

Reproducible Code

from selenium import webdriver

options = webdriver.ChromeOptions()
options.enable_bidi = True
driver = webdriver.Remote(options=options)
driver.browsing_context.navigate(context=driver.current_window_handle, url="https://example.com")
driver.quit()

Debugging Logs

DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:4444/session {'capabilities': {'firstMatch': [{}], 'alwaysMatch': {'browserName': 'chrome', 'pageLoadStrategy': <PageLoadStrategy.normal: 'normal'>, 'webSocketUrl': True, 'goog:chromeOptions': {'extensions': [], 'args': []}}}}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:4444
DEBUG:urllib3.connectionpool:http://127.0.0.1:4444 "POST /session HTTP/1.1" 200 0
DEBUG:selenium.webdriver.remote.remote_connection:Remote response: status=200 | data={
  "value": {
    "sessionId": "ddc64bef762d464e134371a3b790a5f0",
    "capabilities": {
      "acceptInsecureCerts": false,
      "browserName": "chrome",
      "browserVersion": "142.0.7444.176",
      "chrome": {
        "chromedriverVersion": "142.0.7444.175 (302067f14a4ea3f42001580e6101fa25ed343445-refs\u002fbranch-heads\u002f7444@{#2749})",
        "userDataDir": "C:\\Users\\COREY~1.GOL\\AppData\\Local\\Temp\\scoped_dir41712_85938306"
      },
      "fedcm:accounts": true,
      "goog:chromeOptions": {
        "debuggerAddress": "localhost:52633"
      },
      "goog:processID": 16716,
      "networkConnectionEnabled": false,
      "pageLoadStrategy": "normal",
      "platformName": "Windows 11",
      "proxy": {
      },
      "se:cdp": "ws:\u002f\u002f10.0.0.215:4444\u002fsession\u002fddc64bef762d464e134371a3b790a5f0\u002fse\u002fcdp",
      "se:cdpVersion": "142.0.7444.176",
      "se:gridWebSocketUrl": "ws:\u002f\u002flocalhost:17646\u002fsession\u002fddc64bef762d464e134371a3b790a5f0",
      "setWindowRect": true,
      "strictFileInteractability": false,
      "timeouts": {
        "implicit": 0,
        "pageLoad": 300000,
        "script": 30000
      },
      "unhandledPromptBehavior": "dismiss and notify",
      "webSocketUrl": "ws:\u002f\u002f10.0.0.215:4444\u002fsession\u002fddc64bef762d464e134371a3b790a5f0\u002fse\u002fbidi",
      "webauthn:extension:credBlob": true,
      "webauthn:extension:largeBlob": true,
      "webauthn:extension:minPinLength": true,
      "webauthn:extension:prf": true,
      "webauthn:virtualAuthenticators": true
    }
  }
} | headers=HTTPHeaderDict({'content-length': '1595', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
INFO:websocket:Websocket connected
DEBUG:selenium.webdriver.remote.remote_connection:GET http://127.0.0.1:4444/session/ddc64bef762d464e134371a3b790a5f0/window {}
DEBUG:urllib3.connectionpool:http://127.0.0.1:4444 "GET /session/ddc64bef762d464e134371a3b790a5f0/window HTTP/1.1" 200 0
DEBUG:selenium.webdriver.remote.remote_connection:Remote response: status=200 | data={"value":"1FE022C115A10ECABAF3A66EE73E6623"} | headers=HTTPHeaderDict({'content-length': '44', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
DEBUG:selenium.webdriver.remote.websocket_connection:-> {"method": "browsingContext.navigate", "params": {"context": "1FE022C115A10ECABAF3A66EE73E6623", "url": "https://example.com"}, "id": 1}
DEBUG:selenium.webdriver.remote.websocket_connection:<- {'id': 1, 'result': {'navigation': '591f36aa-9fbd-41a3-9a5d-3b6007750b8e', 'url': 'https://example.com/'}, 'type': 'success'}
DEBUG:selenium.webdriver.remote.remote_connection:DELETE http://127.0.0.1:4444/session/ddc64bef762d464e134371a3b790a5f0 {}
DEBUG:urllib3.connectionpool:http://127.0.0.1:4444 "DELETE /session/ddc64bef762d464e134371a3b790a5f0 HTTP/1.1" 200 0
DEBUG:selenium.webdriver.remote.remote_connection:Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'content-length': '14', 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request

[selenium-ci]

@cgoldberg, thank you for creating this issue. We will troubleshoot it as soon as we can.

Selenium Triage Team: remember to follow the Triage Guide

[cgoldberg]

@navin772 can you reproduce this? I haven't tried it on Linux yet, but this happens on Windows. I think it's a Chrome issue since Firefox works fine. I think we should merge #16298 to help with this.

[navin772]

@navin772 can you reproduce this?

Tried this on macOS. Can confirm it hangs with remote chrome but works fine with firefox.
Setting daemon=True works with Chrome to avoid this issue. We should merge it.

The error also blocks the grid and it needs to be restarted in case of chrome remote.

[cgoldberg]

Thanks. I just merged #16298, so that should fix this... but I'm not sure why the threads hang when talking to Chrome remote, so I hope this isn't just hiding another issue.