CDP server issues on MacOS (m2)

[addy999]

Running the nightly build and connecting via puppeteer from another terminal:

Terminal 1

./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222
> info(websocket): starting blocking worker to listen on 127.0.0.1:9222
> info(server): accepting new conn...
> info(server): client connected
> info(server): accepting new conn...
> error(handler): stream close error: error.BrokenPipe

Terminal 2:

In [1]: from pyppeteer import launch, connect

In [2]: b = await connect(browserURL="http://127.0.0.1:9222")

In [3]: page = await b.newPage()
---------------------------------------------------------------------------
NetworkError                              Traceback (most recent call last)
Cell In[3], line 1
----> 1 page = await b.newPage()

File ~/miniconda3/lib/python3.11/site-packages/pyppeteer/browser.py:202, in Browser.newPage(self)
    200 async def newPage(self) -> Page:
    201     """Make new page on this browser and return its object."""
--> 202     return await self._defaultContext.newPage()

File ~/miniconda3/lib/python3.11/site-packages/pyppeteer/browser.py:358, in BrowserContext.newPage(self)
    356 async def newPage(self) -> Page:
    357     """Create a new page in the browser context."""
--> 358     return await self._browser._createPageInContext(self._id)

File ~/miniconda3/lib/python3.11/site-packages/pyppeteer/browser.py:209, in Browser._createPageInContext(self, contextId)
    206 if contextId:
    207     options['browserContextId'] = contextId
--> 209 targetId = (await self._connection.send(
    210     'Target.createTarget', options)).get('targetId')
    211 target = self._targets.get(targetId)
    212 if target is None:

NetworkError: Protocol error Target.createTarget: Target closed.

[krichprollsch]

Hello @addy999
Thank you for your bug report.

Can you retry with --verbose option on ligthpanda please?

[krichprollsch]

Here is what I reproduce with the following gist https://gist.github.com/krichprollsch/f510c3c89a21dd70e406a9c79f3b3b06
@addy999 is it accurate? I'm not used to using python...

$ docker run --rm -ti --privileged lpd:issue339
//issue339.py:8: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(main())
Traceback (most recent call last):
  File "//issue339.py", line 8, in <module>
    asyncio.get_event_loop().run_until_complete(main())
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 720, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "//issue339.py", line 6, in main
    page = await b.newPage()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/pyppeteer/browser.py", line 202, in newPage
    return await self._defaultContext.newPage()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/pyppeteer/browser.py", line 358, in newPage
    return await self._browser._createPageInContext(self._id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/pyppeteer/browser.py", line 213, in _createPageInContext
    raise BrowserError('Failed to create target for page.')
pyppeteer.errors.BrowserError: Failed to create target for page.
debug(cli): Server opts: listening internally on /tmp/lightpanda...
info(websocket): starting blocking worker to listen on 127.0.0.1:9222
info(server): accepting new conn...
debug(websocket): (127.0.0.1:43946) connected
debug(websocket): (127.0.0.1:43952) connected
debug(handler): connected to Stream server
info(server): client connected
debug(websocket): (127.0.0.1:43952) connection successfully upgraded
debug(websocket): (127.0.0.1:43952) received text message
debug(cdp): Req > id 1, method target.getBrowserContexts
debug(cdp): Res > id 1, sessionID null, result cdp.target.getBrowserContexts.Resp{ .browserContextIds = {  } }
debug(websocket): (127.0.0.1:43952) received text message
debug(cdp): Req > id 2, method target.setDiscoverTargets
debug(cdp): Res > id 2, sessionID null, result null
debug(websocket): (127.0.0.1:43952) received text message
debug(cdp): Req > id 3, method target.createTarget
debug(cdp): Event > method Target.attachedToTarget, sessionID null
debug(cdp): Res > id 3, sessionID null, result cdp.target.createTarget { .targetId = 57356548460A8F29706A2ADF14316298 }
debug(websocket): (127.0.0.1:43952) connection closed: error.Closed
info(server): close cmd, closing conn...
debug(server): connection closed
info(server): accepting new conn...
debug(server): read has been canceled

[addy999]

Okay I did it with --verbose

debug(handler): connected to Stream server
info(server): client connected
debug(websocket): (127.0.0.1:61253) connection successfully upgraded
debug(websocket): (127.0.0.1:61253) received text message
debug(cdp): Req > id 1, method target.getBrowserContexts
debug(cdp): Res > id 1, sessionID null, result cdp.target.getBrowserContexts.Resp{ .browserContextIds = {  } }
debug(websocket): (127.0.0.1:61253) received text message
debug(cdp): Req > id 2, method target.setDiscoverTargets
debug(cdp): Res > id 2, sessionID null, result null
debug(websocket): (127.0.0.1:61253) received text message
debug(cdp): Req > id 3, method target.createTarget
debug(cdp): Event > method Target.attachedToTarget, sessionID null
debug(cdp): Res > id 3, sessionID null, result cdp.target.createTarget { .targetId = 57356548460A8F29706A2ADF14316298 }

Now I'm seeing a TegetError

----> 1 page = await b.newPage()

File ~/.pyenv/versions/3.10.10/lib/python3.10/site-packages/pyppeteer/browser.py:202, in Browser.newPage(self)
    200 async def newPage(self) -> Page:
    201     """Make new page on this browser and return its object."""
--> 202     return await self._defaultContext.newPage()

File ~/.pyenv/versions/3.10.10/lib/python3.10/site-packages/pyppeteer/browser.py:358, in BrowserContext.newPage(self)
    356 async def newPage(self) -> Page:
    357     """Create a new page in the browser context."""
--> 358     return await self._browser._createPageInContext(self._id)

File ~/.pyenv/versions/3.10.10/lib/python3.10/site-packages/pyppeteer/browser.py:213, in Browser._createPageInContext(self, contextId)
    211 target = self._targets.get(targetId)
    212 if target is None:
--> 213     raise BrowserError('Failed to create target for page.')
    214 if not await target._initializedPromise:
    215     raise BrowserError('Failed to create target for page.')

BrowserError: Failed to create target for page.

[krichprollsch]

ok thank you, I think I have what I need to start debugging.

[addy999]

Hi, any progress on this?

[krichprollsch]

Not really.
The error source is https://github.com/pyppeteer/pyppeteer/blob/7dc91ee5173d3836f77800a3774beeaf2b448c0e/pyppeteer/browser.py#L213

I tried using the code from #285 which is an effort to bring a better chromedp compatibility, but the result is the same.

[krichprollsch]

I enabled the log level to debug with:

import logging
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)

https://gist.github.com/krichprollsch/554c553a0d17bfe4473d291132b72a23#file-log-python-lpd

vs chrome https://gist.github.com/krichprollsch/554c553a0d17bfe4473d291132b72a23#file-log-python-chrome

[krichprollsch]

ok the event TargetCreated was missing. I added it in #346
Now the script stops on another issue: Target.sendMessageToTarget is unknown.

$ python3 issue339.py 
/home/pierre/wrk/issue339/issue339.py:12: DeprecationWarning: There is no current event loop
  asyncio.get_event_loop().run_until_complete(main())
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:pyppeteer.connection.Connection:SEND: {"id": 1, "method": "Target.getBrowserContexts", "params": {}}
DEBUG:websockets.client:= connection is CONNECTING
DEBUG:websockets.client:> GET / HTTP/1.1
DEBUG:websockets.client:> Host: 127.0.0.1:9222
DEBUG:websockets.client:> Upgrade: websocket
DEBUG:websockets.client:> Connection: Upgrade
DEBUG:websockets.client:> Sec-WebSocket-Key: YXzDchxZh0BQ2D3iwOTd0g==
DEBUG:websockets.client:> Sec-WebSocket-Version: 13
DEBUG:websockets.client:> Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
DEBUG:websockets.client:> User-Agent: Python/3.12 websockets/10.4
DEBUG:websockets.client:< HTTP/1.1 101 Switching Protocols
DEBUG:websockets.client:< Upgrade: websocket
DEBUG:websockets.client:< Connection: upgrade
DEBUG:websockets.client:< Sec-Websocket-Accept: 4dGbyvZ25hQgyHUy6hwDm7zudqY=
DEBUG:websockets.client:= connection is OPEN
DEBUG:websockets.client:> TEXT '{"id": 1, "method": "Target.getBrowserContexts", "params": {}}' [62 bytes]
DEBUG:websockets.client:< TEXT '{"id":1,"result":{"browserContextIds":[]}}' [42 bytes]
DEBUG:pyppeteer.connection.Connection:RECV: {"id":1,"result":{"browserContextIds":[]}}
DEBUG:pyppeteer.connection.Connection:SEND: {"id": 2, "method": "Target.setDiscoverTargets", "params": {"discover": true}}
DEBUG:websockets.client:> TEXT '{"id": 2, "method": "Target.setDiscoverTargets"...s": {"discover": true}}' [78 bytes]
DEBUG:websockets.client:< TEXT '{"id": 2, "result": {}}' [23 bytes]
DEBUG:pyppeteer.connection.Connection:RECV: {"id": 2, "result": {}}
DEBUG:pyppeteer.connection.Connection:SEND: {"id": 3, "method": "Target.createTarget", "params": {"url": "about:blank"}}
DEBUG:websockets.client:> TEXT '{"id": 3, "method": "Target.createTarget", "par...{"url": "about:blank"}}' [76 bytes]
DEBUG:websockets.client:< TEXT '{"method":"Target.targetCreated","params":{"ses...D11109E2D4FEFBE4F89"}}}' [302 bytes]
DEBUG:pyppeteer.connection.Connection:RECV: {"method":"Target.targetCreated","params":{"sessionId":"4FDC2CB760A23A220497A05C95417CF4","targetInfo":{"targetId":"57356548460A8F29706A2ADF14316298","type":"page","title":"about:blank","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"22648B09EDCCDD11109E2D4FEFBE4F89"}}}
DEBUG:websockets.client:< TEXT '{"method":"Target.attachedToTarget","params":{"...tingForDebugger":true}}' [320 bytes]
DEBUG:pyppeteer.connection.Connection:RECV: {"method":"Target.attachedToTarget","params":{"sessionId":"4FDC2CB760A23A220497A05C95417CF4","targetInfo":{"targetId":"57356548460A8F29706A2ADF14316298","type":"page","title":"","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"22648B09EDCCDD11109E2D4FEFBE4F89"},"waitingForDebugger":true}}
DEBUG:websockets.client:< TEXT '{"id":3,"result":{"targetId":"57356548460A8F29706A2ADF14316298"}}' [65 bytes]
DEBUG:pyppeteer.connection.Connection:RECV: {"id":3,"result":{"targetId":"57356548460A8F29706A2ADF14316298"}}
DEBUG:pyppeteer.connection.Connection:SEND: {"id": 4, "method": "Target.attachToTarget", "params": {"targetId": "57356548460A8F29706A2ADF14316298"}}
DEBUG:websockets.client:> TEXT '{"id": 4, "method": "Target.attachToTarget", "p...8F29706A2ADF14316298"}}' [104 bytes]
DEBUG:websockets.client:< TEXT '{"method":"Target.attachedToTarget","params":{"...ingForDebugger":false}}' [343 bytes]
DEBUG:websockets.client:< TEXT '{"id":4,"result":{"sessionId":"65618675CB7D3585A95049E9DFE95EA9"}}' [66 bytes]
DEBUG:pyppeteer.connection.Connection:RECV: {"method":"Target.attachedToTarget","params":{"sessionId":"9559320D92474062597D9875C664CAC0","targetInfo":{"targetId":"CFCD6EC01573CF29BB638E9DC0F52DDC","type":"page","title":"New Incognito tab","url":"chrome://newtab/","attached":true,"canAccessOpener":false,"browserContextId":"65618675CB7D3585A95049E9DFE95EA9"},"waitingForDebugger":false}}
DEBUG:pyppeteer.connection.Connection:RECV: {"id":4,"result":{"sessionId":"65618675CB7D3585A95049E9DFE95EA9"}}
DEBUG:pyppeteer.connection.CDPSession:SEND: {"id": 1, "method": "Page.enable", "params": null}
DEBUG:pyppeteer.connection.Connection:SEND: {"id": 5, "method": "Target.sendMessageToTarget", "params": {"sessionId": "65618675CB7D3585A95049E9DFE95EA9", "message": "{\"id\": 1, \"method\": \"Page.enable\", \"params\": null}"}}
DEBUG:websockets.client:> TEXT '{"id": 5, "method": "Target.sendMessageToTarget... \\"params\\": null}"}}' [183 bytes]
^CTraceback (most recent call last):
  File "/home/pierre/wrk/issue339/issue339.py", line 12, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.12/asyncio/base_events.py", line 673, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 640, in run_forever
    self._run_once()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 1954, in _run_once
    event_list = self._selector.select(timeout)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

$ zig build && ./zig-out/bin/lightpanda                               
debug(cli): Server opts: listening internally on /tmp/lightpanda...                                                                         
info(websocket): starting blocking worker to listen on 127.0.0.1:9222                                                                       
info(server): accepting new conn...                                   
debug(websocket): (127.0.0.1:60146) connected                                                                                               
debug(handler): connected to Stream server                                                                                                  
info(server): client connected                                                                                                              
debug(websocket): (127.0.0.1:60146) connection successfully upgraded                                                                        
debug(websocket): (127.0.0.1:60146) received text message                                                                                   
debug(cdp): Req > id 1, method target.getBrowserContexts          
debug(cdp): Res > id 1, sessionID null, result cdp.target.getBrowserContexts.Resp{ .browserContextIds = {  } }           
debug(websocket): (127.0.0.1:60146) received text message                                                                                   
debug(cdp): Req > id 2, method target.setDiscoverTargets          
debug(cdp): Res > id 2, sessionID null, result null                                                                                         
debug(websocket): (127.0.0.1:60146) received text message                                                                                   
debug(cdp): Req > id 3, method target.createTarget                    
debug(cdp): Event > method Target.targetCreated, sessionID null                                                                              
debug(cdp): Event > method Target.attachedToTarget, sessionID null                                                                          
debug(cdp): Res > id 3, sessionID null, result cdp.target.createTarget { .targetId = 57356548460A8F29706A2ADF14316298 }                     
debug(websocket): (127.0.0.1:60146) received text message                                                                                   
debug(cdp): Req > id 4, method target.attachToTarget                                                                                                                                                                                                                                      
debug(cdp): Event > method Target.attachedToTarget, sessionID null    
debug(cdp): Res > id 4, sessionID null, result cdp.target.attachToTarget.SessionId{ .sessionId = { 54, 53, 54, 49, 56, 54, 55, 53, 67, 66, 55, 68, 51, 53, 56, 53, 65, 57, 53, 48, 52, 57, 69, 57, 68, 70, 69, 57, 53, 69, 65, 57 } }
debug(websocket): (127.0.0.1:60146) received text message                                                                                   
error(server): do error: error.UnknownMethod                                                                                                
debug(server): conn timeout, closing...                                                                                                     
debug(server): connection closed                                                                                                            
info(server): accepting new conn...                                                                                                         
debug(websocket): (127.0.0.1:60146) connection closed: error.Closed   
info(server): close cmd, closing conn...                                  

[krichprollsch]

The next issue was the support of the deprecated Target.sendMessageToTarget which is a kind of message wrapper https://chromedevtools.github.io/devtools-protocol/tot/Target/#method-sendMessageToTarget

#347