custom docker setup - Error in playwright-chrome container _request_retry:188 - No response from Chrome

[MMquant]

Describe the bug
playwright-chrome fetcher doesn't work with current docker-compose.yml settings.
https://github.com/dgtlmoon/changedetection.io/blob/master/docker-compose.yml

Version
v0.45.16

To Reproduce

# docker-compose.yml
version: '3.2'
services:
    changedetection:
      image: ghcr.io/dgtlmoon/changedetection.io
      container_name: changedetection
      hostname: changedetection
      volumes:
        - changedetection-data:/datastore
      environment:
        - PLAYWRIGHT_DRIVER_URL=ws://playwright-chrome:3000
        - BASE_URL=https://changed.ish.mmquant.net
        - USE_X_SETTINGS=1
        - HIDE_REFERER=true
        - FETCH_WORKERS=10
      ports:
        - ${PRIVATE_STATIC_IP}:5000:5000
      restart: always
      depends_on:
          playwright-chrome:
              condition: service_started
    playwright-chrome:
        container_name: playwright-chrome
        hostname: playwright-chrome
        image: dgtlmoon/sockpuppetbrowser:latest
        cap_add:
            - SYS_ADMIN
        restart: always
        environment:
            - SCREEN_WIDTH=1920
            - SCREEN_HEIGHT=1024
            - SCREEN_DEPTH=16
            - MAX_CONCURRENT_CHROME_PROCESSES=10
volumes:
  changedetection-data:

# docker logs -ft changedetection
...
2024-03-12T08:45:32.025075068Z 2024-03-12 08:45:32.010 | INFO     | changedetectionio.update_worker:run:248 - Processing watch UUID d8d6c183-3df6-4224-90c0-38d17b914092 Priority 1 URL https://news.ycombinator.com/
2024-03-12T08:45:39.325983057Z 2024-03-12 08:45:39.318 | ERROR    | changedetectionio.update_worker:run:463 - Exception reached processing watch UUID: d8d6c183-3df6-4224-90c0-38d17b914092
2024-03-12T08:45:39.327237155Z 2024-03-12 08:45:39.326 | ERROR    | changedetectionio.update_worker:run:464 - Target page, context or browser has been closed
2024-03-12T08:45:39.329305174Z 2024-03-12 08:45:39.328 | DEBUG    | changedetectionio.update_worker:run:541 - Watch d8d6c183-3df6-4224-90c0-38d17b914092 done in 7.31s
2024-03-12T08:45:45.582378272Z 2024-03-12 08:45:45.577 | INFO     | changedetectionio.store:sync_to_json:420 - Saving JSON...
...

# docker logs -ft playwright-chrome
...
2024-03-12T08:45:30.673390356Z 2024-03-12 08:45:30.667 | INFO     | __main__:stats_thread_func:320 - Connections: Active count 0 of max 10, Total processed: 2.
2024-03-12T08:45:30.677979875Z 2024-03-12 08:45:30.677 | INFO     | __main__:stats_thread_func:325 - Memory: Used 89.1% (Limit 90%) - Available 75.9MB.
2024-03-12T08:45:34.178383633Z 2024-03-12 08:45:34.175 | DEBUG    | __main__:launchPlaywrightChromeProxy:207 - WebSocket ID: 9cdb3593-36dd-4feb-8016-0486ba9379f1 Got new incoming connection ID from 192.168.160.3:47662 (/)
2024-03-12T08:45:34.217434582Z 2024-03-12 08:45:34.201 | DEBUG    | __main__:launch_chrome:100 - No --window-size in start query, falling back to env var --window-size=1920,1024
2024-03-12T08:45:34.222278560Z 2024-03-12 08:45:34.221 | DEBUG    | __main__:launch_chrome:108 - No user-data-dir in query, using /tmp/chrome-puppeteer-proxy52fa1pum
2024-03-12T08:45:34.341985709Z 2024-03-12 08:45:34.339 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:34.547036490Z 2024-03-12 08:45:34.546 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:34.761506871Z 2024-03-12 08:45:34.760 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:34.965958489Z 2024-03-12 08:45:34.965 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:35.173100288Z 2024-03-12 08:45:35.171 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:35.491678179Z 2024-03-12 08:45:35.488 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:35.698542915Z 2024-03-12 08:45:35.697 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:35.924812099Z 2024-03-12 08:45:35.902 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:36.122507746Z 2024-03-12 08:45:36.106 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:36.335510000Z 2024-03-12 08:45:36.334 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:36.540908705Z 2024-03-12 08:45:36.539 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:36.764586044Z 2024-03-12 08:45:36.762 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:36.975775124Z 2024-03-12 08:45:36.975 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:37.179438122Z 2024-03-12 08:45:37.178 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:37.388613588Z 2024-03-12 08:45:37.387 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:37.614301553Z 2024-03-12 08:45:37.613 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:37.898241231Z 2024-03-12 08:45:37.876 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:38.160261133Z 2024-03-12 08:45:38.139 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:38.457737638Z 2024-03-12 08:45:38.423 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:38.741221716Z 2024-03-12 08:45:38.698 | WARNING  | __main__:_request_retry:188 - No response from Chrome, retrying..
2024-03-12T08:45:38.936396882Z 2024-03-12 08:45:38.927 | CRITICAL | __main__:launchPlaywrightChromeProxy:257 - Uhoh! Looks like Chrome did not start! do you need --cap-add=SYS_ADMIN added to start this container? permissions are OK?
2024-03-12T08:45:38.949305798Z 2024-03-12 08:45:38.936 | CRITICAL | __main__:launchPlaywrightChromeProxy:258 - While trying to connect to http://localhost:55385/json/version - , Closing attempted chrome process
2024-03-12T08:45:39.017980977Z 2024-03-12 08:45:38.964 | CRITICAL | __main__:close_socket:135 - Attempting to close socket
2024-03-12T08:45:39.193463948Z 2024-03-12 08:45:39.189 | DEBUG    | __main__:stats_disconnect:151 - Websocket 9cdb3593-36dd-4feb-8016-0486ba9379f1 - Connection ended, processed in 5.014s
2024-03-12T08:45:39.193497436Z 2024-03-12 08:45:39.190 | DEBUG    | __main__:cleanup_chrome_by_pid:157 - Websocket 9cdb3593-36dd-4feb-8016-0486ba9379f1 - Cleaning up chrome pid: 25
2024-03-12T08:45:40.680291625Z 2024-03-12 08:45:40.679 | INFO     | __main__:stats_thread_func:320 - Connections: Active count 0 of max 10, Total processed: 3.
...

changedetection.io UI screenshots

I tried to test it even with sec_comp set up for the playwright-chrome container but got same error

playwright-chrome:
        container_name: playwright-chrome
        hostname: playwright-chrome
        image: dgtlmoon/sockpuppetbrowser:latest
        ipc: host
        security_opt:
        - seccomp=chrome.json

Expected behavior
I would expect the playwright-chrome service to successfully start the chrome and return data to changedetection service,

[dgtlmoon]

2024-03-12T08:45:38.936396882Z 2024-03-12 08:45:38.927 | CRITICAL | __main__:launchPlaywrightChromeProxy:257 - Uhoh! Looks like Chrome did not start! do you need --cap-add=SYS_ADMIN added to start this container? permissions are OK?

hmm you have that option

what operating system and hardware? you didnt say

[MMquant]

# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"

docker is running on digital ocean droplet s-1vcpu-1gb

[malys]

Hi exactly the same on OCI. (AMD-based compute VMs with 1/8 OCPU and 1 GB memory each)

NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

Docker version 25.0.4, build 1a576c5

@MMquant Have you found a solution?

[MMquant]

TBH if only one user is running only one changedetection service then 1gb and 1 cpu should be sufficient. It's a matter of non-optimized playwright-chrome image.

[malys]

Analysing code, sleep method arguments are statics. It's a problem of timeout management in server.py
I have applied some changes for testing
and somethings like that WAIT_UNIT=200 (increase drastically timeouts)

docker run --rm -p 127.0.0.1:3000:3000 --name playwright-chrome --net detectionchange --ip 172.18.0.3 -e MAX_CONCURRENT_CHROME_PROCESSES=1 -e SCREEN_WIDTH=1680 -e SCREEN_HEIGHT=1050 -e SCREEN_DEPTH=16 -e HARD_MEMORY_USAGE_LIMIT_PERCENT=95 -e STATS_REFRESH_SECONDS=10 -e WAIT_UNIT=200 -v /opt/detectionchange/playbooks/server.py:/usr/src/app/server.py dgtlmoon/sockpuppetbrowser 

No more errors but my VM instance is totally overloaded and is too slow.
It's strange because on my RPI, I have installed manu more apps and everythinh works fine.

docker agents & changedetection are too heavy for this VM

@MMquant Could you test it?

[dgtlmoon]

@malys i deleted that HUGE slab of code, please do not paste in huge slabs of code as a comment, please open it as a PR to that project instead, i'de love to be able to actually see the code differences and be more productive than just a slab of code

[malys]

Hi @dgtlmoon but forking a child project to share an example it's not the easiest way :) but I did it. It's not PR but only a quick and dirty example.

In my case, I will not enable playwright.
Thanks for your help and your project @dgtlmoon 👌

[MMquant]

Hi @malys,
tested your env vars but getting exactly same errors. Currently I'm using changedetection without playwright as well.

[mat926]

Hello I have the same issue. I will get the error on the first check but it works fine if I recheck it.