Browserless Timeout 60000ms exceeded after running for 2 days

[mat926]

Describe the bug
After about 2 days of running the browserless container, the browser checks start failing with Timeout 60000ms exceeded for every site with Playwright.

Version
0.45.16 on linux/docker

To Reproduce
Run and leave the browserless container running for at least 2 days.

Steps to reproduce the behavior:

1. Start the changedetection and browserless containers
2. Let them run for at least 2 days
3. The Chrome checks start failing with timeout errors

Expected behavior
The checks shouldn't error

Screenshots

Desktop (please complete the following information):

• Device: Synology DS224+
• OS: Synology DSM 7.2
• Version 7.2.1-69057 Update 4

Additional context
Current memory usage is at 60% so it's not a memory issue. The temporary workaround is to restart the browserless container and the checks work again.

Browserless logs:

2024-04-01T19:18:34.032Z browserless:server 3XK7JPGHF6PCBGCLRS0M7DZLHNW3X4G1: Recording timedout stat.
2024-04-01T19:18:34.032Z browserless:job 3XK7JPGHF6PCBGCLRS0M7DZLHNW3X4G1: Job has timed-out, closing the WebSocket.
2024-04-01T19:18:34.062Z browserless:job 3XK7JPGHF6PCBGCLRS0M7DZLHNW3X4G1: Cleaning up job
2024-04-01T19:18:34.062Z browserless:job 3XK7JPGHF6PCBGCLRS0M7DZLHNW3X4G1: No browser to cleanup, exiting
2024-04-01T19:18:34.062Z browserless:server Current workload complete.
2024-04-01T19:18:34.104Z browserless:job 3XK7JPGHF6PCBGCLRS0M7DZLHNW3X4G1: Websocket closed early, removing from queue and closing.
2024-04-01T19:18:34.104Z browserless:job 3XK7JPGHF6PCBGCLRS0M7DZLHNW3X4G1: Removing job from queue and cleaning up.
2024-04-01T19:18:34.104Z browserless:job 3XK7JPGHF6PCBGCLRS0M7DZLHNW3X4G1: Cleaning up job
2024-04-01T19:18:34.104Z browserless:job 3XK7JPGHF6PCBGCLRS0M7DZLHNW3X4G1: No browser to cleanup, exiting
2024-04-01T19:19:24.784Z browserless:server Health check stats: CPU 8%,6% MEM: 61%,61%
2024-04-01T19:19:24.784Z browserless:server Current period usage: {"date":1711998864804,"error":0,"rejected":0,"successful":0,"timedout":1,"totalTime":300030,"maxTime":300030,"minTime":300030,"meanTime":300030,"maxConcurrent":0,"units":11}
2024-04-01T19:24:24.784Z browserless:server Health check stats: CPU 9%,8% MEM: 61%,61%
2024-04-01T19:24:24.784Z browserless:server Current period usage: {"date":1711999164784,"error":0,"rejected":0,"successful":0,"timedout":0,"totalTime":0,"maxTime":0,"minTime":0,"meanTime":0,"maxConcurrent":0,"units":0}
2024-04-01T19:29:24.803Z browserless:server Health check stats: CPU 8%,9% MEM: 63%,61%
2024-04-01T19:29:24.803Z browserless:server Current period usage: {"date":1711999464784,"error":0,"rejected":0,"successful":0,"timedout":0,"totalTime":0,"maxTime":0,"minTime":0,"meanTime":0,"maxConcurrent":0,"units":0}
2024-04-01T19:33:14.736Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: /: Inbound WebSocket request.
2024-04-01T19:33:14.824Z browserless:hardware Checking overload status: CPU 8% Memory 64%
2024-04-01T19:33:14.824Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: Adding new job to queue.
2024-04-01T19:33:14.825Z browserless:server Starting new job
2024-04-01T19:33:14.825Z browserless:system Waiting pre-booted chrome instance
2024-04-01T19:33:14.825Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: Getting browser.
2024-04-01T19:34:24.827Z browserless:server Health check stats: CPU 7%,8% MEM: 63%,63%
2024-04-01T19:34:24.827Z browserless:server Current period usage: {"date":1711999764803,"error":0,"rejected":0,"successful":0,"timedout":0,"totalTime":0,"maxTime":0,"minTime":0,"meanTime":0,"maxConcurrent":1,"units":0}
2024-04-01T19:38:14.825Z browserless:server 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: Recording timedout stat.
2024-04-01T19:38:14.825Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: Job has timed-out, closing the WebSocket.
2024-04-01T19:38:14.854Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: Cleaning up job
2024-04-01T19:38:14.854Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: No browser to cleanup, exiting
2024-04-01T19:38:14.854Z browserless:server Current workload complete.
2024-04-01T19:38:14.896Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: Websocket closed early, removing from queue and closing.
2024-04-01T19:38:14.896Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: Removing job from queue and cleaning up.
2024-04-01T19:38:14.896Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: Cleaning up job
2024-04-01T19:38:14.896Z browserless:job 98RTW57TRD35EPIAGRTH29S3UW36P7NZ: No browser to cleanup, exiting
2024-04-01T19:39:24.945Z browserless:server Health check stats: CPU 8%,7% MEM: 66%,63%
2024-04-01T19:39:24.945Z browserless:server Current period usage: {"date":1712000064827,"error":0,"rejected":0,"successful":0,"timedout":1,"totalTime":300001,"maxTime":300001,"minTime":300001,"meanTime":300001,"maxConcurrent":0,"units":11}
2024-04-01T19:44:24.893Z browserless:server Health check stats: CPU 9%,8% MEM: 66%,66%
2024-04-01T19:44:24.893Z browserless:server Current period usage: {"date":1712000364945,"error":0,"rejected":0,"successful":0,"timedout":0,"totalTime":0,"maxTime":0,"minTime":0,"meanTime":0,"maxConcurrent":0,"units":0}

Docker-compose:

version: "3.2"
name: "change-detection"
services:
#https://github.com/dgtlmoon/changedetection.io/wiki/Synology-NAS-setup
  changedetection:
    image: dgtlmoon/changedetection.io:latest
    container_name: changedetection
    hostname: changedetection
    mem_limit: 1g
    cpu_shares: 768
    security_opt:
      - no-new-privileges:true
    volumes:
      - /volume1/docker/changedetection:/datastore:rw
      # Configurable proxy list support, see https://github.com/dgtlmoon/changedetection.io/wiki/Proxy-configuration#proxy-list-support
#        - ./proxies.json:/datastore/proxies.json
    environment:
      - PUID=1027
      - PGID=65536
      - TZ=America/Phoenix
      - UMASK=002
       #- WEBDRIVER_URL=http://localhost:4444/wd/hub
      #- PLAYWRIGHT_DRIVER_URL=ws://192.168.0.143:3000/
      - PLAYWRIGHT_DRIVER_URL=ws://playwright-chrome:3000
      - HIDE_REFERER=true
    ports:
      - 5555:5000
    # links:
      # - browser-chrome
    restart: always
    # networks:
     # - changedetection
    depends_on:
     playwright-chrome:
       condition: service_started
    logging:
      driver: json-file
      options:
        max-file: 10
        max-size: 10m

#https://github.com/dgtlmoon/changedetection.io/wiki/Synology-NAS-setup
  playwright-chrome:
    container_name: playwright-chrome
    hostname: playwright-chrome
    image: browserless/chrome:1.60-chrome-stable
    #image: browserless/chrome:latest #trying latest to fix the timeout errors
    restart: always
    mem_limit: 4g 
    # networks:
      # - changedetection
    ports:
      - 3000:3000
    environment:
      - PUID=1027
      - PGID=65536
      - TZ=America/Phoenix
      - UMASK=002
      - SCREEN_WIDTH=1920
      - SCREEN_HEIGHT=1024
      - SCREEN_DEPTH=16
      - ENABLE_DEBUGGER=false
      - PREBOOT_CHROME=true
      - KEEP_ALIVE=true
      - CONNECTION_TIMEOUT=300000
      - MAX_CONCURRENT_SESSIONS=4
      - CHROME_REFRESH_TIME=600000
      #https://github.com/dgtlmoon/changedetection.io/issues/1536
      - DEFAULT_BLOCK_ADS=false
      - DEFAULT_STEALTH=true
    #Use incognito mode 
     #- FUNCTION_ENABLE_INCOGNITO_MODE=true
        # #Ignore HTTPS errors, like for self-signed certs
      - DEFAULT_IGNORE_HTTPS_ERRORS=true
      #browserless routinely checks on the health of the image as it's running. Sometimes it's helpful to have it restart automatically when CPU or Memory usage are above 100% for a period of time (by default 5 minutes). In order for browserless to restart on health-failure checks, you'll have to set a parameter of 
      #EXIT_ON_HEALTH_FAILURE=true.
    #dns:
    #  - 192.168.0.45
    logging:
      driver: json-file
      options:
        max-file: 10
        max-size: 10m
        
  # playwright-chrome:
    # container_name: playwright-chrome
    # hostname: playwright-chrome
    # image: dgtlmoon/sockpuppetbrowser:latest
    # cap_add:
        # - SYS_ADMIN
# # SYS_ADMIN might be too much, but it can be needed on your platform https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-on-gitlabci
    # restart: unless-stopped
    # environment:
       # - PUID=1027
       # - PGID=65536
       # - TZ=America/Phoenix
       # - SCREEN_WIDTH=1920
       # - SCREEN_HEIGHT=1024
       # - SCREEN_DEPTH=16
       # - MAX_CONCURRENT_CHROME_PROCESSES=10
    # logging:
       # driver: json-file
       # options:
         # max-file: 10
         # max-size: 10m

volumes:
 changedetection:

[dgtlmoon]

hmm yeah that means forwhatever reason browserless didnt respond after 60 seconds, so its a problem in their project not ours

not sure what we can do about that..

[dgtlmoon]

btw, what hardware?

[mat926]

btw, what hardware?

This is Synology DS224+ with 6GB ram.

[dgtlmoon]

I just remembered something that I came across - you should specify --user-data-dir=/tmp/data then make sure you check /tmp is not full, i bet its disk usage related?

[SpanishHearts]

I'm having the same issue on my Synology DS218+. I just installed it but I get this error.