Large threaded, kubernetes scrape = Target page, context or browser has been closed

[JoshuaPerk]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/playwright (PlaywrightCrawler)

Issue description

I'm:

1. Running a node app with worker threads (usually 32 of them)
2. Running multiple containers in kubernetes

Each thread:

1. Grabs 5 domains from my postgres DB (of 5 million!)
2. Loops through each domain
3. Creates a new PlaywrightCrawler with unique-named storages (to prevent collision / global deletion from crawlers in other threads)
4. Queues the domains home page
5. Controllers then queue up some additional pages based on what's found on the home page
6. The results are processed in real-time and pushed to the database (since we don't want to wait until all 5M all are complete
7. The thread-specific sotrages are then deleted used drop()

The Problem

This works flawlessly... for about 60 minutes... afterwards, I get plagued with Target page, context or browser has been closed. It appears at the ~ hour mark is when this first presents itself and then incrementally increases in frequency until I'm getting more failed records than successful (at which point, I kill the cluster or restart it).

What I've tried:

1. browserPoolOptions like retireBrowserAfterPageCount: 100 and closeInactiveBrowserAfterSecs: 200
2. await crawler.teardown(); in hopes that this would clear and sort of cache/memory that could be stacking up
3. A cron to restart my cluster 🤣
4. Ensuring the EBS volumes are not running out of space (they're 20GB each and seem to be 50% full when crashing)
5. Ensuring the pods have plenty of memory (running EC2s with 64GB memory and 16 CPU (32 threads). Seems to handle the load in the first hour just fine.

I suspect there's a leak or store not being cleared out since it happens gradually?

Code sample

No response

Package version

"crawlee": "^3.9.2",

Node.js version

v20.14.0

Operating system

linux/amd64

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

It's hard to help with such a production issue without any reproduction, what I would try is using different playwright versions, as the Target page, context or browser has been closed error is a pretty common problem on their end and could easily be caused by something outside of our control. From our experience, these things unfortunately tend to happen randomly and often it is about the specific playwright version.

[B4nan]

"crawlee": "^3.9.2",

Also not sure what this means (you should put there a version you use, not a version range), if you are not on latest, upgrade, there was one issue that might be causing it actually.