[Bug]: Memory Leak on Repeated /md Requests via Docker (MacOS) — Container Crashes Randomly Over Time

[Parth7030]

crawl4ai version

0.6.0-r2 or Latest docker image

Expected Behavior

When sending repeated POST requests to /md every 10 seconds with dynamic URLs, the Docker container should handle each request efficiently without memory leaks or crashes, maintaining stable memory usage over time.

Current Behavior

Initially, everything works fine. However, after some time (randomly, depending on duration and frequency), memory usage starts increasing significantly until the container crashes. This suggests a possible memory leak or improper cleanup of resources on repeated executions.

Attached below is a screenshot showing the memory spike before the crash.

Is this reproducible?

Yes

Inputs Causing the Bug

Example request payload sent via Postman every 10 seconds:

{
  "url": "http://www.example.com",
  "f": "fit",
  "q": null,
  "c": "0"
}

Steps to Reproduce

1.	Run the crawl4ai app using the Docker setup.
2.	Use a tool (like Postman or a script) to send POST requests to http://localhost/md every 10 seconds.
3.	Use a changing URL in the request payload for each call.
4.	Monitor Docker memory usage over time.
5.	Observe the steady increase in memory and eventual container crash.

Code snippets

OS

macOS Sequoia Version 15.5

Python version

3.12

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

[tanushshukla]

yeah I have same issues as well.

[emyco]

We are experiencing the same issue and I would like to provide some additional details:

1. I have attached the logs for your review.
2. Notably, when the container is hanging, the /health endpoint still returns a status of 200.

Please let me know if you need any further information.

logs-from-crawl4ai-in-crawl4ai-575d75bc87-bljrk.log

[NuoWenLei]

Is there a current workaround for this? I notice this in 0.6.0 as well as in 0.6.3 (built into docker image).

[NuoWenLei]

Also a tangentially related issue to this thread. In crawler_pool.py since it uses psutil.virtual_memory() to check for memory usage percentage, it returns the host memory usage percentage. I think it would be better if the crawler pool is thresholded based on the container memory usage.

giampaolo/psutil#1011

[rawinkler]

I observed the same issue. Is it possible that all the crawled content like images etc. are simply added up during the multiple calls and never get released, leading to the observed continuous usage of RAM?

Would there be any dirty workarounds for this, such as restarting "the browsers" periodically?

[Reinstaal]

same issue here, only restarting the container helps

[arturbacilla]

@unclecode I belive this is a recurrent behaviour in every endpoint. I'm having similar issues on Linux:

• endpoint: /crawl
• version 0.7.0

In my case I have an n8n automation calling for this endpoint every ~7 or 8 seconds and it works pretty well for the first ~30 to 50 iterations, but then it just starts returning error 500 (crawl4ai endpoint, not n8n automation).
I've registered the time spent on each request and it builds up gradually in a clear pattern (like 1 to 2s every 30 to 50 iterations) until it stops working throwing an "Internal Server Error" and the stack trace having something related to axios.

Sadly I wasnt' able to investigate further

[SimonMayerhofer]

Same issue on linux version 0.7.2

[NuoWenLei]

I looked a bit into the code, and I think there are multiple sources to the memory leak though I couldn't identify all of them. There seems to be some kind of held references of the results even after the API calls because I was able to reduce the memory leak somewhat by emptying the result objects in async_dispatcher.py.

445                    # Process completed tasks
446                    for completed_task in done:
447                        result = await completed_task
+++                        result.result.html = ""
+++                        result.result.fit_html = ""
448                        results.append(result)

Additionally, a temporary workaround I found, though somehow it still doesn't seem to clear all the memory leak on my end, was add a restarter script in the supervisord config that periodically restarts the gunicorn service.

[alitavanaali]

Good afternoon everyone,
in my case, I discovered that I was calling the crawler without closing it properly, which was eating up my memory.
For now, the solution I found is this:

class LLMExtractor:
    def __init__(self):
        # crawler initiation
        self.crawler = AsyncWebCrawler()
        self.crawler_config = CrawlerRunConfig(
            delay_before_return_html=delay_before_html, cache_mode=CacheMode.DISABLED
        )

    async def close_crawler(self):
        """Close the crawler to free resources."""
        if self.crawler:
            await self.crawler.close()
            self.crawler = None
            gc.collect()

and whenever crawling is done, I call this close function. Hope it helps you

[bockmurat]

Finally, I thought I am crazy - it eats so much memory and is unstable with the Docker version.
Best performance I could get was by setting those settings for my container:
env:
PLAYWRIGHT_MAX_CONCURRENCY=2
Resource limits:

then it kills itself, but if used in some workflows you need to detect that it's down, else you get no crawling

[ntohidi]

Hello everyone! We have made some improvements to our latest Docker image: version 0.7.7
Please use this updated image and let us know if you are still experiencing the same memory issue.

I just tried to reproduce the issue with the code, and here are the results: