Long Apify API calls can block crawler

[Pijukatel]

BasicCrawler calls fetch_next_request which under normal circumstances should fetch request within miliseconds, but sometimes can freeze and it can take tens of seconds.

Example normal request processing with injected timestamps for each task:

Example of long wait for fetch_next_request:

How to reproduce.
Use crawlee branch with injected timestamps
Create BeautifulSoup crawler from crawlee templates, push it to apify platform and let it run on with start page "https://warehouse-theme-metal.myshopify.com/" and max_requests_per_crawl=3000. It is not deterministic and I had to do several runs to capture the error. But it is also not some super rare glitch as it can be reproduced after few attempts.
Example run: https://console.apify.com/actors/C0lWh1UCQvgdArp6R/runs/xsqRrdd2s1f6HRY9N#log

Either fetch_next_request is taking too much time or something is blocking the event loop.

[Pijukatel]

Issue is most likely some race condition that causes the RequestQueue to lock itself out from accessing some requests. Possibly in _get_or_hydrate_request that has many execution paths and many awaits - including locking requets.

For example this request was stuck in the very beginning of the _get_or_hydrate_request for 47 s. Once the request is available, it is handled quickly. In this scenario there is no one else who could lock the requests, but this instance of RequestQueue. So it is waiting for locks that it caused itself.

[Pijukatel]

In the following BSCrawlerFreeze.log, the crawler freezes on processing the last request for couple of minutes. No load, free capacity and just reporting "Deciding if we are finished based on queue_has_locked_requests = True". Seems like the last request then somehow times out after 180s, is retried and immediately succeeds.

[B4nan]

could be about expo backoff retries in the client

[Pijukatel]

could be about expo backoff retries in the client

I tried adding more logs around that part and I have a theory now. Http client used for the Apify API calls have default timeout for one request 360 seconds!!! In the log snippet you can see that in the response we receive 504 Gateway Timeout. We could be in theory waiting 6 minutes for this response. After getting the response the request is retried and immediately it passes. But this whole time it was blocking the crawler from stopping.

@B4nan do you think there is any justification to have such a long default timeout there since we have it wrapped in exponential back off anyway?

2025-04-07T14:18:31.223Z [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 1; desired_concurrency = 13; cpu = 0.0; mem = 0.0; event_loop = 0.0; client_info = 0.0
2025-04-07T14:19:21.927Z [apify_client] WARN  Request unsuccessful ({"status_code": 504, "client_method": "RequestQueueClientAsync.batch_add_requests", "resource_id": "uC1DxnYi6uiKoPMxk", "method": "POST", "url": "http://10.0.47.96:8010/v2/request-queues/uC1DxnYi6uiKoPMxk/requests/batch", "attempt": 1})
2025-04-07T14:19:21.929Z [apify_client] WARN  Request unsuccessful ({"status_code": 504, "client_method": "RequestQueueClientAsync.batch_add_requests", "resource_id": "uC1DxnYi6uiKoPMxk", "method": "POST", "url": "http://10.0.47.96:8010/v2/request-queues/uC1DxnYi6uiKoPMxk/requests/batch", "attempt": 1, "message": "Request unsuccessful"})
2025-04-07T14:19:21.930Z [apify_client] WARN  Backoff sleep 0.5927505920639138. for attempt 1. Max retries: 8 ({"client_method": "RequestQueueClientAsync.batch_add_requests", "resource_id": "uC1DxnYi6uiKoPMxk", "method": "POST", "url": "http://10.0.47.96:8010/v2/request-queues/uC1DxnYi6uiKoPMxk/requests/batch", "attempt": 1})
2025-04-07T14:19:21.931Z [apify_client] WARN  Backoff sleep 0.5927505920639138. for attempt 1. Max retries: 8 ({"client_method": "RequestQueueClientAsync.batch_add_requests", "resource_id": "uC1DxnYi6uiKoPMxk", "method": "POST", "url": "http://10.0.47.96:8010/v2/request-queues/uC1DxnYi6uiKoPMxk/requests/batch", "attempt": 1, "message": "Backoff sleep 0.5927505920639138. for attempt 1. Max retries: 8"})
2025-04-07T14:19:22.564Z [apify_client] WARN  <Response [201 Created]> ({"client_method": "RequestQueueClientAsync.batch_add_requests", "resource_id": "uC1DxnYi6uiKoPMxk", "method": "POST", "url": "http://10.0.47.96:8010/v2/request-queues/uC1DxnYi6uiKoPMxk/requests/batch", "attempt": 2})

[B4nan]

As discussed on the sync, we can try lowering the timeout, but only from the context of crawlee, not directly in the client.

In the JS version, we are already doing this for the request queue here:

https://github.com/apify/crawlee/blob/07cd2c43fcbf21c6917b5bf55cd1af64c4bdcb00/packages/core/src/storages/request_provider.ts#L69-L72

[janbuchar]

Well, I think we could also tackle this in the SDK - the ApifyStorageClient could make its own ApifyClient with adjusted settings.

[B4nan]

But SDK is not doing any retries, crawlee is. People can use SDK without crawlee.

[Pijukatel]

But SDK is not doing any retries, crawlee is. People can use SDK without crawlee.

In this context it is the apify client doing the retries, so neither crawlee nor SDK. But I do not think the retires are the problem, the problem is the initially long timeout, so we wait on the first attempt already. Are there any calls to the Apify API that actually takes 6 minutes? Even if that is the case I guess we should get to this timeout through the exponential back off and not start with such large timeout out of the box.

[B4nan]

Are you sure it's the initial timeout? Back then when I was testing this in the JS version, it was the upper bound for the expo backoff, not the initial timeout, at least I think so.

[B4nan]

Maybe my memory is too rusty, looking at the code, it seems like you are correct. But it feels quite crazy we would have such a long timeout as the default, combined with expo backoff, so hard to believe in that. I guess lets try it out in isolation to be sure.

[Pijukatel]

I would suggest some change like this for example: apify/apify-sdk-python@master...decrease-ApifyStorageClient-default-timeout#diff-0de5dbb2bc0c5dafb72bcc910c47b7693b65e6c7437ba55bb96b28836ab5a7daR34

Actual code change is 1 line, but it was quite complicated to test it as the timeouts are on the level where respx can't be used. So I used simplified test server from our crawlee repo.

(I am not very sure about including such test in the change. This is more of a configuration parameter. On the other hand, without the test it can be changed back and nobody will notice... )

What can go wrong?
Imagine two scenarios:

1. We send request to the API, but nothing happens and server does not react for whatever reason and it will never process the first request. We try again after timeout and then it works. If the timeout is set to default 360 seconds, we wait so long for no reason and we waste a lot of time doing nothing. This currently happens and it is not so rare. This can block whole crawler for minutes and burn resources.

2. We send request to the API, but nothing happens. We try again after timeout, but server was just delayed and actually it processes both requests. So in the end it receives two identical calls and it tries to execute both. Depending on the request, it could be issue or not. This can happen in theory even now, but due to large timeout of 360 seconds, it is probably not happening in practice.

So this change is about probability and impact change between two scenarios which can both still happen. Reducing negative impact of the first one and increasing the probability of the second one.