Skip to content

Commit 7951a66

Browse files
drobnikjTC-MO
andauthored
fix(request-queue): Update request queue locking docs (#1322)
I made some changes in docs regarding the API changes. * stress out that locking works on the client level and the same as on the run level. * update code example by separating code into two actors using code tabs --------- Co-authored-by: Michał Olender <[email protected]>
1 parent 35d2b5e commit 7951a66

File tree

1 file changed

+79
-28
lines changed

1 file changed

+79
-28
lines changed

sources/platform/storage/request_queue.md

Lines changed: 79 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -407,7 +407,20 @@ You can lock a request so that no other clients receive it when they fetch the q
407407
This feature is seamlessly integrated into Crawlee, requiring minimal extra setup. By default, requests are locked for the same duration as the timeout for processing requests in the crawler ([`requestHandlerTimeoutSecs`](https://crawlee.dev/api/next/basic-crawler/interface/BasicCrawlerOptions#requestHandlerTimeoutSecs)).
408408
If the Actor processing the request fails, the lock expires, and the request is processed again eventually. For more details, refer to the [Crawlee documentation](https://crawlee.dev/docs/next/experiments/experiments-request-locking).
409409

410-
In the following example, we demonstrate how we can use locking mechanisms to avoid concurrent processing of the same request.
410+
In the following example, we demonstrate how you can use locking mechanisms to avoid concurrent processing of the same request across multiple Actor runs.
411+
412+
:::info
413+
The lock mechanism works on the client level, as well as the run level, when running the Actor on the Apify platform.
414+
415+
This means you can unlock or prolong the lock the locked request only if:
416+
417+
- You are using the same client key, or
418+
- The operation is being called from the same Actor run.
419+
420+
:::
421+
422+
<Tabs groupId="main">
423+
<TabItem value="Actor 1" label="Actor 1">
411424

412425
```js
413426
import { Actor, ApifyClient } from 'apify';
@@ -422,15 +435,12 @@ const client = new ApifyClient({
422435
const requestQueue = await client.requestQueues().getOrCreate('example-queue');
423436

424437
// Creates two clients with different keys for the same request queue.
425-
const requestQueueClientOne = client.requestQueue(requestQueue.id, {
438+
const requestQueueClient = client.requestQueue(requestQueue.id, {
426439
clientKey: 'requestqueueone',
427440
});
428-
const requestQueueClientTwo = client.requestQueue(requestQueue.id, {
429-
clientKey: 'requestqueuetwo',
430-
});
431441

432442
// Adds multiple requests to the queue.
433-
await requestQueueClientOne.batchAddRequests([
443+
await requestQueueClient.batchAddRequests([
434444
{
435445
url: 'http://example.com/foo',
436446
uniqueKey: 'http://example.com/foo',
@@ -454,53 +464,94 @@ await requestQueueClientOne.batchAddRequests([
454464
]);
455465

456466
// Locks the first two requests at the head of the queue.
457-
const processingRequestsClientOne = await requestQueueClientOne.listAndLockHead(
467+
const processingRequestsClientOne = await requestQueueClient.listAndLockHead(
458468
{
459469
limit: 2,
460-
lockSecs: 60,
470+
lockSecs: 120,
461471
},
462472
);
463473

474+
// Checks when the lock will expire. The locked request will have a lockExpiresAt attribute.
475+
const lockedRequest = processingRequestsClientOne.items[0];
476+
const lockedRequestDetail = await requestQueueClient.getRequest(
477+
lockedRequest.id,
478+
);
479+
console.log(`Request locked until ${lockedRequestDetail?.lockExpiresAt}`);
480+
481+
// Prolongs the lock of the first request or unlocks it.
482+
await requestQueueClient.prolongRequestLock(
483+
lockedRequest.id,
484+
{ lockSecs: 120 },
485+
);
486+
await requestQueueClient.deleteRequestLock(
487+
lockedRequest.id,
488+
);
489+
490+
await Actor.exit();
491+
```
492+
493+
</TabItem>
494+
<TabItem value="Actor 2" label="Actor 2">
495+
496+
```js
497+
import { Actor, ApifyClient } from 'apify';
498+
499+
await Actor.init();
500+
501+
const client = new ApifyClient({
502+
token: 'MY-APIFY-TOKEN',
503+
});
504+
505+
// Waits for the first Actor to lock the requests.
506+
await new Promise((resolve) => setTimeout(resolve, 5000));
507+
508+
// Get the same request queue in different Actor run and with a different client key.
509+
const requestQueue = await client.requestQueues().getOrCreate('example-queue');
510+
511+
const requestQueueClient = client.requestQueue(requestQueue.id, {
512+
clientKey: 'requestqueuetwo',
513+
});
514+
515+
// Get all requests from the queue and check one locked by the first Actor.
516+
const requests = await requestQueueClient.listRequests();
517+
const requestsLockedByAnotherRun = requests.items.filter((request) => request.lockByClient === 'requestqueueone');
518+
const requestLockedByAnotherRunDetail = await requestQueueClient.getRequest(
519+
requestsLockedByAnotherRun[0].id,
520+
);
521+
464522
// Other clients cannot list and lock these requests; the listAndLockHead call returns other requests from the queue.
465-
const processingRequestsClientTwo = await requestQueueClientTwo.listAndLockHead(
523+
const processingRequestsClientTwo = await requestQueueClient.listAndLockHead(
466524
{
467-
limit: 2,
525+
limit: 10,
468526
lockSecs: 60,
469527
},
470528
);
471-
472-
// Checks when the lock will expire. The locked request will have a lockExpiresAt attribute.
473-
const theFirstRequestLockedByClientOne = processingRequestsClientOne.items[0];
474-
const requestLockedByClientOne = await requestQueueClientOne.getRequest(
475-
theFirstRequestLockedByClientOne.id,
529+
const wasBothRunsLockedSameRequest = !!processingRequestsClientTwo.items.find(
530+
(request) => request.id === requestLockedByAnotherRunDetail.id,
476531
);
477-
console.log(`Request locked until ${requestLockedByClientOne?.lockExpiresAt}`);
532+
533+
console.log(`Was the request locked by the first run locked by the second run? ${wasBothRunsLockedSameRequest}`);
534+
console.log(`Request locked until ${requestLockedByAnotherRunDetail?.lockExpiresAt}`);
478535
479536
// Other clients cannot modify the lock; attempting to do so will throw an error.
480537
try {
481-
await requestQueueClientTwo.prolongRequestLock(
482-
theFirstRequestLockedByClientOne.id,
538+
await requestQueueClient.prolongRequestLock(
539+
requestLockedByAnotherRunDetail.id,
483540
{ lockSecs: 60 },
484541
);
485542
} catch (err) {
486543
// This will throw an error.
487544
}
488545
489-
// Prolongs the lock of the first request or unlocks it.
490-
await requestQueueClientOne.prolongRequestLock(
491-
theFirstRequestLockedByClientOne.id,
492-
{ lockSecs: 60 },
493-
);
494-
await requestQueueClientOne.deleteRequestLock(
495-
theFirstRequestLockedByClientOne.id,
496-
);
497-
498546
// Cleans up the queue.
499-
await requestQueueClientOne.delete();
547+
await requestQueueClient.delete();
500548
501549
await Actor.exit();
502550
```
503551
552+
</TabItem>
553+
</Tabs>
554+
504555
A detailed tutorial on how to process one request queue with multiple Actor runs can be found in [Academy tutorials](https://docs.apify.com/academy/node-js/multiple-runs-scrape).
505556
506557
## Sharing

0 commit comments

Comments
 (0)