Skip to content

Commit df52238

Browse files
committed
fix(request-queue): Update rq locking docs
1 parent 6873e01 commit df52238

File tree

1 file changed

+62
-22
lines changed

1 file changed

+62
-22
lines changed

sources/platform/storage/request_queue.md

Lines changed: 62 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -407,7 +407,10 @@ You can lock a request so that no other clients receive it when they fetch the q
407407
This feature is seamlessly integrated into Crawlee, requiring minimal extra setup. By default, requests are locked for the same duration as the timeout for processing requests in the crawler ([`requestHandlerTimeoutSecs`](https://crawlee.dev/api/next/basic-crawler/interface/BasicCrawlerOptions#requestHandlerTimeoutSecs)).
408408
If the Actor processing the request fails, the lock expires, and the request is processed again eventually. For more details, refer to the [Crawlee documentation](https://crawlee.dev/docs/next/experiments/experiments-request-locking).
409409

410-
In the following example, we demonstrate how we can use locking mechanisms to avoid concurrent processing of the same request.
410+
In the following example, we demonstrate how we can use locking mechanisms to avoid concurrent processing of the same request across multiple Actor runs.
411+
412+
<Tabs groupId="main">
413+
<TabItem value="Actor 1" label="Actor 1">
411414

412415
```js
413416
import { Actor, ApifyClient } from 'apify';
@@ -425,9 +428,6 @@ const requestQueue = await client.requestQueues().getOrCreate('example-queue');
425428
const requestQueueClientOne = client.requestQueue(requestQueue.id, {
426429
clientKey: 'requestqueueone',
427430
});
428-
const requestQueueClientTwo = client.requestQueue(requestQueue.id, {
429-
clientKey: 'requestqueuetwo',
430-
});
431431

432432
// Adds multiple requests to the queue.
433433
await requestQueueClientOne.batchAddRequests([
@@ -457,23 +457,71 @@ await requestQueueClientOne.batchAddRequests([
457457
const processingRequestsClientOne = await requestQueueClientOne.listAndLockHead(
458458
{
459459
limit: 2,
460-
lockSecs: 60,
460+
lockSecs: 120,
461461
},
462462
);
463463

464+
// Checks when the lock will expire. The locked request will have a lockExpiresAt attribute.
465+
const theFirstRequestLockedByClientOne = processingRequestsClientOne.items[0];
466+
const requestLockedByClientOne = await requestQueueClientOne.getRequest(
467+
theFirstRequestLockedByClientOne.id,
468+
);
469+
console.log(`Request locked until ${requestLockedByClientOne?.lockExpiresAt}`);
470+
471+
// Prolongs the lock of the first request or unlocks it.
472+
await requestQueueClientOne.prolongRequestLock(
473+
theFirstRequestLockedByClientOne.id,
474+
{ lockSecs: 120 },
475+
);
476+
await requestQueueClientOne.deleteRequestLock(
477+
theFirstRequestLockedByClientOne.id,
478+
);
479+
480+
// Cleans up the queue.
481+
await requestQueueClientOne.delete();
482+
483+
await Actor.exit();
484+
```
485+
486+
</TabItem>
487+
<TabItem value="Actor 2" label="Actor 2">
488+
489+
```js
490+
import { Actor, ApifyClient } from 'apify';
491+
492+
await Actor.init();
493+
494+
const client = new ApifyClient({
495+
token: 'MY-APIFY-TOKEN',
496+
});
497+
498+
// Waits for the first Actor to lock the requests.
499+
await new Promise((resolve) => setTimeout(resolve, 5000));
500+
501+
// Creates a new request queue.
502+
const requestQueue = await client.requestQueues().getOrCreate('example-queue');
503+
504+
const requestQueueClientTwo = client.requestQueue(requestQueue.id, {
505+
clientKey: 'requestqueuetwo',
506+
});
507+
508+
// Get all requests from the queue and check one locked by the first Actor.
509+
const requests = await requestQueueClientTwo.listRequests();
510+
const requestLockedByClientOne = requests.items.filter((request) => request.lockedByClientKey === 'requestqueueone');
511+
const theFirstRequestLockedByClientOne = requestLockedByClientOne[0];
512+
464513
// Other clients cannot list and lock these requests; the listAndLockHead call returns other requests from the queue.
465514
const processingRequestsClientTwo = await requestQueueClientTwo.listAndLockHead(
466515
{
467-
limit: 2,
516+
limit: 10,
468517
lockSecs: 60,
469518
},
470519
);
471-
472-
// Checks when the lock will expire. The locked request will have a lockExpiresAt attribute.
473-
const theFirstRequestLockedByClientOne = processingRequestsClientOne.items[0];
474-
const requestLockedByClientOne = await requestQueueClientOne.getRequest(
475-
theFirstRequestLockedByClientOne.id,
520+
const wasTheClientTwoLockedSameRequest = !!processingRequestsClientTwo.items.find(
521+
(request) => request.id === theFirstRequestLockedByClientOne.id,
476522
);
523+
524+
console.log(`Was the request locked by the first client locked by the second client? ${wasTheClientTwoLockedSameRequest}`);
477525
console.log(`Request locked until ${requestLockedByClientOne?.lockExpiresAt}`);
478526
479527
// Other clients cannot modify the lock; attempting to do so will throw an error.
@@ -486,21 +534,13 @@ try {
486534
// This will throw an error.
487535
}
488536
489-
// Prolongs the lock of the first request or unlocks it.
490-
await requestQueueClientOne.prolongRequestLock(
491-
theFirstRequestLockedByClientOne.id,
492-
{ lockSecs: 60 },
493-
);
494-
await requestQueueClientOne.deleteRequestLock(
495-
theFirstRequestLockedByClientOne.id,
496-
);
497-
498-
// Cleans up the queue.
499-
await requestQueueClientOne.delete();
500537
501538
await Actor.exit();
502539
```
503540
541+
</TabItem>
542+
</Tabs>
543+
504544
A detailed tutorial on how to process one request queue with multiple Actor runs can be found in [Academy tutorials](https://docs.apify.com/academy/node-js/multiple-runs-scrape).
505545
506546
## Sharing

0 commit comments

Comments
 (0)