-
Notifications
You must be signed in to change notification settings - Fork 137
fix(request-queue): Update request queue locking docs #1322
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
df52238
dee9669
c00842f
d6dd1af
540a8a6
983ad20
8e1f283
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -407,7 +407,20 @@ You can lock a request so that no other clients receive it when they fetch the q | |
| This feature is seamlessly integrated into Crawlee, requiring minimal extra setup. By default, requests are locked for the same duration as the timeout for processing requests in the crawler ([`requestHandlerTimeoutSecs`](https://crawlee.dev/api/next/basic-crawler/interface/BasicCrawlerOptions#requestHandlerTimeoutSecs)). | ||
| If the Actor processing the request fails, the lock expires, and the request is processed again eventually. For more details, refer to the [Crawlee documentation](https://crawlee.dev/docs/next/experiments/experiments-request-locking). | ||
|
|
||
| In the following example, we demonstrate how we can use locking mechanisms to avoid concurrent processing of the same request. | ||
| In the following example, we demonstrate how we can use locking mechanisms to avoid concurrent processing of the same request across multiple Actor runs. | ||
|
|
||
| :::info | ||
| The lock mechanism works on the client level, as well as the run level, when running the Actor on the Apify platform. | ||
|
|
||
| This means you can unlock or prolong the lock the locked request only if: | ||
|
|
||
| 1. You are using the same client key, or | ||
| 2. The operation is being called from the same Actor run. | ||
drobnikj marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ::: | ||
|
|
||
| <Tabs groupId="main"> | ||
| <TabItem value="Actor 1" label="Actor 1"> | ||
|
|
||
| ```js | ||
| import { Actor, ApifyClient } from 'apify'; | ||
|
|
@@ -425,9 +438,6 @@ const requestQueue = await client.requestQueues().getOrCreate('example-queue'); | |
| const requestQueueClientOne = client.requestQueue(requestQueue.id, { | ||
| clientKey: 'requestqueueone', | ||
| }); | ||
| const requestQueueClientTwo = client.requestQueue(requestQueue.id, { | ||
| clientKey: 'requestqueuetwo', | ||
| }); | ||
|
|
||
| // Adds multiple requests to the queue. | ||
| await requestQueueClientOne.batchAddRequests([ | ||
|
|
@@ -457,23 +467,71 @@ await requestQueueClientOne.batchAddRequests([ | |
| const processingRequestsClientOne = await requestQueueClientOne.listAndLockHead( | ||
| { | ||
| limit: 2, | ||
| lockSecs: 60, | ||
| lockSecs: 120, | ||
| }, | ||
| ); | ||
|
|
||
| // Checks when the lock will expire. The locked request will have a lockExpiresAt attribute. | ||
| const theFirstRequestLockedByClientOne = processingRequestsClientOne.items[0]; | ||
| const requestLockedByClientOne = await requestQueueClientOne.getRequest( | ||
|
||
| theFirstRequestLockedByClientOne.id, | ||
| ); | ||
| console.log(`Request locked until ${requestLockedByClientOne?.lockExpiresAt}`); | ||
|
|
||
| // Prolongs the lock of the first request or unlocks it. | ||
| await requestQueueClientOne.prolongRequestLock( | ||
| theFirstRequestLockedByClientOne.id, | ||
| { lockSecs: 120 }, | ||
| ); | ||
| await requestQueueClientOne.deleteRequestLock( | ||
| theFirstRequestLockedByClientOne.id, | ||
| ); | ||
|
|
||
| // Cleans up the queue. | ||
| await requestQueueClientOne.delete(); | ||
|
|
||
| await Actor.exit(); | ||
| ``` | ||
|
|
||
| </TabItem> | ||
| <TabItem value="Actor 2" label="Actor 2"> | ||
|
|
||
| ```js | ||
| import { Actor, ApifyClient } from 'apify'; | ||
|
|
||
| await Actor.init(); | ||
|
|
||
| const client = new ApifyClient({ | ||
| token: 'MY-APIFY-TOKEN', | ||
| }); | ||
|
|
||
| // Waits for the first Actor to lock the requests. | ||
| await new Promise((resolve) => setTimeout(resolve, 5000)); | ||
|
|
||
| // Creates a new request queue. | ||
| const requestQueue = await client.requestQueues().getOrCreate('example-queue'); | ||
|
|
||
| const requestQueueClientTwo = client.requestQueue(requestQueue.id, { | ||
| clientKey: 'requestqueuetwo', | ||
| }); | ||
|
|
||
| // Get all requests from the queue and check one locked by the first Actor. | ||
| const requests = await requestQueueClientTwo.listRequests(); | ||
| const requestLockedByClientOne = requests.items.filter((request) => request.lockedByClientKey === 'requestqueueone'); | ||
| const theFirstRequestLockedByClientOne = requestLockedByClientOne[0]; | ||
|
|
||
| // Other clients cannot list and lock these requests; the listAndLockHead call returns other requests from the queue. | ||
| const processingRequestsClientTwo = await requestQueueClientTwo.listAndLockHead( | ||
| { | ||
| limit: 2, | ||
| limit: 10, | ||
| lockSecs: 60, | ||
| }, | ||
| ); | ||
|
|
||
| // Checks when the lock will expire. The locked request will have a lockExpiresAt attribute. | ||
| const theFirstRequestLockedByClientOne = processingRequestsClientOne.items[0]; | ||
| const requestLockedByClientOne = await requestQueueClientOne.getRequest( | ||
| theFirstRequestLockedByClientOne.id, | ||
| const wasTheClientTwoLockedSameRequest = !!processingRequestsClientTwo.items.find( | ||
|
||
| (request) => request.id === theFirstRequestLockedByClientOne.id, | ||
| ); | ||
|
|
||
| console.log(`Was the request locked by the first client locked by the second client? ${wasTheClientTwoLockedSameRequest}`); | ||
| console.log(`Request locked until ${requestLockedByClientOne?.lockExpiresAt}`); | ||
|
|
||
| // Other clients cannot modify the lock; attempting to do so will throw an error. | ||
|
|
@@ -486,21 +544,12 @@ try { | |
| // This will throw an error. | ||
| } | ||
|
|
||
| // Prolongs the lock of the first request or unlocks it. | ||
| await requestQueueClientOne.prolongRequestLock( | ||
| theFirstRequestLockedByClientOne.id, | ||
| { lockSecs: 60 }, | ||
| ); | ||
| await requestQueueClientOne.deleteRequestLock( | ||
| theFirstRequestLockedByClientOne.id, | ||
| ); | ||
|
|
||
| // Cleans up the queue. | ||
| await requestQueueClientOne.delete(); | ||
|
|
||
| await Actor.exit(); | ||
| ``` | ||
|
|
||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| A detailed tutorial on how to process one request queue with multiple Actor runs can be found in [Academy tutorials](https://docs.apify.com/academy/node-js/multiple-runs-scrape). | ||
|
|
||
| ## Sharing | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.