Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ Emitted when a task is queued via [Cluster.queue] or [Cluster.execute]. The firs
- `retryDelay` <[number]> How much time should pass at minimum between the job execution and its retry. Ignored by tasks queued via [Cluster.execute]. Defaults to `0`.
- `sameDomainDelay` <[number]> How much time should pass at minimum between two requests to the same domain. If you use this field, the queued `data` must be your URL or `data` must be an object containing a field called `url`.
- `skipDuplicateUrls` <[boolean]> If set to `true`, will skip URLs which were already crawled by the cluster. Defaults to `false`. If you use this field, the queued `data` must be your URL or `data` must be an object containing a field called `url`.
- `timeout` <[number]> Specify a timeout for all tasks. Defaults to `30000` (30 seconds).
- `timeout` <[number]> Specify a timeout for all tasks. `0` disable, defaults to `30000` (30 seconds).
- `monitor` <[boolean]> If set to `true`, will provide a small command line output to provide information about the crawling process. Defaults to `false`.
- `workerCreationDelay` <[number]> Time between creation of two workers. Set this to a value like `100` (0.1 seconds) in case you want some time to pass before another worker is created. You can use this to prevent a network peak right at the start. Defaults to `0` (no delay).
- `puppeteer` <[Object]> In case you want to use a different puppeteer library (like [puppeteer-core](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteer-vs-puppeteer-core) or [puppeteer-extra](https://github.com/berstend/puppeteer-extra)), pass the object here. If not set, will default to using puppeteer. When using `puppeteer-core`, make sure to also provide `puppeteerOptions.executablePath`.
Expand Down
23 changes: 12 additions & 11 deletions src/Worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,18 +91,19 @@ export default class Worker<JobData, ReturnData> implements WorkerOptions {

let result: any;
try {
result = await timeoutExecute(
const taskToLaunch = task({
page,
// data might be undefined if queue is only called with a function
// we ignore that case, as the user should use Cluster<undefined> in that case
// to get correct typings
data: job.data as JobData,
worker: {
id: this.id,
},
});
result = timeout === 0 ? await taskToLaunch : await timeoutExecute(
timeout,
task({
page,
// data might be undefined if queue is only called with a function
// we ignore that case, as the user should use Cluster<undefined> in that case
// to get correct typings
data: job.data as JobData,
worker: {
id: this.id,
},
}),
taskToLaunch,
);
} catch (err) {
errorState = err;
Expand Down