You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The `/crawl` endpoint automates the process of scraping content from webpages starting with a single URL and crawling to a specified number or depth of links. The response can be returned in either HTML, Markdown, or JSON.
11
11
@@ -39,14 +39,14 @@ If you are on a Workers Free plan, your crawl may fail if it hits the [limit of
39
39
### Initiate the crawl job
40
40
41
41
Here are the basic parameters you can use to initiate your crawl job:
42
-
-`url`— (Required) Starts crawling from this URL
43
-
-`limit` — (Optional) Maximum number of pages to crawl (default is 10, maximum is 100,000)
44
-
-`depth` — (Optional) Maximum link depth to crawl from the starting URL
45
-
-`formats` — (Optional) Response format (default is HTML, other options are Markdown and JSON)
46
-
47
-
The API will respond immediately with a job `id` you will use to retrieve the status and results of the crawl job.
48
-
49
-
See the [advanced usage section below](/browser-rendering/rest-api/crawl-endpoint/#initiate-the-crawl-job) for additional parameters.
0 commit comments