|
1 | 1 | ## Website Checker
|
2 | 2 |
|
3 |
| -Website checker is a simple actor that allows you to scan any website for performance and blocking. |
| 3 | +Website checker is a simple actor that allows you to scan any website for performance and blocking using various scraping methods as Cheerio, Puppeteer and Playwright. |
4 | 4 |
|
5 | 5 | ### Features
|
6 | 6 |
|
7 | 7 | The actor provides these useful features out of the box:
|
8 | 8 |
|
9 | 9 | - Collects response status codes
|
10 | 10 | - Recognizes the most common captchas
|
11 |
| -- Saves HTML snapshots and screenshots (if Puppeteer is chosen) |
12 |
| -- Enables choosing between Cheerio (plain HTTP) and Puppeteer (browser) scraper |
| 11 | +- Saves HTML snapshots and screenshots (if Puppeteer or Playwright is chosen) |
| 12 | +- Enables choosing between Cheerio (plain HTTP) and Puppeteer/Playwright (browser) scraper |
| 13 | +- Enables choosing different browsers for Playwright - Chrome, Firefox and Webkit (Safari) |
13 | 14 | - Enables re-scraping start URLs or enqueueing with a familiar link selector + pseudo URLs system
|
14 | 15 | - Handles different failure states like timeouts and network errors
|
15 | 16 | - Enables basic proxy and browser configuration
|
16 | 17 |
|
17 |
| -#### Planned features |
18 |
| - |
19 |
| -- Usage calculation/stats |
20 |
| -- Better automatic workloads/workload actors |
21 |
| -- Add support for Playwright + Firefox |
22 |
| - |
23 | 18 | ### How to use
|
24 | 19 |
|
25 |
| -The most common use-case is to do a quick check on how aggressively the target site is blocking. In that case just supply a start URL, ideally a category one or product one. You can either set `replicateStartUrls` or add enqueueing with `linkSelector` + `pseudoUrls`, both are good options to test different proxies. You can test a few different proxy groups and compare `cheerio` vs `puppeteer` options. |
| 20 | +The most common use-case is to do a quick check on how aggressively the target site is blocking. In that case just supply a start URL, ideally a category one or product one. You can either set `replicateStartUrls` or add enqueueing with `linkSelector` + `pseudoUrls`, both are good options to test different proxies. |
26 | 21 |
|
27 |
| -In the end you will get a simple statistics about the blocking rate. It is recommended to check a few screenshots just to make sure the actor correctly recognized the page status. You can get to the detailed output (per URL) via KV store or dataset (the KV output sorts by response status while dataset is simply ordered by scraping order). |
| 22 | +You can pick any combination of run options and the checker will spawn runner actor for every combination of scraping tool & proxies and then combine the results into single output. |
28 | 23 |
|
29 |
| -#### Checker workloads |
30 |
| - |
31 |
| -To make your life easier, you can use other actors that will start more checker runs at once and aggregate the result. This way you can test more sites at once or different cheerio/browser and proxy combinations and compare those. |
| 24 | +In the end you will get a simple statistics about the blocking rate. It is recommended to check a few screenshots just to make sure the actor correctly recognized the page status. You can get to the detailed output (per URL) via KV store or dataset (the KV output sorts by response status while dataset is simply ordered by scraping order). |
32 | 25 |
|
33 |
| -All of these actors are very young so we are glad for any feature ideas: |
34 |
| -[lukaskrivka/website-checker-workload](https://apify.com/lukaskrivka/website-checker-workload) |
35 |
| -[vaclavrut/website-checker-starter](https://apify.com/vaclavrut/website-checker-starter) |
| 26 | +#### Multiple URLs and configurations |
| 27 | +Website checker doesn't have any limitation of how many websites and configs you can check. For each website, it will run each config. You just need to set a reasonable `maxConcurrentDomainsChecked` so that all parallel runs fit into your total memory (4 GB for Cheerio and 8 GB for Puppeteer/Playwright checks). |
36 | 28 |
|
37 | 29 | ### Input
|
38 | 30 |
|
|
0 commit comments