refactored to SDK 1.3.1

Author: metalwarrior665

.gitignore

@@ -1,2 +1,3 @@
 apify_storage
-node_modules
+node_modules
+apify_storage

CHANGELOG.md

@@ -0,0 +1,9 @@
+### 2021-07-29
+*Features*
+- Pushing metadata about each page to dataset
+
+*Changes*
+- Removed `useGoogleBotHeaders` option (we don't want to impersonate Google anyway)
+- Updated `apify` from `0.18.1` to `1.3.1`
+- `saveSnapshots` is `true` by default
+- `success` and `wasSuccess` metrics added to output. Success is measured by status being less than 400 and no captcha

INPUT_SCHEMA.json

@@ -92,17 +92,11 @@
             "description": "Only works for puppeteer type. Will wait on each page. You can provide number in ms or a selector.",
             "editor": "textfield"
         },
-        "useGoogleBotHeaders": {
-            "title": "Use Google Bot headers",
-            "type": "boolean",
-            "description": "This option will allow you to bypass protection and/or paywall on some sites. Use with caution as it might get blocked.",
-            "default": false
-        },
         "saveSnapshots": {
             "title": "Save Snapshots",
             "type": "boolean",
             "description": "Will save HTML for Cheerio and HTML + screenshot for Puppeteer",
-            "default": false
+            "default": true
         }
     },
     "required": ["startUrls"]