Merge pull request #416 from apify/sdk-crawlee-examples

docs(academy): Update references to Apify SDK with Crawlee

Author: mstephen19

content/academy/anti_scraping/mitigation/generating_fingerprints.md

@@ -85,8 +85,8 @@ const page = await context.newPage();
 await page.goto('https://google.com');
 ```
 
-> Note that the Apify SDK automatically applies wide variety fingerprints by default, so it is not required to do this unless you aren't using the Apify SDK or if you need a super specific custom fingerprint to scrape with.
+> Note that [Crawlee](https://crawlee.dev) automatically applies wide variety of fingerprints by default, so it is not required to do this unless you aren't using Crawlee or if you need a super specific custom fingerprint to scrape with.
 
-## [](#next) Next up
+## Wrap up
 
 That's it for the **Mitigation** course for now, but be on the lookout for future lessons! We release lessons as we write them, and will be updating the Academy frequently, so be sure to check back every once in a while for new content! Alternatively, you can subscribe to our mailing list to get periodic updates on the Academy, as well as what Apify is up to.

content/academy/anti_scraping/mitigation/proxies.md

@@ -45,4 +45,4 @@ Web scrapers can implement a method called "proxy rotation" to **rotate** the IP
 
 ## [](#next) Next up
 
-Proxies are one of the most important things to understand when it comes to mitigating anti-scraping techniques in a scraper. Now that you're familiar with what they are, the next lesson will be teaching you how to configure your crawler in the Apify SDK to use and automatically rotate proxies. [Let's get right into it!]({{@link anti_scraping/mitigation/using_proxies.md}})
+Proxies are one of the most important things to understand when it comes to mitigating anti-scraping techniques in a scraper. Now that you're familiar with what they are, the next lesson will be teaching you how to configure your crawler in Crawlee to use and automatically rotate proxies. [Let's get right into it!]({{@link anti_scraping/mitigation/using_proxies.md}})

content/academy/anti_scraping/mitigation/using_proxies.md

@@ -1,84 +1,82 @@
 ---
 title: Using proxies
-description: Learn how to use and automagically rotate proxies in your scrapers by using the Apify SDK, and a bit about how to easily obtain pools of proxies.
+description: Learn how to use and automagically rotate proxies in your scrapers by using Crawlee, and a bit about how to easily obtain pools of proxies.
 menuWeight: 2
 paths:
 - anti-scraping/mitigation/using-proxies
 ---
 
 # [](#using-proxies) Using proxies
 
-In the [**Web scraping for beginners**]({{@link web_scraping_for_beginners.md}}) course, we learned about the power of the Apify SDK, and how it can streamline the development process of web crawlers. You've already seen how powerful the `apify` package is; however, what you've been exposed to thus far is only the tip of the iceberg.
+In the [**Web scraping for beginners**]({{@link web_scraping_for_beginners/crawling/pro_scraping.md}}) course, we learned about the power of Crawlee, and how it can streamline the development process of web crawlers. You've already seen how powerful the `crawlee` package is; however, what you've been exposed to thus far is only the tip of the iceberg.
 
-Because proxies are so widely used in the scraping world, we at Apify have equipped our SDK with features which make it easy to implement them in an effective way. One of the main functionalities that comes baked into the SDK is proxy rotation, which is when each request is sent through a different proxy from a proxy pool.
+Because proxies are so widely used in the scraping world, Crawlee as been equipped with features which make it easy to implement them in an effective way. One of the main functionalities that comes baked into Crawlee is proxy rotation, which is when each request is sent through a different proxy from a proxy pool.
 
 ## [](#implementing-proxies) Implementing proxies in a scraper
 
 Let's borrow some scraper code from the end of the [pro-scraping]({{@link web_scraping_for_beginners/crawling/pro_scraping.md}}) lesson in our **Web Scraping for Beginners** course and paste it into a new file called **proxies.js**. This code enqueues all of the product links on [demo-webstore.apify.org](https://demo-webstore.apify.org)'s on-sale page, then makes a request to each product page and scrapes data about each one:
 
 ```JavaScript
-// proxies.js
-import Apify from 'apify';
-
-await Apify.utils.purgeLocalStorage();
-
-const requestQueue = await Apify.openRequestQueue();
-await requestQueue.addRequest({
-    url: 'https://demo-webstore.apify.org/search/on-sale',
-    userData: {
-        label: 'START',
-    },
-});
-
-const crawler = new Apify.CheerioCrawler({
-    requestQueue,
-    handlePageFunction: async ({ $, request }) => {
-        if (request.userData.label === 'START') {
-            await Apify.utils.enqueueLinks({
-                $,
-                requestQueue,
-                selector: 'a[href*="/product/"]',
-                baseUrl: new URL(request.url).origin,
+// crawlee.js
+import { CheerioCrawler, Dataset } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+    requestHandler: async ({ $, request, enqueueLinks }) => {
+        if (request.label === 'START') {
+            await enqueueLinks({
+                selector: 'a[href*="/product/"]'
             });
+
+            // When on the START page, we don't want to
+            // extract any data after we extract the links.
             return;
         }
 
+        // We copied and pasted the extraction code
+        // from the previous lesson
         const title = $('h3').text().trim();
         const price = $('h3 + div').text().trim();
         const description = $('div[class*="Text_body"]').text().trim();
 
-        await Apify.pushData({
+        // Instead of saving the data to a variable,
+        // we immediately save everything to a file.
+        await Dataset.pushData({
             title,
             description,
             price,
         });
     },
 });
 
+await crawler.addRequests([{
+    url: 'https://demo-webstore.apify.org/search/on-sale',
+    // By labeling the Request, we can very easily
+    // identify it later in the requestHandler.
+    label: 'START',
+}]);
+
 await crawler.run();
 ```
 
-In order to implement a proxy pool, we will first need some proxies. We'll quickly use the free [proxy scraper](https://apify.com/mstephen190/proxy-scraper) on the Apify platform to get our hands on some quality proxies. Next, we'll need to set up a [`proxyConfiguration`](https://sdk.apify.com/docs/api/proxy-configuration#docsNav) and configure it with our custom proxies, like so:
+In order to implement a proxy pool, we will first need some proxies. We'll quickly use the free [proxy scraper](https://apify.com/mstephen190/proxy-scraper) on the Apify platform to get our hands on some quality proxies. Next, we'll need to set up a [`ProxyConfiguration`](https://crawlee.dev/api/core/class/ProxyConfiguration) and configure it with our custom proxies, like so:
 
 ```JavaScript
-const proxyConfiguration = await Apify.createProxyConfiguration({
+import { ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({
     proxyUrls: ['http://45.42.177.37:3128', 'http://43.128.166.24:59394', 'http://51.79.49.178:3128'],
 });
 ```
 
 Awesome, so there's our proxy pool! Usually, a proxy pool is much larger than this; however, a three proxie pool is total fine for tutorial purposes. Finally, we can pass the `proxyConfiguration` into our crawler's options:
 
 ```JavaScript
-const crawler = new Apify.CheerioCrawler({
+const crawler = new CheerioCrawler({
     proxyConfiguration,
-    requestQueue,
-    handlePageFunction: async ({ $, request }) => {
-        if (request.userData.label === 'START') {
-            await Apify.utils.enqueueLinks({
-                $,
-                requestQueue,
+    requestHandler: async ({ $, request, enqueueLinks }) => {
+        if (request.label === 'START') {
+            await enqueueLinks({
                 selector: 'a[href*="/product/"]',
-                baseUrl: new URL(request.url).origin,
             });
             return;
         }
@@ -87,7 +85,7 @@ const crawler = new Apify.CheerioCrawler({
         const price = $('h3 + div').text().trim();
         const description = $('div[class*="Text_body"]').text().trim();
 
-        await Apify.pushData({
+        await Dataset.pushData({
             title,
             description,
             price,
@@ -96,7 +94,7 @@ const crawler = new Apify.CheerioCrawler({
 });
 ```
 
-> Note that if you run this code, it may not work, as the proxies could potentially be down at the time you are going through this course.
+> Note that if you run this code, it may not work, as the proxies could potentially be down/non-operating at the time you are going through this course.
 
 That's it! The crawler will now automatically rotate through the proxies we provided in the `proxyUrls` option.
 
@@ -105,9 +103,8 @@ That's it! The crawler will now automatically rotate through the proxies we prov
 At the time of writing, our above scraper utilizing our custom proxy pool is working just fine. But how can we check that the scraper is for sure using the proxies we provided it, and more importantly, how can we debug proxies within our scraper? Luckily, within the same `context` object we've been destructuring `$` and `request` out of, there is a `proxyInfo` key as well. `proxyInfo` is an object which includes useful data about the proxy which was used to make the request.
 
 ```JavaScript
-const crawler = new Apify.CheerioCrawler({
+const crawler = new CheerioCrawler({
     proxyConfiguration,
-    requestQueue,
     // Destructure "proxyInfo" from the "context" object
     handlePageFunction: async ({ $, request, proxyInfo }) => {
         // Log its value
@@ -122,15 +119,16 @@ After modifying your code to log `proxyInfo` to the console and running the scra
 
 ![proxyInfo being logged by the scraper]({{@asset anti_scraping/mitigation/images/proxy-info-logs.webp}})
 
-These logs confirm that our proxies are being used and rotated successfully by the Apify SDK, and can also be used to debug slow or broken proxies.
+These logs confirm that our proxies are being used and rotated successfully by Crawlee, and can also be used to debug slow or broken proxies.
 
 ## [](#higher-level-proxy-scraping) Higher level proxy scraping
 
-Though we will discuss it more in-depth in future courses, it is still important to mention that the Apify SDK has integrated support for [Apify Proxy](https://apify.com/proxy), which is a service that provides access to pools of both residential and datacenter IP addresses. A `proxyConfiguration` using Apify Proxy might look something like this:
+Though we will discuss it more in-depth in future courses, it is still important to mention that Crawlee has integrated support for the Apify SDK, which supports [Apify Proxy](https://apify.com/proxy) - a service that provides access to pools of both residential and datacenter IP addresses. A `proxyConfiguration` using Apify Proxy might look something like this:
 
 ```JavaScript
-const proxyConfiguration = await Apify.createProxyConfiguration({
-    groups: ['SHADER'],
+import { Actor } from 'apify';
+
+const proxyConfiguration = await  Actor.createProxyConfiguration({
     countryCode: 'US'
 });
 ```

content/academy/anti_scraping/techniques/rate_limiting.md

@@ -18,15 +18,16 @@ In cases when a higher number of requests is expected for the crawler, using a [
 
 The most popular and effective way of avoiding rate-limiting issues is by rotating [proxies]({{@link anti_scraping/mitigation/proxies.md}}) after every **n** number of requests, which makes your scraper appear as if it is making requests from various different places. Since the majority of rate-limiting solutions are based on IP addresses, rotating IPs allows a scraper to make large amounts to a website without getting restricted.
 
-In the Apify SDK, proxies are automatically rotated for you when you use `proxyConfiguration` and a [**SessionPool**]((https://sdk.apify.com/docs/api/session-pool)) within a crawler. The SessionPool handles a lot of the nitty gritty of proxy rotating, especially with [browser based crawlers]({{@link puppeteer_playwright.md}}) by retiring a browser instance after a certain number of requests have been sent from it in order to use a new proxy (a browser instance must be retired in order to use a new proxy).
+In Crawlee, proxies are automatically rotated for you when you use `ProxyConfiguration` and a [**SessionPool**](https://crawlee.dev/api/core/class/SessionPool) within a crawler. The SessionPool handles a lot of the nitty gritty of proxy rotating, especially with [browser based crawlers]({{@link puppeteer_playwright.md}}) by retiring a browser instance after a certain number of requests have been sent from it in order to use a new proxy (a browser instance must be retired in order to use a new proxy).
 
 Here is an example of these features being used in a **PuppeteerCrawler** instance:
 
 ```JavaScript
-import Apify from 'apify';
+import { PuppeteerCrawler } from 'crawlee';
+import { Actor } from 'apify';
 
-const myCrawler = new Apify.PuppeteerCrawler({
-    proxyConfiguration: await Apify.createProxyConfiguration({
+const myCrawler = new PuppeteerCrawler({
+    proxyConfiguration: await Actor.createProxyConfiguration({
         groups: ['RESIDENTIAL'],
     }),
     sessionPoolOptions: {
@@ -44,17 +45,17 @@ const myCrawler = new Apify.PuppeteerCrawler({
 });
 ```
 
-> Take a look at the [**Using proxies**]({{@link anti_scraping/mitigation/using_proxies.md}}) lesson to learn more about how to use proxies and rotate them in the Apify SDK.
+> Take a look at the [**Using proxies**]({{@link anti_scraping/mitigation/using_proxies.md}}) lesson to learn more about how to use proxies and rotate them in Crawlee.
 
 ### [](#configuring-session-pool) Configuring a session pool
 
 There are various configuration options available in `sessionPoolOptions` that can be used to set up the SessionPool for different rate-limiting scenarios. In the example above, we used `maxUsageCount` within `sessionOptions` to prevent more than 15 requests from being sent using a session before it was thrown away; however, a maximum age can also be set using `maxAgeSecs`.
 
 When dealing with frequent and unpredictable blockage, the `maxErrorScore` option can be set to trash a session after it's hit a certain number of errors.
 
-To learn more about all configurations available in `sessionPoolOptions`, refer to the [SDK documentation](https://sdk.apify.com/docs/typedefs/session-pool-options).
+To learn more about all configurations available in `sessionPoolOptions`, refer to the [Crawlee documentation](https://crawlee.dev/api/core/interface/SessionPoolOptions).
 
-> Don't worry too much about these configurations. The Apify SDK's defaults are usually good enough for the majority of use cases.
+> Don't worry too much about these configurations. Crawlee's defaults are usually good enough for the majority of use cases.
 
 ## [](#next) Next up
 

content/academy/api_scraping/general_api_scraping/handling_pagination.md

@@ -138,7 +138,7 @@ while (items.flat().length < 100) {
 
 All that's left to do now is flesh out this `while` loop with pagination logic and finally return the **items** array once the loop has finished.
 
-> Note that it's better to add requests to a requests queue rather than processing them in memory. The crawlers offered by the [Apify SDK](https://sdk.apify.com) provide this functionality out of the box.
+> Note that it's better to add requests to a requests queue rather than processing them in memory. The crawlers offered by [Crawlee](https://crawlee.dev/docs/) provide this functionality out of the box.
 
 ```JavaScript
 // index.js

content/academy/apify_platform/deploying_your_code/deploying.md

@@ -50,7 +50,7 @@ That's it! the actor should now pull its source code from the repo and automatic
 
 If you're logged in to the Apify CLI, the `apify push` command can be used to push the code straight onto the Apify platform from your local machine (no GitHub repository required), where it will automatically be built for you. Prior to running this command, make sure that you have an **apify.json** file at the root of the project. If you don't already have one, you can use `apify init .` to automatically generate one for you.
 
-One important thing to note is that you can use a `.gitignore` file to exclude files from being pushed. When you use `apify push` without a `.gitignore`, the full folder contents will be pushed, meaning that even the even **apify_storage** and **node_modules** will be pushed. These files are unnecessary to push, as they are both generated on the platform.
+One important thing to note is that you can use a `.gitignore` file to exclude files from being pushed. When you use `apify push` without a `.gitignore`, the full folder contents will be pushed, meaning that even the even **storage** and **node_modules** will be pushed. These files are unnecessary to push, as they are both generated on the platform.
 
 > The `apify push` command should only really be used for quickly pushing and testing actors on the platform during development. If you are ready to make your actor public, use a Git repository instead, as you will reap the benefits of using Git and others will be able to contribute to the project.