docs: address Michal's remarks

Author: metalwarrior665

sources/academy/tutorials/node_js/scraping_from_sitemaps.md

@@ -9,9 +9,13 @@ import Example from '!!raw-loader!roa-loader!./scraping_from_sitemaps.js';
 
 # How to scrape from sitemaps {#scraping-with-sitemaps}
 
->Crawlee recently introduced a new feature that allows you to scrape sitemaps with ease. If you are using Crawlee, you can skip the following steps and just gather all the URLs from the sitemap in a few lines of code:
+:::note
 
-```javascript
+Crawlee allows you to scrape sitemaps with ease. If you are using Crawlee, you can skip the following steps and just gather all the URLs from the sitemap in a few lines of code.
+
+:::
+
+```js
 import { RobotsFile } from 'crawlee';
 
 const robots = await RobotsFile.find('https://www.mysite.com');

sources/academy/webscraping/advanced_web_scraping/crawling/crawling-sitemaps.md

@@ -1,9 +1,8 @@
 ---
 title: Crawling sitemaps
 description: Learn how to extract all of a website's listings even if they limit the number of results pages. See code examples for setting up your scraper.
-menuWeight: 2
-paths:
-- advanced-web-scraping/crawling/crawling-sitemaps
+sidebar_position:: 2
+slug: /advanced-web-scraping/crawling/crawling-sitemaps
 ---
 
 In the previous lesson, we learned what is the utility (and dangers) of crawling sitemaps. In this lesson, we will go in-depth to how to crawl sitemaps.
@@ -15,19 +14,19 @@ We will look at the following topics:
 - How to parse URLs from sitemaps
 - Using Crawlee to get all URLs in a few lines of code
 
-## [](#how-to-find-sitemap-urls) How to find sitemap URLs
+## How to find sitemap URLs
 
 Sitemaps are commonly restricted to contain a maximum of 50k URLs so usually, there will be a whole list of them. There can be a master sitemap containing URLs of all other sitemaps or the sitemaps might simply be indexed in robots.txt and/or have auto-incremented URLs like `/sitemap1.xml`, `/sitemap2.xml`, etc.
 
-### [](#google) Google
+### Google
 
 You can try your luck on Google by searching for `site:example.com sitemap.xml` or `site:example.com sitemap.xml.gz` and see if you get any results. If you do, you can try to download the sitemap and see if it contains any useful URLs. The success of this approach depends on the website telling Google to index the sitemap file itself which is rather uncommon.
 
-### [](#robots-txt) robots.txt
+### robots.txt {#robots-txt}
 
 If the website has a robots.txt file, it often contains sitemap URLs. The sitemap URLs are usually listed under `Sitemap:` directive.
 
-### [](#common-url-paths) Common URL paths
+### Common URL paths
 
 You can try to iterate over common URL paths like:
 
@@ -52,26 +51,26 @@ Some websites also provide an HTML version, to help indexing bots find new conte
 
 Apify provides the [Sitemap Sniffer actor](https://apify.com/vaclavrut/sitemap-sniffer) (open-source code), that scans the URL variations automatically for you so that you don't have to check manually.
 
-## [](#how-to-set-up-http-requests-to-download-sitemaps) How to set up HTTP requests to download sitemaps
+## How to set up HTTP requests to download sitemaps
 
 For most sitemaps, you can make a simple HTTP request and parse the downloaded XML text with Cheerio (or just use `CheerioCrawler`). Some sitemaps are compressed and have to be streamed and decompressed. The code for that is fairly complicated so we recommend just [using Crawlee](#using-crawlee) which handles streamed and compressed sitemaps by default.
 
-## [](#how-to-parse-urls-from-sitemaps) How to parse URLs from sitemaps
+## How to parse URLs from sitemaps
 
 The easiest part is to parse the actual URLs from the sitemap. The URLs are usually listed under `<loc>` tags. You can use Cheerio to parse the XML text and extract the URLs. Just be careful that the sitemap might contain other URLs that you don't want to crawl (e.g. /about, /contact, or various special category sections). [This article](/academy/node-js/scraping-from-sitemaps.md) provides code examples for parsing sitemaps.
 
-## [](#using-crawlee) Using Crawlee
+## Using Crawlee
 
 Fortunately, you don't have to worry about any of the above steps if you use [Crawlee](https://crawlee.dev) which has rich traversing and parsing support for sitemap. Crawlee can traverse nested sitemaps, download, and parse compressed sitemaps, and extract URLs from them. You can get all URLs in a few lines of code:
 
-```javascript
+```js
 import { RobotsFile } from 'crawlee';
 
 const robots = await RobotsFile.find('https://www.mysite.com');
 
 const allWebsiteUrls = await robots.parseUrlsFromSitemaps();
 ```
 
-## [](#next) Next up
+## Next up
 
 That's all we need to know about sitemaps for now. Let's dive into a much more interesting topic - search, filters, and pagination.

sources/academy/webscraping/advanced_web_scraping/crawling/crawling-with-search.md

@@ -1,9 +1,8 @@
 ---
 title: Crawling with search 
 description: Learn how to extract all of a website's listings even if they limit the number of results pages. See code examples for setting up your scraper.
-menuWeight: 3
-paths:
-- advanced-web-scraping/crawling/crawling-with-search
+sidebar_position:: 3
+slug: /advanced-web-scraping/crawling/crawling-with-search
 ---
 
 # Scraping websites with search 

sources/academy/webscraping/advanced_web_scraping/crawling/sitemaps-vs-search.md

@@ -1,14 +1,13 @@
 ---
 title: Sitemaps vs search
 description: Learn how to extract all of a website's listings even if they limit the number of results pages. 
-menuWeight: 1
-paths:
-- advanced-web-scraping/crawling/sitemaps-vs-search
+sidebar_position:: 1
+slug: /advanced-web-scraping/crawling/sitemaps-vs-search
 ---
 
 The core crawling problem comes to down to ensuring that we reliably find all detail pages on the target website or inside its categories. This is trivial for small sites. We just open the home page or category pages and paginate to the end as we did in the Web Scraping for Beginners course.
 
-Unfortunately, **most modern websites restrict pagination** only to somewhere between 1 and 10 thousand products. Solving this problem might seem relatively straightforward at first but there are multiple hurdles that we will explore in this lesson.
+Unfortunately, **most modern websites restrict pagination** only to somewhere between 1 and 10 000 products. Solving this problem might seem relatively straightforward at first but there are multiple hurdles that we will explore in this lesson.
 
 There are two main approaches to solving this problem:
 
@@ -58,6 +57,6 @@ Fortunately, most websites list a total number of detail pages somewhere. It mig
 
 Unfortunately, some sites like Amazon do not provide exact numbers. In this case, we have to work with what they give us and put even more effort into making our scraping logic accurate. We will tackle this in the next lessons as well.
 
-## [](#next) Next up
+## Next up
 
 First, we will look into the easier approach, the [sitemap crawling](./crawling-sitemaps.md). Then we will go through all the intricacies of the category, search and filter crawling, and build up a generic framework that we can use on any website. At last, we will combine the results of both approaches and set up monitoring and persistence to ensure we can run this regularly without any manual controls.

sources/academy/webscraping/advanced_web_scraping/index.md

@@ -1,10 +1,9 @@
 ---
 title: Advanced web scraping
 description: Take your scrapers to a production-ready level by learning various advanced concepts and techniques that will help you build highly scalable and reliable crawlers.
-menuWeight: 6
+sidebar_position:: 6
 category: web scraping & automation
-paths:
-- advanced-web-scraping
+slug: /advanced-web-scraping
 ---
 
 # Advanced web scraping
@@ -13,7 +12,7 @@ In [**Web scraping for beginners**](/academy/web-scraping-for-beginners) course,
 
 In this course, we will take all of that knowledge, add a few more advanced concepts, and apply them to learn how to build a production-ready web scraper.
 
-## [](#what-does-production-ready-mean) What does production-ready mean?
+## What does production-ready mean
 
 To scrape large and complex websites, we need to scale two essential aspects of the scraper: crawling and data extraction. Big websites can have millions of pages and the data we want to extract requires more sophisticated parsing techniques than just selecting elements by CSS selectors or using APIs as they are.
 
@@ -27,6 +26,6 @@ We will also touch on monitoring, performance, anti-scraping protections, and de
 
 If you've managed to follow along with all of the courses prior to this one, then you're more than ready to take these upcoming lessons on 😎
 
-## [](#first-up) First up
+## First up
 
 First, we will explore [advanced crawling section](academy/webscraping/advanced-web-scraping/advanced-crawling) that will help us to find all pages or products on the website.