fixes

Author: aninibread

src/content/docs/autorag/configuration/data-source/website.mdx

@@ -14,13 +14,13 @@ Refer to [Onboard a domain](/fundamentals/manage-domains/add-site/) for more inf
 :::
 
 ## How website crawling works
-When you connect a domain, the crawler looks for your site’s sitemap to determine which pages to visit:  
+When you connect a domain, the crawler looks for your website’s sitemap to determine which pages to visit:  
 
-1. The crawler first checks for a sitemap at `/sitemap.xml`.  
-2. If no sitemap is found, it checks `robots.txt` for listed sitemaps.  
+1. The crawler first checks the `robots.txt` for listed sitemaps. If it exists, it reads all sitemaps existing inside.
+2. If no `robots.txt` is found, the crawler first checks for a sitemap at `/sitemap.xml`.
 3. If no sitemap is available, the domain cannot be crawled.  
 
-Pages are visited in the order defined by your sitemap.  
+Pages are visited, according to the `<priority>` attribute set on the sitemaps, if this field is defined.
 
 ## Parsing options
 You can choose how pages are parsed during crawling:  
@@ -31,8 +31,12 @@ You can choose how pages are parsed during crawling:
 ## Storage
 During setup, AutoRAG creates a dedicated R2 bucket in your account to store the pages that have been crawled and downloaded as HTML files. This bucket is automatically managed and is used only for content discovered by the crawler. Any files or objects that you add directly to this bucket will not be indexed.  
 
+We recommend not to modify the bucket as it may distrupt the indexing flow and cause content to not be updated properly.
+
 ## Sync and updates
-During scheduled or manual [sync jobs](/autorag/configuration/indexing/), the crawler will check for changes on your website. If a page changes, the updated version is stored in the R2 bucket and automatically reindexed so that your search results always reflect the latest content.
+During scheduled or manual [sync jobs](/autorag/configuration/indexing/), the crawler will check for changes to the `<lastmod>` attribute in your sitemap. If it has been changed to a date occuring after the last sync date, then the page will be crawled, the updated version is stored in the R2 bucket, and automatically reindexed so that your search results always reflect the latest content.
+
+If the `<lastmod>` attribute is not defined, then AutoRAG will automatically crawl each link defined in the sitemap once a day.
 
 ## Limits
 The regular AutoRAG [limits](/autorag/platform/limits-pricing/) apply when using the Website data source.