notes and disclaimers

Author: aninibread

src/content/docs/autorag/configuration/data-source/website.mdx

@@ -13,6 +13,10 @@ You can only crawl domains that you have onboarded onto the same Cloudflare acco
 Refer to [Onboard a domain](/fundamentals/manage-domains/add-site/) for more information on adding a domain to your Cloudflare account.
 :::
 
+:::caution[Bot protection may block crawling]
+If you use Cloudflare products that control or restrict bot traffic such as [Bot Management](/bots/), [WAF](/waf/), or [Turnstile](/turnstile/), the same rules will apply to the AutoRAG crawler. Make sure to configure an exception or allow-list for the AutoRAG crawler in your settings.
+:::
+
 ## How website crawling works
 When you connect a domain, the crawler looks for your website’s sitemap to determine which pages to visit:  
 

src/content/docs/autorag/tutorial/brower-rendering-autorag-tutorial.mdx

@@ -9,6 +9,13 @@ import { TypeScriptExample } from "~/components";
 
 AutoRAG is designed to work out of the box with data in R2 buckets. But what if your content lives on a website or needs to be rendered dynamically?
 
+:::note
+
+AutoRAG now lets you use your [website](/autorag/configuration/data-source/website/) as a data source. When enabled, AutoRAG will automatically crawl and parse your site content for you.
+
+:::
+
+
 In this tutorial, we’ll walk through how to:
 
 1. Render your website using Cloudflare's Browser Rendering API