Skip to content

notes and disclaimers #25238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 18, 2025
Merged

notes and disclaimers #25238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
14379cc
notes and disclaimers
aninibread Sep 17, 2025
bd45dd9
Update src/content/docs/autorag/configuration/data-source/website.mdx
Oxyjun Sep 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/content/docs/autorag/configuration/data-source/website.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ You can only crawl domains that you have onboarded onto the same Cloudflare acco
Refer to [Onboard a domain](/fundamentals/manage-domains/add-site/) for more information on adding a domain to your Cloudflare account.
:::

:::caution[Bot protection may block crawling]
If you use Cloudflare products that control or restrict bot traffic such as [Bot Management](/bots/), [WAF](/waf/), or [Turnstile](/turnstile/), the same rules will apply to the AutoRAG crawler. Make sure to configure an exception or an allow-list for the AutoRAG crawler in your settings.
:::

## How website crawling works
When you connect a domain, the crawler looks for your website’s sitemap to determine which pages to visit:

Expand Down
7 changes: 7 additions & 0 deletions src/content/docs/autorag/tutorial/brower-rendering-autorag-tutorial.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,13 @@ import { TypeScriptExample } from "~/components";

AutoRAG is designed to work out of the box with data in R2 buckets. But what if your content lives on a website or needs to be rendered dynamically?

:::note

AutoRAG now lets you use your [website](/autorag/configuration/data-source/website/) as a data source. When enabled, AutoRAG will automatically crawl and parse your site content for you.

:::


In this tutorial, we’ll walk through how to:

1. Render your website using Cloudflare's Browser Rendering API
Expand Down
Loading