Skip to content

Commit bd45dd9

Browse files
authored
Update src/content/docs/autorag/configuration/data-source/website.mdx
1 parent 14379cc commit bd45dd9

File tree

1 file changed

+1
-1
lines changed
  • src/content/docs/autorag/configuration/data-source

1 file changed

+1
-1
lines changed

src/content/docs/autorag/configuration/data-source/website.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Refer to [Onboard a domain](/fundamentals/manage-domains/add-site/) for more inf
1414
:::
1515

1616
:::caution[Bot protection may block crawling]
17-
If you use Cloudflare products that control or restrict bot traffic such as [Bot Management](/bots/), [WAF](/waf/), or [Turnstile](/turnstile/), the same rules will apply to the AutoRAG crawler. Make sure to configure an exception or allow-list for the AutoRAG crawler in your settings.
17+
If you use Cloudflare products that control or restrict bot traffic such as [Bot Management](/bots/), [WAF](/waf/), or [Turnstile](/turnstile/), the same rules will apply to the AutoRAG crawler. Make sure to configure an exception or an allow-list for the AutoRAG crawler in your settings.
1818
:::
1919

2020
## How website crawling works

0 commit comments

Comments
 (0)