Skip to content

[Bots] Clarification wording to bots list #22984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 11, 2025
Merged

[Bots] Clarification wording to bots list #22984

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a15fa1f
Adding clarification wording.
Oxyjun Jun 10, 2025
3eaa935
Update src/content/partials/bots/list-ai-bots.mdx
Oxyjun Jun 10, 2025
07af21d
Adding wording about WAF
Oxyjun Jun 11, 2025
2f49253
Merge branch 'jun/ai-audit/bot-list-wording' of github.com:cloudflare…
Oxyjun Jun 11, 2025
1833533
Adding note on bot fight mode.
Oxyjun Jun 11, 2025
80e59de
Cross referencing to bots as well
Oxyjun Jun 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions src/content/docs/ai-audit/get-started.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ description: Learn how to set up AI Audit.

import { Details, Render, Steps } from "~/components";

This guide instructs you through
This guide instructs you on how to:

- Viewing AI crawlers that are interacting with your domain.
- Creating a rule to block AI crawlers on your website.
- View AI crawlers that are interacting with pages in your domain (a [Cloudflare zone](/fundamentals/concepts/accounts-and-zones/#zones)).
- Create a rule to block AI crawlers on your pages.

## Prerequisites

Expand All @@ -36,6 +36,11 @@ To use AI Audit:
4. From the **Bot traffic** page, under **Block AI Bots**, select **Enable**.
</Steps>

This feature also enables Cloudflare's [Bot Fight Mode: Block AI bots](/bots/get-started/bot-fight-mode/#block-ai-bots).

You can also create more complex rules when taking action on AI crawlers. For more information on creating more specific rules, refer to [Create a custom rule in the dashboard
](/waf/custom-rules/create-dashboard/).

<Details header="Which bots will Cloudflare block?">
<Render file="list-ai-bots" product="bots" />
</Details>
Expand Down
6 changes: 5 additions & 1 deletion src/content/partials/bots/list-ai-bots.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
{}
---

When you enable this feature, Cloudflare will block the following bots:
When you enable this feature, Cloudflare will block bots that have been identified as AI Crawlers. These bots include, but are not limited to, the following bots:

- `Amazonbot` (Amazon)
- `Applebot` (Apple)
Expand All @@ -24,8 +24,12 @@ When you enable this feature, Cloudflare will block the following bots:

In addition to this list, all verified bots that are classified as `AI Search`, `AI Assistant`, `AI Crawler`, or an `Archiver`, as well as a number of unverified bots that fall into the [verified bot categories](/bots/concepts/bot/verified-bots/categories/) are blocked. It does not block verified bots that fall into the `Search Engine` categories.

The bots classified in these categories may change in real time as new AI bots are detected.

:::note
Some AI bots overlap with definitely automated bots and verified bots, the latter becoming verified AI bots.

For a partial list of verified AI Bots, refer to the [Cloudflare Radar](https://radar.cloudflare.com/bots#verified-bots) categories of AI Search, AI Assistant, or AI Crawler, as well as some other bots that harvest data for AI training.
:::

If you are a bot operator and feel your bot may have been incorrectly categorized, [add your bot to the list of verified bots](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots).
Loading