Skip to content

Commit 32f2c62

Browse files
Added setup documentation for enterprise bot management (#21486)
* Added setup documentation for enterprise bot management * Update src/content/partials/bots/latest-ml-model-enable.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/get-started/bot-management.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/get-started/bot-management.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/get-started/bot-management.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/get-started/bot-management.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/get-started/bot-management.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/get-started/bot-management.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/reference/machine-learning-models.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/reference/machine-learning-models.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/get-started/bot-management.mdx Co-authored-by: Patricia Santa Ana <[email protected]> * Update src/content/docs/bots/get-started/bot-management.mdx * Update src/content/partials/bots/latest-ml-model-enable.mdx * Update src/content/docs/bots/additional-configurations/ai-labyrinth.mdx * Update src/content/docs/bots/additional-configurations/ai-labyrinth.mdx --------- Co-authored-by: Ross Jacobs <[email protected]> Co-authored-by: Patricia Santa Ana <[email protected]>
1 parent 1f6d518 commit 32f2c62

File tree

6 files changed

+64
-19
lines changed

6 files changed

+64
-19
lines changed
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
pcx_content_type: overview
3+
title: AI Labyrinth
4+
---
5+
6+
import { Render } from "~/components"
7+
8+
The AI Labyrinth adds invisible links on your webpage with specific `Nofollow` tags to block AI crawlers that do not adhere to the recommended guidelines and crawl without permission. AI crawlers that scrape your website content without permission will be stuck in a maze of never-ending links, and their details are recorded and used by all Cloudflare customers who choose to block [AI bots](/bots/concepts/bot/#ai-bots).
9+
10+
These links do not impact your search engine optimization (SEO) or your website's appearance, and are only seen by bots. AI bots that respect no-crawl instructions will safely ignore this honeypot.

src/content/docs/bots/get-started/bot-management.mdx

Lines changed: 31 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -28,22 +28,47 @@ To enable a [Bot Management](https://dash.cloudflare.com/?to=/:account/:zone/sec
2828

2929
:::note
3030

31-
If you are not seeing Bot Management enabled on your zone or if you still see **Add Bot Management** on the Cloudflare dashboard, contact your account team for the proper entitlements.
31+
If you are not seeing Bot Management enabled on your zone or if you still see **Add Bot Management** on the Cloudflare dashboard, contact your account team for the proper entitlements.
3232
:::
3333

34-
## Block AI bots
34+
## Setup
35+
36+
Cloudflare recommends that you deploy the following basic settings and customize them according to the traffic in your zone.
37+
38+
### Enable the latest Machine Learning version
39+
40+
<Render file="latest-ml-model-enable" />
41+
42+
### Block AI Bots
3543

3644
<Render file="block-ai-bots-enable" params={{ one: "Bot Management" }} />
3745

3846
:::note
39-
40-
You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
47+
You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
4148
:::
4249

4350
### Enable AI Labyrinth
4451

4552
<Render file="ai-labyrinth-enable" params={{ one: "Bot Management" }} />
4653

47-
## Setup
54+
### Enable Javascript Detections
55+
56+
Enabling [JavaScript Detections](/bots/additional-configurations/javascript-detections/) validates that the browser can run JavaScript, and is stored in the `cf.bot_management.js_detection.passed` variable.
57+
58+
<Render file="javascript-detections-enable" params={{ one: "Bot Management" }} />
59+
60+
### Deploy default templates
61+
62+
Cloudflare has [default templates](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules?template=bot_traffic) for definite bots, which we are very confident are automated (bot score 1) and likely bots that have many bot tells (bot score 2-29). In our templates, we recommend to allow verified bots like Google SEO crawler and access to static resources, which should be cached anyway.
63+
64+
- [Definite Bots template](https://dash.cloudflare.com/?to=/:account/:zone:/security/security-rules/custom-rules/create?template=Definitely%20Bots): Targets malicious bot traffic while ignoring verified bots and routes delivering static content.
65+
66+
`(cf.bot_management.score eq 1 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)`
67+
68+
- [Likely Bots template](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules/custom-rules/create?template=Likely%20Bots): Targets traffic likely to be malicious bots while ignoring verified bots and routes with static content. It may contain a small amount of non-bot traffic.
69+
70+
`(cf.bot_management.score ge 2 and cf.bot_management.score le 29 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)`
71+
72+
- (Optional) [JavaScript detections template](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules/custom-rules/create?template=JavaScript%20Verified%20URLs): If you enabled JavaScript detections, then set up a [managed challenge](/fundamentals/security/cloudflare-challenges/#managed-challenge-recommended), make sure to add a method and URI path. JavaScript detections improves security for URLs that should only expect JavaScript-enabled clients.
4873

49-
For more guidance on setup, refer to your Customer Success Manager.
74+
`(not cf.bot_management.js_detection.passed and http.request.method eq "" and http.request.uri.path in {""})`

src/content/docs/bots/reference/machine-learning-models.mdx

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,11 @@ sidebar:
66

77
---
88

9-
## Enable Auto-updates to the Machine Learning Models
9+
import { Render } from "~/components"
1010

11-
Cloudflare allows Enterprise customers to enable Auto-updates to its Machine Learning models for the newest bot detection models as they are released.
11+
## Enable auto-updates to the Machine Learning models
1212

13-
To enable Auto-updates:
14-
15-
1. Log in to the Cloudflare dashboard and select your account and domain.
16-
2. Go to **Security** > **Bots**.
17-
3. Select **Configure Bot Management**.
18-
4. Enable **Auto-updates to the Machine Learning Model**.
13+
<Render file="latest-ml-model-enable" />
1914

2015
### What will change
2116

@@ -38,4 +33,4 @@ By not updating to the latest version, you will be using a Machine Learning mode
3833
| v5 | Recalibrated model for the [removal of `_cfduid` cookie](https://blog.cloudflare.com/deprecating-cfduid-cookie/). <br/><br/> Introduced new signals to reduce false negatives. | Q2 2021 |
3934
| v6 | Significantly improved scoring for native Android application traffic. <br/><br/>Improved scoring on the newest versions of Chromium browsers. | Q1 2022 |
4035
| v7 | Increased recognition of distributed botnets. <br/><br/>Improved HTTP/3 scoring. | Q1 2024 |
41-
| v8 | Improved detection of residential proxies. <br/><br/>Increased weight on network level traffic characteristics. | Q2 2024 |
36+
| v8 | Improved detection of residential proxies. <br/><br/>Increased weight on network level traffic characteristics. | Q2 2024 |

src/content/partials/bots/ai-labyrinth-enable.mdx

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,9 @@ inputParameters: param1
55

66
import { Markdown } from "~/components"
77

8-
The AI Labyrinth adds invisible links on your webpage with specific `Nofollow` tags to block AI crawlers that do not adhere to the recommended guidelines and crawl without permission. AI crawlers that scrape your website content without permission will be stuck in a maze of never-ending links, and their details are recorded and used by all Cloudflare customers who choose to block [AI bots](/bots/concepts/bot/#ai-bots).
8+
AI Labyrinth decreases the accuracy and wastes the resources of AI Bots by confabulating articles on your website that are not visible to users.
99

10-
These links do not impact your search engine optimization (SEO) or your website's appearance, and are only seen by bots. AI bots that respect no-crawl instructions will safely ignore this honeypot.
11-
12-
To enable AI Labyrinth:
10+
To enable [AI Labyrinth](/bots/additional-configurations/ai-labyrinth):
1311

1412
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
1513
2. Go to **Security** > **Bots**.

src/content/partials/bots/block-ai-bots-enable.mdx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ inputParameters: param1
55

66
import { Markdown } from "~/components"
77

8+
AI Bots will block Definitely Automated bots and Verified AI Bots, such as AI Search, AI Assistant, AI Crawler, or an AI Archiver.
9+
810
To block [AI bots](/bots/concepts/bot/#ai-bots):
911

1012
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
{}
3+
4+
---
5+
6+
import { Markdown } from "~/components"
7+
8+
Cloudflare encourages Enterprise customers to enable auto-updates to its Machine Learning models to get the newest bot detection models as they are released.
9+
10+
To enable auto-updates:
11+
12+
1. Log in to the Cloudflare dashboard and select your account and domain.
13+
2. Go to **Security** > **Bots**.
14+
3. Select **Configure Bot Management**.
15+
4. Enable **Auto-updates to the Machine Learning Model**.

0 commit comments

Comments
 (0)