Skip to content
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
type: overview
pcx_content_type: reference
title: AI Labyrinth
---

import { Render } from "~/components"

The AI Labyrinth adds invisible links on your webpage with specific `Nofollow` tags to block AI crawlers that do not adhere to the recommended guidelines and crawl without permission. AI crawlers that scrape your website content without permission will be stuck in a maze of never-ending links, and their details are recorded and used by all Cloudflare customers who choose to block [AI bots](/bots/concepts/bot/#ai-bots).

These links do not impact your search engine optimization (SEO) or your website's appearance, and are only seen by bots. AI bots that respect no-crawl instructions will safely ignore this honeypot.

<Render file="ai-labyrinth-enable" params={{ one: " " }} />
37 changes: 31 additions & 6 deletions src/content/docs/bots/get-started/bot-management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,22 +28,47 @@ To enable a [Bot Management](https://dash.cloudflare.com/?to=/:account/:zone/sec

:::note

If you are not seeing Bot Management enabled on your zone or if you still see **Add Bot Management** on the Cloudflare dashboard, contact your account team for the proper entitlements.
If you are not seeing Bot Management enabled on your zone or if you still see **Add Bot Management** on the Cloudflare dashboard, contact your account team for the proper entitlements.
:::

## Block AI bots
## Setup

Cloudflare recommends that you deploy the following basic settings and customize them according to the traffic in your zone.

### Enable the latest Machine Learning version

<Render file="latest-ml-model-enable" />

### Block AI Bots

<Render file="block-ai-bots-enable" params={{ one: "Bot Management" }} />

:::note

You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/).
:::

### Enable AI Labyrinth

<Render file="ai-labyrinth-enable" params={{ one: "Bot Management" }} />

## Setup
### Enable Javascript Detections

Enabling [JavaScript Detections](/bots/additional-configurations/javascript-detections/) validates that the browser can run JavaScript, and is stored in the `cf.bot_management.js_detection.passed` variable.

<Render file="javascript-detections-enable" params={{ one: "Bot Management" }} />

### Deploy default templates

Cloudflare has [default templates](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules?template=bot_traffic) for definite bots, which we are very confident are automated (bot score 1) and likely bots that have many bot tells (bot score 2-29). In our templates, we recommend to allow verified bots like Google SEO crawler and access to static resources, which should be cached anyway.

- [Definite Bots template](https://dash.cloudflare.com/?to=/:account/:zone:/security/security-rules/custom-rules/create?template=Definitely%20Bots): Targets malicious bot traffic while ignoring verified bots and routes delivering static content.

`(cf.bot_management.score eq 1 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)`

- [Likely Bots template](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules/custom-rules/create?template=Likely%20Bots): Targets traffic likely to be malicious bots while ignoring verified bots and routes with static content. It may contain a small amount of non-bot traffic.

`(cf.bot_management.score ge 2 and cf.bot_management.score le 29 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)`

* Optionally a [JSD Template](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules/custom-rules/create?template=JavaScript%20Verified%20URLs). If you toggled JSD above, then managed challenge. Make sure to add a method and URI path. JSD improves security for URLs that should only expect JavaScript enabled clients.

For more guidance on setup, refer to your Customer Success Manager.
`(not cf.bot_management.js_detection.passed and http.request.method eq "" and http.request.uri.path in {""})`
13 changes: 4 additions & 9 deletions src/content/docs/bots/reference/machine-learning-models.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,11 @@ sidebar:

---

## Enable Auto-updates to the Machine Learning Models
import { Render } from "~/components"

Cloudflare allows Enterprise customers to enable Auto-updates to its Machine Learning models for the newest bot detection models as they are released.
## Enable auto-updates to the Machine Learning models

To enable Auto-updates:

1. Log in to the Cloudflare dashboard and select your account and domain.
2. Go to **Security** > **Bots**.
3. Select **Configure Bot Management**.
4. Enable **Auto-updates to the Machine Learning Model**.
<Render file="latest-ml-model-enable" />

### What will change

Expand All @@ -38,4 +33,4 @@ By not updating to the latest version, you will be using a Machine Learning mode
| v5 | Recalibrated model for the [removal of `_cfduid` cookie](https://blog.cloudflare.com/deprecating-cfduid-cookie/). <br/><br/> Introduced new signals to reduce false negatives. | Q2 2021 |
| v6 | Significantly improved scoring for native Android application traffic. <br/><br/>Improved scoring on the newest versions of Chromium browsers. | Q1 2022 |
| v7 | Increased recognition of distributed botnets. <br/><br/>Improved HTTP/3 scoring. | Q1 2024 |
| v8 | Improved detection of residential proxies. <br/><br/>Increased weight on network level traffic characteristics. | Q2 2024 |
| v8 | Improved detection of residential proxies. <br/><br/>Increased weight on network level traffic characteristics. | Q2 2024 |
6 changes: 2 additions & 4 deletions src/content/partials/bots/ai-labyrinth-enable.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,9 @@ inputParameters: param1

import { Markdown } from "~/components"

The AI Labyrinth adds invisible links on your webpage with specific `Nofollow` tags to block AI crawlers that do not adhere to the recommended guidelines and crawl without permission. AI crawlers that scrape your website content without permission will be stuck in a maze of never-ending links, and their details are recorded and used by all Cloudflare customers who choose to block [AI bots](/bots/concepts/bot/#ai-bots).
AI Labyrinth decreases the accuracy and wastes the resources of AI Bots by confabulating articles on your website that are not visible to users.

These links do not impact your search engine optimization (SEO) or your website's appearance, and are only seen by bots. AI bots that respect no-crawl instructions will safely ignore this honeypot.

To enable AI Labyrinth:
To enable [AI Labyrinth](/bots/additional-configurations/ai-labyrinth):

1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
2. Go to **Security** > **Bots**.
Expand Down
2 changes: 2 additions & 0 deletions src/content/partials/bots/block-ai-bots-enable.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ inputParameters: param1

import { Markdown } from "~/components"

AI Bots will block Definitely Automated bots and Verified AI Bots, such as AI Search, AI Assistant, AI Crawler, or an AI Archiver.

To block [AI bots](/bots/concepts/bot/#ai-bots):

1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
Expand Down
15 changes: 15 additions & 0 deletions src/content/partials/bots/latest-ml-model-enable.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
{}

---

import { Markdown } from "~/components"

Cloudflare encourages Enterprise customers to enable Auto-updates to its Machine Learning models to get the newest bot detection models as they are released.

To enable Auto-updates:

1. Log in to the Cloudflare dashboard and select your account and domain.
2. Go to **Security** > **Bots**.
3. Select **Configure Bot Management**.
4. Enable **Auto-updates to the Machine Learning Model**.
Loading