diff --git a/src/content/docs/bots/additional-configurations/ai-labyrinth.mdx b/src/content/docs/bots/additional-configurations/ai-labyrinth.mdx new file mode 100644 index 000000000000000..a87bdc7a9f9baf3 --- /dev/null +++ b/src/content/docs/bots/additional-configurations/ai-labyrinth.mdx @@ -0,0 +1,10 @@ +--- +pcx_content_type: overview +title: AI Labyrinth +--- + +import { Render } from "~/components" + +The AI Labyrinth adds invisible links on your webpage with specific `Nofollow` tags to block AI crawlers that do not adhere to the recommended guidelines and crawl without permission. AI crawlers that scrape your website content without permission will be stuck in a maze of never-ending links, and their details are recorded and used by all Cloudflare customers who choose to block [AI bots](/bots/concepts/bot/#ai-bots). + +These links do not impact your search engine optimization (SEO) or your website's appearance, and are only seen by bots. AI bots that respect no-crawl instructions will safely ignore this honeypot. diff --git a/src/content/docs/bots/get-started/bot-management.mdx b/src/content/docs/bots/get-started/bot-management.mdx index 2e32a0efa38bc99..5d821649cdc0e45 100644 --- a/src/content/docs/bots/get-started/bot-management.mdx +++ b/src/content/docs/bots/get-started/bot-management.mdx @@ -28,22 +28,47 @@ To enable a [Bot Management](https://dash.cloudflare.com/?to=/:account/:zone/sec :::note -If you are not seeing Bot Management enabled on your zone or if you still see **Add Bot Management** on the Cloudflare dashboard, contact your account team for the proper entitlements. +If you are not seeing Bot Management enabled on your zone or if you still see **Add Bot Management** on the Cloudflare dashboard, contact your account team for the proper entitlements. ::: -## Block AI bots +## Setup + +Cloudflare recommends that you deploy the following basic settings and customize them according to the traffic in your zone. + +### Enable the latest Machine Learning version + + + +### Block AI Bots :::note - -You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/). +You can view blocked AI bot traffic via [Security Analytics](/waf/analytics/security-analytics/). ::: ### Enable AI Labyrinth -## Setup +### Enable Javascript Detections + +Enabling [JavaScript Detections](/bots/additional-configurations/javascript-detections/) validates that the browser can run JavaScript, and is stored in the `cf.bot_management.js_detection.passed` variable. + + + +### Deploy default templates + +Cloudflare has [default templates](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules?template=bot_traffic) for definite bots, which we are very confident are automated (bot score 1) and likely bots that have many bot tells (bot score 2-29). In our templates, we recommend to allow verified bots like Google SEO crawler and access to static resources, which should be cached anyway. + +- [Definite Bots template](https://dash.cloudflare.com/?to=/:account/:zone:/security/security-rules/custom-rules/create?template=Definitely%20Bots): Targets malicious bot traffic while ignoring verified bots and routes delivering static content. + + `(cf.bot_management.score eq 1 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)` + +- [Likely Bots template](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules/custom-rules/create?template=Likely%20Bots): Targets traffic likely to be malicious bots while ignoring verified bots and routes with static content. It may contain a small amount of non-bot traffic. + + `(cf.bot_management.score ge 2 and cf.bot_management.score le 29 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)` + +- (Optional) [JavaScript detections template](https://dash.cloudflare.com/?to=/:account/:zone/security/security-rules/custom-rules/create?template=JavaScript%20Verified%20URLs): If you enabled JavaScript detections, then set up a [managed challenge](/fundamentals/security/cloudflare-challenges/#managed-challenge-recommended), make sure to add a method and URI path. JavaScript detections improves security for URLs that should only expect JavaScript-enabled clients. -For more guidance on setup, refer to your Customer Success Manager. + `(not cf.bot_management.js_detection.passed and http.request.method eq "" and http.request.uri.path in {""})` \ No newline at end of file diff --git a/src/content/docs/bots/reference/machine-learning-models.mdx b/src/content/docs/bots/reference/machine-learning-models.mdx index 4c582a6c8155544..fe7d95ff69319f5 100644 --- a/src/content/docs/bots/reference/machine-learning-models.mdx +++ b/src/content/docs/bots/reference/machine-learning-models.mdx @@ -6,16 +6,11 @@ sidebar: --- -## Enable Auto-updates to the Machine Learning Models +import { Render } from "~/components" -Cloudflare allows Enterprise customers to enable Auto-updates to its Machine Learning models for the newest bot detection models as they are released. +## Enable auto-updates to the Machine Learning models -To enable Auto-updates: - -1. Log in to the Cloudflare dashboard and select your account and domain. -2. Go to **Security** > **Bots**. -3. Select **Configure Bot Management**. -4. Enable **Auto-updates to the Machine Learning Model**. + ### What will change @@ -38,4 +33,4 @@ By not updating to the latest version, you will be using a Machine Learning mode | v5 | Recalibrated model for the [removal of `_cfduid` cookie](https://blog.cloudflare.com/deprecating-cfduid-cookie/).

Introduced new signals to reduce false negatives. | Q2 2021 | | v6 | Significantly improved scoring for native Android application traffic.

Improved scoring on the newest versions of Chromium browsers. | Q1 2022 | | v7 | Increased recognition of distributed botnets.

Improved HTTP/3 scoring. | Q1 2024 | -| v8 | Improved detection of residential proxies.

Increased weight on network level traffic characteristics. | Q2 2024 | +| v8 | Improved detection of residential proxies.

Increased weight on network level traffic characteristics. | Q2 2024 | \ No newline at end of file diff --git a/src/content/partials/bots/ai-labyrinth-enable.mdx b/src/content/partials/bots/ai-labyrinth-enable.mdx index 2659c734ed82aae..235c6ed47ebfe00 100644 --- a/src/content/partials/bots/ai-labyrinth-enable.mdx +++ b/src/content/partials/bots/ai-labyrinth-enable.mdx @@ -5,11 +5,9 @@ inputParameters: param1 import { Markdown } from "~/components" -The AI Labyrinth adds invisible links on your webpage with specific `Nofollow` tags to block AI crawlers that do not adhere to the recommended guidelines and crawl without permission. AI crawlers that scrape your website content without permission will be stuck in a maze of never-ending links, and their details are recorded and used by all Cloudflare customers who choose to block [AI bots](/bots/concepts/bot/#ai-bots). +AI Labyrinth decreases the accuracy and wastes the resources of AI Bots by confabulating articles on your website that are not visible to users. -These links do not impact your search engine optimization (SEO) or your website's appearance, and are only seen by bots. AI bots that respect no-crawl instructions will safely ignore this honeypot. - -To enable AI Labyrinth: +To enable [AI Labyrinth](/bots/additional-configurations/ai-labyrinth): 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain. 2. Go to **Security** > **Bots**. diff --git a/src/content/partials/bots/block-ai-bots-enable.mdx b/src/content/partials/bots/block-ai-bots-enable.mdx index 862c28d5eb7e940..ff18e47e5e38599 100644 --- a/src/content/partials/bots/block-ai-bots-enable.mdx +++ b/src/content/partials/bots/block-ai-bots-enable.mdx @@ -5,6 +5,8 @@ inputParameters: param1 import { Markdown } from "~/components" +AI Bots will block Definitely Automated bots and Verified AI Bots, such as AI Search, AI Assistant, AI Crawler, or an AI Archiver. + To block [AI bots](/bots/concepts/bot/#ai-bots): 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain. diff --git a/src/content/partials/bots/latest-ml-model-enable.mdx b/src/content/partials/bots/latest-ml-model-enable.mdx new file mode 100644 index 000000000000000..9b1c2776448f2d1 --- /dev/null +++ b/src/content/partials/bots/latest-ml-model-enable.mdx @@ -0,0 +1,15 @@ +--- +{} + +--- + +import { Markdown } from "~/components" + +Cloudflare encourages Enterprise customers to enable auto-updates to its Machine Learning models to get the newest bot detection models as they are released. + +To enable auto-updates: + +1. Log in to the Cloudflare dashboard and select your account and domain. +2. Go to **Security** > **Bots**. +3. Select **Configure Bot Management**. +4. Enable **Auto-updates to the Machine Learning Model**. \ No newline at end of file