Skip to content

Commit 3b29f5b

Browse files
patriciasantaanapatriciasantaana
committed
bots
1 parent f026e4e commit 3b29f5b

File tree

10 files changed

+151
-126
lines changed

10 files changed

+151
-126
lines changed

‎src/content/docs/bots/additional-configurations/ai-labyrinth.mdx‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar:
77
order: 7
88
---
99

10-
import { Render, Tabs, TabItem, Steps } from "~/components"
10+
import { Render, Tabs, TabItem, Steps, DashButton } from "~/components"
1111

1212
The AI Labyrinth adds invisible links on your webpage with specific `Nofollow` tags to block AI crawlers that do not adhere to the recommended guidelines and crawl without permission. AI crawlers that scrape your website content without permission will be stuck in a maze of never-ending links, and their details are recorded and used by all Cloudflare customers who choose to block [AI bots](/bots/concepts/bot/#ai-bots).
1313

@@ -26,10 +26,11 @@ To enable [AI Labyrinth](/bots/additional-configurations/ai-labyrinth):
2626
</TabItem>
2727
<TabItem label="New dashboard" icon="rocket">
2828
<Steps>
29-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login), and select your account and domain.
30-
2. Go to **Security** > **Settings**.
31-
3. Filter by **Bot traffic**.
32-
4. Go to **AI Labyrinth**.
29+
1. In the Cloudflare dashboard, go to the **Security Settings** page.
30+
31+
<DashButton url="/?to=/:account/:zone/security/settings" />
32+
2. Filter by **Bot traffic**.
33+
3. Go to **AI Labyrinth**.
3334
4. Turn **AI Labyrinth** on.
3435
</Steps>
3536
</TabItem>

‎src/content/docs/bots/additional-configurations/block-ai-bots.mdx‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ sidebar:
88
label: Block AI Bots
99
---
1010

11-
import { Render, Steps } from "~/components"
11+
import { Render, Steps, DashButton } from "~/components"
1212

1313
:::note[Block AI bots availability]
1414
The **Block AI bots** feature is only available in the new [application security dashboard](/security/).
@@ -19,15 +19,16 @@ You can choose to block AI bots by activating **Block AI bots**. Activating this
1919
To block [AI bots](/bots/concepts/bot/#ai-bots):
2020

2121
<Steps>
22-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
23-
2. Go to **Security** > **Settings**.
24-
3. Filter by **Bot traffic**.
25-
4. Go to **Block AI bots**.
26-
5. Under **Configurations**, select the edit icon. Choose from:
27-
- **Only block on hostnames with ads**: Use this option if you wish to block AI bots only on portions of your site that show ads. Cloudflare automatically detects whether ads are present on a subdomain, and only block on hostnames that contain those ad units.
28-
- **Block on all pages**: Use this option if you wish to block AI bots on all your pages.
29-
- **Do not block (off)**: Use this option if you wish to allow AI bots on all your pages.
30-
6. Select **Save** to save your configuration.
22+
1. In the Cloudflare dashboard, go to the **Security Settings** page.
23+
24+
<DashButton url="/?to=/:account/:zone/security/settings" />
25+
2. Filter by **Bot traffic**.
26+
3. Go to **Block AI bots**.
27+
4. Under **Configurations**, select the edit icon. Choose from:
28+
- **Only block on hostnames with ads**: Use this option if you wish to block AI bots only on portions of your site that show ads. Cloudflare automatically detects whether ads are present on a subdomain, and only block on hostnames that contain those ad units.
29+
- **Block on all pages**: Use this option if you wish to block AI bots on all your pages.
30+
- **Do not block (off)**: Use this option if you wish to allow AI bots on all your pages.
31+
5. Select **Save** to save your configuration.
3132
</Steps>
3233

3334
To block individual AI crawlers (rather than blocking all crawlers), use [AI Crawl Control](/ai-crawl-control/).

‎src/content/docs/bots/additional-configurations/detection-ids.mdx‎

Lines changed: 47 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ sidebar:
55
order: 2
66
---
77

8-
import { Render, Tabs, TabItem, Steps } from "~/components"
8+
import { Render, Tabs, TabItem, Steps, DashButton } from "~/components"
99

1010
Detection IDs are static rules used to detect predictable bot behavior with no overlap with human traffic. Detection IDs refer to the precise [detection](/bots/concepts/bot-detection-engines/) used to identify a bot, which could be from heuristics, verified bot detections, or anomaly detections. For example, a detection ID can identify if you sent your headers in a different order than what was expected of your browser.
1111

@@ -38,27 +38,6 @@ Detection tags are available in Security Analytics, but not in the Security Even
3838

3939
---
4040

41-
## Bot Detection IDs via Logpush
42-
43-
You can create or edit existing Logpush jobs to include the new Bot Detection IDs field which will provide an array of IDs for each request that has heuristics match on it. The `BotDetectionIDs` field is available as part of the HTTP Requests dataset and you can add it to new or existing jobs via the Logpush API or on the Cloudflare dashboard. This is the primary method to discover Detection IDs.
44-
45-
### Via the Cloudflare dashboard
46-
47-
<Steps>
48-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
49-
2. Go to **Analytics & Logs** > **Logs**.
50-
3. Select **Add Logpush Job**.
51-
4. Select **HTTP Requests** as the dataset.
52-
5. Select **BotDetectionIDs** under the General data field category.
53-
6. Select and enter the destination information.
54-
7. Prove the ownership.
55-
8. Select **Save**.
56-
</Steps>
57-
58-
### Via the API
59-
60-
[Update your logpush job](/logs/logpush/logpush-job/log-output-options/) by adding `BotDetectionIDs` to the `output_options:` parameters.
61-
6241
## Create or edit an expression
6342

6443
<Tabs syncKey="dashNewNav">
@@ -72,11 +51,16 @@ You can create or edit existing Logpush jobs to include the new Bot Detection ID
7251
</TabItem>
7352
<TabItem label="New dashboard" icon="rocket">
7453
<Steps>
75-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login), and select your account and domain.
76-
2. Go to **Security** > **Analytics**.
77-
3. Apply filters and select **Create custom security rule** to create a custom rule based on your filters. Alternatively, if you have already created a custom rule, you can go to the existing rule in **Security** > **Security rules** and edit the expression based on your filters.
78-
4. Use the `cf.bot_management.detection_ids` field in the rule expression.
79-
5. Select **Deploy**.
54+
1. In the Cloudflare dashboard, go to the **Security Analytics** page.
55+
56+
<DashButton url="/?to=/:account/:zone/security/analytics" />
57+
2. Apply filters and select **Create custom security rule** to create a custom rule based on your filters.
58+
59+
Alternatively, if you have already created a custom rule, you can go to the existing rule in **Security rules** and edit the expression based on your filters.
60+
61+
<DashButton url="/?to=/:account/:zone/security/security-rules" />
62+
3. Use the `cf.bot_management.detection_ids` field in the rule expression.
63+
4. Select **Deploy**.
8064
</Steps>
8165
</TabItem>
8266
</Tabs>
@@ -134,8 +118,9 @@ To access account takeover detections:
134118
</TabItem>
135119
<TabItem label="New dashboard" icon="rocket">
136120
<Steps>
137-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login), and select your account and domain.
138-
2. Go to **Security** > **Security rules**.
121+
1. In the Cloudflare dashboard, go to the **Security rules** page.
122+
123+
<DashButton url="/?to=/:account/:zone/security/security-rules" />
139124
3. Select **Create rule** and choose **Custom rule**.
140125
4. Fill out the form using **Bot Detection IDs** along with other necessary information.
141126
5. Select **Save as draft** to return to it later, or **Deploy** to deploy the rule.
@@ -166,11 +151,12 @@ To use rate limiting rules with account takeover detections:
166151
</TabItem>
167152
<TabItem label="New dashboard" icon="rocket">
168153
<Steps>
169-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login), and select your account and domain.
170-
2. Go to **Security** > **Security rules**.
171-
3. Select **Create rule** and choose **Rate limiting rule**.
172-
4. Fill out the form using the **Custom expression builder** and `cf.bot_management_detection_ids` along with other necessary information.
173-
5. Select **Save as draft** to return to it later, or **Deploy** to deploy the rule.
154+
1. In the Cloudflare dashboard, go to the **Security rules** page.
155+
156+
<DashButton url="/?to=/:account/:zone/security/security-rules" />
157+
2. Select **Create rule** and choose **Rate limiting rule**.
158+
3. Fill out the form using the **Custom expression builder** and `cf.bot_management_detection_ids` along with other necessary information.
159+
4. Select **Save as draft** to return to it later, or **Deploy** to deploy the rule.
174160
</Steps>
175161
</TabItem>
176162
</Tabs>
@@ -186,6 +172,33 @@ The rule can be enhanced with Leaked Credential Checks. Refer to the [WAF docume
186172
| ------------ | ---------------------------------------- |
187173
| `50331651` | Observes traffic from residential proxy networks and similar commercial proxies. <br /><br />When the ID matches a request, Bot Management sets the bot score to 29 and uses [anomaly detection](/bots/concepts/bot-detection-engines/#anomaly-detection-enterprise) as its score source. |
188174

175+
---
176+
177+
## Bot Detection IDs via Logpush
178+
179+
You can create or edit existing Logpush jobs to include the new Bot Detection IDs field which will provide an array of IDs for each request that has heuristics match on it. The `BotDetectionIDs` field is available as part of the HTTP Requests dataset and you can add it to new or existing jobs via the Logpush API or on the Cloudflare dashboard. This is the primary method to discover Detection IDs.
180+
181+
<Tabs>
182+
<TabItem label="Dashboard">
183+
<Steps>
184+
1. In the Cloudflare dashboard, go to the **Logpush** page.
185+
186+
<DashButton url="/?to=/:account/logs" />
187+
2. Select **Create a Logpush Job**.
188+
3. Select and enter the destination information.
189+
4. Select **HTTP Requests** as the dataset.
190+
5. Select **BotDetectionIDs** under the General data field category.
191+
6. Prove the ownership.
192+
7. Select **Save**.
193+
</Steps>
194+
</TabItem>
195+
<TabItem label="API">
196+
[Update your logpush job](/logs/logpush/logpush-job/log-output-options/) by adding `BotDetectionIDs` to the `output_options:` parameters.
197+
</TabItem>
198+
</Tabs>
199+
200+
---
201+
189202
## Availability
190203

191204
Detection IDs are available for Enterprise Bot Management customers.

‎src/content/docs/bots/additional-configurations/managed-robots-txt.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ To implement a `robots.txt` file on your domain:
123123
</TabItem>
124124
<TabItem label="New dashboard" icon="rocket">
125125
<Steps>
126-
1. In the Cloudflare dashboard, go to the Security Settings page.
126+
1. In the Cloudflare dashboard, go to the **Security Settings** page.
127127

128128
<DashButton url="/?to=/:account/:zone/security/settings" />
129129
2. Filter by **Bot traffic**.

‎src/content/docs/bots/additional-configurations/static-resources.mdx‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ sidebar:
66

77
---
88

9-
import { Render, Tabs, TabItem, Steps, GlossaryTooltip } from "~/components";
9+
import { Render, Tabs, TabItem, Steps, GlossaryTooltip, DashButton } from "~/components";
1010

1111
Pro, Business, and Enterprise customers can use Cloudflare's bot solutions to protect their <GlossaryTooltip term="static content">static resources</GlossaryTooltip> from bots.
1212

@@ -30,11 +30,12 @@ To enable this feature as a Pro or Business customer or an Enterprise customer w
3030
</TabItem>
3131
<TabItem label="New dashboard" icon="rocket">
3232
<Steps>
33-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login), and select your account and domain.
34-
2. Go to **Security** > **Settings**.
35-
3. Filter by **Bot traffic**.
36-
4. Go to **Super Bot Fight Mode**.
37-
5. Under **Configurations**, select the edit icon for **Static resource protection** and turn it on.
33+
1. In the Cloudflare dashboard, go to the **Security Settings** page.
34+
35+
<DashButton url="/?to=/:account/:zone/security/settings" />
36+
2. Filter by **Bot traffic**.
37+
3. Go to **Super Bot Fight Mode**.
38+
4. Under **Configurations**, select the edit icon for **Static resource protection** and turn it on.
3839
</Steps>
3940
</TabItem>
4041
</Tabs>
@@ -56,4 +57,4 @@ Static resources are files with the following extensions:
5657

5758
`ico|jpg|png|jpeg|gif|css|js|tif|tiff|bmp|pict|webp|svg|svgz|class|jar|txt|csv|doc|docx|xls|xlsx|pdf|ps|pls|ppt|pptx|ttf|otf|woff|woff2|eot|eps|ejs|swf|torrent|midi|mid|m3u8|m4a|mp3|ogg|ts`
5859

59-
Additionally, the `/.well-known/` URL path and all elements in it are considered a static resource, regardless of file extension.
60+
Additionally, the `/.well-known/` URL path and all elements in it are considered a static resource, regardless of the file extension.

‎src/content/docs/bots/concepts/feedback-loop.mdx‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ sidebar:
55
order: 4
66
---
77

8-
import { GlossaryTooltip, Steps, Tabs, TabItem } from "~/components"
8+
import { GlossaryTooltip, Steps, Tabs, TabItem, DashButton } from "~/components"
99

1010
The Bot Feedback Loop is a way for customers to send Cloudflare direct feedback in the case of Bot Management potentially <GlossaryTooltip term="bot score" link="/bots/concepts/bot-score/">scoring</GlossaryTooltip> a request incorrectly. When a customer submits a False Negative or a False Positive report, Cloudflare manually analyzes this data and uses it as a training dataset for our next Machine Learning model.
1111

@@ -53,8 +53,9 @@ If Cloudflare is unable to detect a portion of automated traffic on your site, s
5353
</TabItem>
5454
<TabItem label="New dashboard" icon="rocket">
5555
<Steps>
56-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
57-
2. Go to **Security** > **Analytics**.
56+
1. In the Cloudflare dashboard, go to the **Security Analytics** page.
57+
58+
<DashButton url="/?to=/:account/:zone/security/analytics" />
5859
3. Apply one or more filters.
5960
4. Under **Request activity**, filter by **Bot analysis**.
6061
5. Select **Report incorrect data** and fill out the form.

‎src/content/docs/bots/get-started/bot-fight-mode.mdx‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ head:
99

1010
---
1111

12-
import { Tabs, TabItem, Steps, Render } from '~/components';
12+
import { Tabs, TabItem, Steps, Render, DashButton } from '~/components';
1313

1414
Bot Fight Mode is a simple, free product that helps detect and mitigate bot traffic on your domain. When enabled, the product:
1515

@@ -37,11 +37,12 @@ To start using Bot Fight Mode:
3737
</TabItem>
3838
<TabItem label="New dashboard" icon="rocket">
3939
<Steps>
40-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login), and select your account and domain.
41-
2. Go to **Security** > **Settings**.
42-
3. Filter by **Bot traffic**.
43-
4. Go to **Bot Fight Mode**.
44-
5. Turn **Bot Fight Mode** on.
40+
1. In the Cloudflare dashboard, go to the **Security Settings** page.
41+
42+
<DashButton url="/?to=/:account/:zone/security/settings" />
43+
2. Filter by **Bot traffic**.
44+
3. Go to **Bot Fight Mode**.
45+
4. Turn **Bot Fight Mode** on.
4546
</Steps>
4647
</TabItem>
4748
</Tabs>
@@ -66,11 +67,12 @@ To disable Bot Fight Mode:
6667
</TabItem>
6768
<TabItem label="New dashboard" icon="rocket">
6869
<Steps>
69-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login), and select your account and domain.
70-
2. Go to **Security** > **Settings**.
71-
3. Filter by **Bot traffic**.
72-
4. Go to **Bot Fight Mode**.
73-
5. Turn **Bot Fight Mode** on.
70+
1. In the Cloudflare dashboard, go to the **Security Settings** page.
71+
72+
<DashButton url="/?to=/:account/:zone/security/settings" />
73+
2. Filter by **Bot traffic**.
74+
3. Go to **Bot Fight Mode**.
75+
4. Turn **Bot Fight Mode** off.
7476
</Steps>
7577
</TabItem>
7678
</Tabs>

‎src/content/docs/bots/get-started/bot-management.mdx‎

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ learning_center:
99

1010
---
1111

12-
import { Tabs, TabItem, Steps, Render } from '~/components';
12+
import { Tabs, TabItem, Steps, Render, DashButton } from '~/components';
1313

1414
Bot Management for Enterprise is a paid add-on that provides sophisticated bot protection for your domain. Customers can identify automated traffic, take appropriate action, and view detailed analytics within the dashboard.
1515

@@ -34,20 +34,21 @@ Bot Management is automatically enabled for Enterprise zones entitled with the a
3434
3. Select **Add Bot Management**.
3535
</Steps>
3636
</TabItem>
37-
<TabItem label="New dashboard" icon="rocket">
38-
<Steps>
39-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login), and select your account and domain.
40-
2. Go to **Security** > **Settings**.
41-
3. Filter by **Bot traffic**.
42-
4. Go to **Bot management**.
43-
5. Turn **Bot management** on.
44-
6. Choose how your domain should respond to various types of traffic by selecting the associated edit icon.
45-
- For more details on verified bots, refer to [Verified Bots](/bots/concepts/bot/#verified-bots).
46-
- For more details on supported file types, refer to [Static resource protection](/bots/additional-configurations/static-resources/).
47-
- For more details on invisible code injection, refer to [JavaScript detections](/bots/additional-configurations/javascript-detections/).
48-
- For more details on WordPress optimization, refer to [Super Bot Fight Mode for WordPress](/bots/troubleshooting/wordpress-loopback-issue/).
49-
</Steps>
50-
</TabItem>
37+
<TabItem label="New dashboard" icon="rocket">
38+
<Steps>
39+
1. In the Cloudflare dashboard, go to the **Security Settings** page.
40+
41+
<DashButton url="/?to=/:account/:zone/security/settings" />
42+
2. Filter by **Bot traffic**.
43+
3. Go to **Bot management**.
44+
4. Turn **Bot management** on.
45+
5. Choose how your domain should respond to various types of traffic by selecting the associated edit icon.
46+
- For more details on verified bots, refer to [Verified Bots](/bots/concepts/bot/#verified-bots).
47+
- For more details on supported file types, refer to [Static resource protection](/bots/additional-configurations/static-resources/).
48+
- For more details on invisible code injection, refer to [JavaScript detections](/bots/additional-configurations/javascript-detections/).
49+
- For more details on WordPress optimization, refer to [Super Bot Fight Mode for WordPress](/bots/troubleshooting/wordpress-loopback-issue/).
50+
</Steps>
51+
</TabItem>
5152
</Tabs>
5253

5354
:::note

0 commit comments

Comments
 (0)