You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx
+44Lines changed: 44 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -88,6 +88,8 @@ For more details on how this rule interacts with other Cloudflare settings, refe
88
88
-**Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
89
89
-**When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
90
90
-**Implementation:** From the **Actions** column, select **Block**.
91
+
92
+
Note that you can configure the response that gets returned when blocking an AI crawler. Refer to [Configure block response](/ai-crawl-control/features/manage-ai-crawlers/#configure-block-response).
91
93
</Example>
92
94
93
95
<Exampletitle="Charge for crawl (private beta)">
@@ -117,11 +119,53 @@ Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/featu
117
119
-**Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
118
120
-**When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
119
121
-**Implementation:** From the **Actions** column, select **Block**.
122
+
123
+
Note that you can configure the response that gets returned when blocking an AI crawler. Refer to [Configure block response](/ai-crawl-control/features/manage-ai-crawlers/#configure-block-response).
120
124
</Example>
121
125
122
126
</TabItem>
123
127
</Tabs>
124
128
129
+
## Configure block response
130
+
131
+
When blocking an AI crawler, you can configure the details of the response that gets returned to the AI crawler. Specifically, you can configure:
132
+
133
+
- The response code
134
+
- The response body
135
+
136
+
To edit these values:
137
+
138
+
<Steps>
139
+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
140
+
2. Go to **AI Crawl Control**.
141
+
3. Go to the **Settings** tab.
142
+
4. Under **Block response**, select **Edit**.
143
+
5. Once you have edited the values, select **Save**.
144
+
</Steps>
145
+
146
+
### Edit the response code
147
+
148
+
You can choose which HTTP response code to return when blocking an AI crawler.
149
+
150
+
Use the dropdown menu to select the desired response code. You can choose from:
151
+
152
+
-**403 Forbidden**: Use this option if you wish to indicate that you do not want the AI crawler to access your content.
153
+
-**402 Payment Required:** Use this option if you wish to indicate that the AI crawler must pay to access your content.
154
+
155
+
### Edit the response body
156
+
157
+
You can write a custom message (HTTP response body) to return when blocking an AI crawler.
158
+
159
+
In the **Response body** text field, enter the response you wish to display for the AI crawler in plain text.
160
+
161
+
{/* Need guidance on what sort of information they should include */}
162
+
163
+
:::note
164
+
Behind the scenes, AI Crawl Control uses WAF to return custom block responses.
165
+
166
+
Refer to [Configure a custom response for blocked requests](/waf/custom-rules/create-dashboard/#configure-a-custom-response-for-blocked-requests).
167
+
:::
168
+
125
169
## Related resources
126
170
127
171
- Use [pay per crawl](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers every time they access your content.
0 commit comments