Skip to content

Commit 3adbd4f

Browse files
committed
Adding info on configuring the response
1 parent 559fd82 commit 3adbd4f

File tree

1 file changed

+44
-0
lines changed

1 file changed

+44
-0
lines changed

src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,8 @@ For more details on how this rule interacts with other Cloudflare settings, refe
8888
- **Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
8989
- **When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
9090
- **Implementation:** From the **Actions** column, select **Block**.
91+
92+
Note that you can configure the response that gets returned when blocking an AI crawler. Refer to [Configure block response](/ai-crawl-control/features/manage-ai-crawlers/#configure-block-response).
9193
</Example>
9294

9395
<Example title="Charge for crawl (private beta)">
@@ -117,11 +119,53 @@ Note that you can still choose to [Enforce `robots.txt`](/ai-crawl-control/featu
117119
- **Summary:** You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
118120
- **When to use:** Block AI crawlers when their behavior do not align with your content strategy, or violate your policies.
119121
- **Implementation:** From the **Actions** column, select **Block**.
122+
123+
Note that you can configure the response that gets returned when blocking an AI crawler. Refer to [Configure block response](/ai-crawl-control/features/manage-ai-crawlers/#configure-block-response).
120124
</Example>
121125

122126
</TabItem>
123127
</Tabs>
124128

129+
## Configure block response
130+
131+
When blocking an AI crawler, you can configure the details of the response that gets returned to the AI crawler. Specifically, you can configure:
132+
133+
- The response code
134+
- The response body
135+
136+
To edit these values:
137+
138+
<Steps>
139+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
140+
2. Go to **AI Crawl Control**.
141+
3. Go to the **Settings** tab.
142+
4. Under **Block response**, select **Edit**.
143+
5. Once you have edited the values, select **Save**.
144+
</Steps>
145+
146+
### Edit the response code
147+
148+
You can choose which HTTP response code to return when blocking an AI crawler.
149+
150+
Use the dropdown menu to select the desired response code. You can choose from:
151+
152+
- **403 Forbidden**: Use this option if you wish to indicate that you do not want the AI crawler to access your content.
153+
- **402 Payment Required:** Use this option if you wish to indicate that the AI crawler must pay to access your content.
154+
155+
### Edit the response body
156+
157+
You can write a custom message (HTTP response body) to return when blocking an AI crawler.
158+
159+
In the **Response body** text field, enter the response you wish to display for the AI crawler in plain text.
160+
161+
{/* Need guidance on what sort of information they should include */}
162+
163+
:::note
164+
Behind the scenes, AI Crawl Control uses WAF to return custom block responses.
165+
166+
Refer to [Configure a custom response for blocked requests](/waf/custom-rules/create-dashboard/#configure-a-custom-response-for-blocked-requests).
167+
:::
168+
125169
## Related resources
126170

127171
- Use [pay per crawl](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) to charge AI crawlers every time they access your content.

0 commit comments

Comments
 (0)