Skip to content

Commit e86e041

Browse files
committed
Setting up redirects, PCX review.
1 parent 4c5ceb3 commit e86e041

File tree

9 files changed

+44
-49
lines changed

9 files changed

+44
-49
lines changed

public/__redirects

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,12 @@
153153
/ai-audit/features/enforce-robots-txt/ /ai-crawl-control/features/analyze-ai-crawlers/ 301
154154
/ai-audit/features/analyze-ai-crawlers/ /ai-crawl-control/features/analyze-ai-traffic/ 301
155155

156+
# ai crawl control
157+
/ai-crawl-control/reference/faq/ /ai-crawl-control/features/pay-per-crawl/faq/ 301
158+
/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account/ /ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings/ 301
159+
/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/connect-to-stripe/ /ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/ 301
160+
/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-metrics/ /ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity/ 301
161+
156162
# AI Gateway
157163
/ai-gateway/get-started/configuring-settings/ /ai-gateway/get-started/ 301
158164
/ai-gateway/get-started/connecting-applications/ /ai-gateway/get-started/ 301
@@ -209,7 +215,7 @@
209215
/analytics/analytics-integrations/google-cloud/ /analytics/analytics-integrations/ 301
210216
/analytics/dashboards/ /log-explorer/custom-dashboards/ 301
211217
/analytics/analytics-integrations/looker/ /analytics/analytics-integrations/ 301
212-
/analytics/network-analytics/reference/network-analytics-v1/ /analytics/graphql-api/migration-guides/network-analytics-v2/ 301
218+
/analytics/network-analytics/reference/network-analytics-v1/ /analytics/graphql-api/migration-guides/network-analytics-v2/ 301
213219

214220
# email-security
215221
/email-security/reporting/search/detection-search/ /email-security/reporting/search/ 301

src/content/docs/ai-crawl-control/features/analyze-ai-traffic.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ You can find meaningful information across both **Crawlers** and **Metrics** tab
2222

2323
The **Crawlers** tab provides you with the following information:
2424

25-
- Total requests to crawl your website from common AI crawlers
25+
- Total number of requests to crawl your website from common AI crawlers
2626
- Number of requests made by each AI crawler
2727
- Number of `robots.txt` violations for each crawler
2828

src/content/docs/ai-crawl-control/features/manage-ai-crawlers.mdx

Lines changed: 14 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,7 @@ sidebar:
55
order: 5
66
---
77

8-
import {
9-
Steps,
10-
GlossaryTooltip,
11-
Tabs,
12-
TabItem,
13-
Example,
14-
Plan,
15-
} from "~/components";
8+
import { Steps, GlossaryTooltip, Tabs, TabItem, Example, Plan } from "~/components";
169

1710
AI Crawl Control enables you to take specific action for each AI crawler.
1811

@@ -26,13 +19,13 @@ To manage AI crawlers:
2619

2720
The **Crawlers** tab displays a table of AI crawlers that are requesting access to your content, and how they interact with your pages. The table provides the following information.
2821

29-
| Column | Details |
30-
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
31-
| Crawler | The name of the AI crawler and the operator that owns it. |
32-
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/#categories). |
33-
| Requests | Total allowed and unsuccessful requests with trend chart. Unsuccessful requests may come from any rule or response error, not just the block action in AI Crawl Control. |
34-
| Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file. |
35-
| Action | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). |
22+
| Column | Details |
23+
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
24+
| Crawler | The name of the AI crawler and the operator that owns it. |
25+
| Category | The category of the AI crawler. Refer to [Verified bot categories](/bots/concepts/bot/verified-bots/#categories). |
26+
| Requests | The total number of allowed and unsuccessful requests, with trend chart. Unsuccessful requests may come from any rule or response error, not just the block action in AI Crawl Control. |
27+
| Robots.txt violations | The number of times the AI crawler has violated your <GlossaryTooltip term="robots.txt">`robots.txt`</GlossaryTooltip> file. |
28+
| Action | The action you wish to take for the AI crawler. Refer to [Take action for each AI crawler](/ai-crawl-control/features/manage-ai-crawlers/#take-action-for-each-ai-crawler). |
3629

3730
:::note[Quality of AI crawler detection]
3831
On the free plan, AI Crawl Control identifies AI crawlers based on their [user agent strings](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent). This enables AI Crawl Control to detect well-known, self-identifying AI crawlers.
@@ -112,7 +105,7 @@ Note that you can configure the response that gets returned when blocking an AI
112105

113106
<Example title="Charge for crawl (private beta)">
114107

115-
- **Summary:** You can charge the owner of the AI crawler for each crawl request.
108+
- **Summary:** You can charge the owner of the AI crawler for each successful crawl request.
116109
- **When to use:** Charge AI crawlers when your content has training value, and you want to explore monetization options.
117110
- **Implementation:** From the **Actions** column, select **Charge**.
118111

@@ -142,10 +135,11 @@ This provides you with a channel to open dialogue with the AI crawler owner, and
142135
To edit these values:
143136

144137
<Steps>
145-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and
146-
select your account and domain. 2. Go to **AI Crawl Control**. 3. Go to the
147-
**Settings** tab. 4. Under **Block response**, select **Edit**. 5. Once you
148-
have edited the values, select **Save**.
138+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain.
139+
2. Go to **AI Crawl Control**.
140+
3. Go to the **Settings** tab.
141+
4. Under **Block response**, select **Edit**.
142+
5. Once you have edited the values, select **Save**.
149143
</Steps>
150144

151145
:::note

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings.mdx

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Enable in account settings
3-
pcx_content_type: how_to
3+
pcx_content_type: how-to
44
sidebar:
55
order: 2
66
---
@@ -25,9 +25,9 @@ click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owne
2525

2626
To configure pay per crawl, you must have the following:
2727

28-
- **Cloudflare account** - You need an active Cloudflare account with domains added
29-
- **Domain on Cloudflare** - Your domain must be using Cloudflare's nameservers or have DNS records managed by Cloudflare
30-
- **Administrator access** - Account-level configuration requires Administrator or Super Administrator permissions
28+
- **Cloudflare account**: You need an active Cloudflare account with domains added
29+
- **Domain on Cloudflare**: Your domain must be using Cloudflare's nameservers, or have DNS records managed by Cloudflare
30+
- **Administrator access**: You need Administrator or Super Administrator permissions for account-level configuration
3131

3232
## Configure domain access
3333

@@ -41,7 +41,7 @@ An Administrator or Super Administrator with access to all domains must select w
4141
4. Set the **Visibility** to **Visible** for each domain you want to configure.
4242
</Steps>
4343

44-
:::note[Visibility vs. security]
44+
:::note[Visibility vs Security]
4545
Setting a domain to **Visible** will not affect security rules. This only makes the pay per crawl configuration controls accessible to domain-level administrators.
4646
:::
4747

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,4 +56,4 @@ Cloudflare manages the complete billing lifecycle:
5656
### Limitations
5757

5858
- Your accrued balance is not currently visible in the dashboard. You can request balance updates from your Cloudflare team.
59-
- Payouts are subject to settlement periods and minimum payout thresholds
59+
- Payouts are subject to settlement periods and minimum payout thresholds.

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/monitor-activity.mdx

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Monitor activity
3-
pcx_content_type: how_to
3+
pcx_content_type: how-to
44
sidebar:
55
order: 5
66
---
@@ -21,7 +21,7 @@ click C "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owne
2121
click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/"
2222
```
2323

24-
After configuring pay per crawl, monitor crawler activity to understand how AI crawlers interact with your content and track your earnings.
24+
After configuring pay per crawl, monitor crawler activity to understand how AI crawlers interact with your content, and track your earnings.
2525

2626
## View crawler activity
2727

@@ -35,11 +35,11 @@ After configuring pay per crawl, monitor crawler activity to understand how AI c
3535
The metrics help you understand:
3636

3737
- Which crawlers are accessing your content
38-
- How often they're being charged
38+
- How often they are being charged
3939
- Request patterns and trends
4040
- Robots.txt violations
4141

42-
For detailed information about available metrics, refer to [View AI Crawl Control metrics](/ai-crawl-control/features/analyze-ai-traffic/#view-ai-audit-metrics).
42+
For detailed information about available metrics, refer to [View AI Crawl Control metrics](/ai-crawl-control/features/analyze-ai-traffic/#view-ai-crawl-control-metrics).
4343

4444
:::note[Balance visibility]
4545
Your accrued earnings balance is not currently visible in the dashboard. You can request balance updates from your Cloudflare team.
@@ -53,6 +53,8 @@ Consider updating your `robots.txt` file to clearly indicate which pages should
5353

5454
### Ongoing optimization
5555

56+
Do the following to ensure you are using pay per crawl most effectively:
57+
5658
- Review crawler activity regularly to identify patterns
5759
- Adjust pricing based on demand and content value
5860
- Modify crawler actions (charge, allow, block) as needed

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge.mdx

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Select crawlers to charge
3-
pcx_content_type: how_to
3+
pcx_content_type: how-to
44
sidebar:
55
order: 4
66
---
@@ -21,17 +21,17 @@ click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owne
2121
click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/"
2222
```
2323

24-
Once you've enabled pay per crawl and set a price, you can specify which AI crawlers to charge for accessing your content.
24+
Once you have enabled pay per crawl and set a price, you can specify which AI crawlers to charge for accessing your content.
2525

2626
{/* prettier-ignore */}
2727
<Steps>
2828
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
2929
2. Go to **AI Crawl Control**.
3030
3. Go to the **Crawlers** tab.
3131
4. For each crawler, choose an action from the **Actions** column:
32-
- **Charge** - Charge the set price for successful content access
33-
- **Allow** - Allow free access without charging
34-
- **Block** - Block access completely
32+
- **Charge**: Charge the set price for successful content access
33+
- **Allow**: Allow free access without charging
34+
- **Block**: Block access completely
3535
</Steps>
3636

3737
:::tip[Search Engine Crawlers and SEO]

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Set a pay per crawl price
3-
pcx_content_type: how_to
3+
pcx_content_type: how-to
44
sidebar:
55
order: 3
66
---
@@ -21,20 +21,20 @@ click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owne
2121
click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/"
2222
```
2323

24-
Once your domain's visibility is set to **Visible** in Account Settings, you can Set a pay <br/>per crawl price and enable pay per crawl for that domain.
24+
Once your domain's visibility is set to **Visible** in Account Settings, you can set a pay per crawl price and enable pay per crawl for that domain.
2525

2626
{/* prettier-ignore */}
2727
<Steps>
2828
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
2929
2. Go to **AI Crawl Control**.
3030
3. Go to the **Settings** tab.
3131
4. In the **Pay Per Crawl** card, select **Enable**.
32-
5. Set your per-crawl price - this is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler.
32+
5. Set your per crawl price - this is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler.
3333
6. Select **Save**.
3434
</Steps>
3535

3636
After enabling and setting a price, the domain's status in Account Settings will change to **Enabled**.
3737

38-
:::tip[Pricing considerations]
38+
:::note[Pricing considerations]
3939
The minimum price is $0.01 USD per crawl. Consider your content value and expected crawler volume when setting your price.
4040
:::

src/content/docs/ai-crawl-control/get-started.mdx

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,7 @@ head:
1111
description: Learn how to set up AI Crawl Control.
1212
---
1313

14-
import {
15-
Details,
16-
Render,
17-
Steps,
18-
Tabs,
19-
TabItem,
20-
GlossaryTooltip,
21-
} from "~/components";
14+
import { Details, Render, Steps, Tabs, TabItem, GlossaryTooltip } from "~/components";
2215

2316
This guide instructs you on how to:
2417

0 commit comments

Comments
 (0)