You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
import { Steps, GlossaryTooltip, Tabs, TabItem, Example } from"~/components";
9
9
10
10
AI Audit enables you to take specific action for each AI crawler.
11
11
@@ -44,27 +44,11 @@ The **AI Crawlers** tab displays a table of AI crawlers that are requesting acce
44
44
## Take action for each AI crawler
45
45
46
46
<Tabs>
47
-
<TabItemlabel="Without pay per crawl">
48
-
49
-
For each AI crawler, you can choose to block or allow access.
50
-
51
-
### Block access
52
-
53
-
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
54
-
55
-
Use the toggles in the **Block** column to block specific AI crawlers from accessing your content.
56
-
57
-
### Allow access
58
-
59
-
You can allow an AI crawler to scrape your content. Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
60
-
61
-
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
62
-
63
-
</TabItem><TabItemlabel="With pay per crawl">
47
+
<TabItemlabel="With pay per crawl">
64
48
65
49
For each AI crawler, you can take one of three actions: allow, charge, or block.
66
50
67
-
### Allow access
51
+
<Exampletitle="Allow access">
68
52
69
53
You can allow an AI crawler to scrape your content.
70
54
@@ -74,19 +58,43 @@ Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/mana
74
58
75
59
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
76
60
77
-
### Charge for crawl (Enterprise only)
61
+
</Example>
62
+
63
+
<Exampletitle="Charge for crawl (Enterprise only)">
78
64
79
65
You can charge the owner of the AI crawler for each crawl request.
80
66
81
67
From the **Actions** column, select **Charge**.
82
68
83
69
For more information, refer to [What is Pay Per Crawl?](/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl/).
84
70
85
-
### Block access
71
+
</Example>
72
+
73
+
<Exampletitle="Block access">
86
74
87
75
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
88
76
89
77
From the **Actions** column, select **Block**.
78
+
</Example>
79
+
80
+
</TabItem>
81
+
<TabItemlabel="Without pay per crawl">
82
+
83
+
For each AI crawler, you can choose to allow or block access.
84
+
85
+
<Exampletitle="Allow access">
86
+
87
+
You can allow an AI crawler to scrape your content. Note that you can still choose to [Enforce `robots.txt`](/ai-audit/features/manage-ai-crawlers/#take-action-for-each-ai-crawler).
88
+
89
+
For more details on how this rule interacts with other Cloudflare settings, refer to [How it works](/bots/concepts/bot/#how-it-works).
90
+
91
+
</Example>
92
+
93
+
<Exampletitle="Block access">
94
+
You can block an AI crawler to completely stop the AI crawler from scraping your webpage.
95
+
96
+
Use the toggles in the **Block** column to block specific AI crawlers from accessing your content.
0 commit comments