-
Notifications
You must be signed in to change notification settings - Fork 10k
[AI Audit] Initialising AI Audit docs #22950
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 4 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
bc249b4
Initialising AI Audit docs
Oxyjun bf572b5
Fixing product yaml file.
Oxyjun 95d5ae8
Setting up more pages for AI Audit docs
Oxyjun cacb286
Documenting AI Audit metrics
Oxyjun 543f9c3
Iterating over feedback part 1
Oxyjun 054fa20
changelog update
kodster28 e952c41
proxy clarification
kodster28 968ff6c
Adding more content, matching text to current UI.
Oxyjun ec02bcb
Adding docs on Violations only toggle.
Oxyjun ebf9e01
Removing inaccuracy.
Oxyjun 5b165ca
Wording clarification.
Oxyjun ce8b300
Fixing incorrect sentence
Oxyjun 2baeb44
minor tweaks
kodster28 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
29 changes: 29 additions & 0 deletions
29
src/content/docs/ai-audit/features/ai-crawler-detection.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| title: AI crawler detection | ||
| pcx_content_type: concept | ||
| sidebar: | ||
| order: 2 | ||
| --- | ||
|
|
||
| AI Audit metrics provides you with insight on how AI crawlers are interacting with your website. | ||
|
|
||
| ## View AI Audit metrics | ||
|
|
||
| AI Audit provides you with the following metrics to help you understand how AI crawlers are interacting with your website. | ||
|
|
||
| | Metric | Description | | ||
| | ------------------------------------- | ----------------------------------------------------------------------------- | | ||
| | Total AI crawler requests | The total number of requests to crawl your website, from all AI crawlers | | ||
| | Blocked requests | The number of requests you have blocked (by blocking AI crawlers) | | ||
| | Allowed requests | The number of requests you have allowed (by allowing AI crawlers) | | ||
| | Request by AI crawlers | A graph which displays the number of crawl requests from each AI crawler | | ||
| | Most popular paths by AI crawlers | The most popular pages crawled by AI crawlers, for each AI crawler | | ||
| | Hosts | The owner of the AI crawler | | ||
| | Overall popular paths | The most popular pages crawled by AI crawlers, from all AI crawlers | | ||
| | Most active AI crawlers by bot owners | The AI crawler owners with the highest number of requests to access your site | | ||
|
|
||
| ## Filter date range | ||
|
|
||
| You can use the date filter to choose the period of time you wish to analyze. | ||
|
|
||
| Click on the date filter dropdown and select the range of dates. The values of the AI Audit metrics will update according to your filter. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| --- | ||
| title: Features | ||
| pcx_content_type: navigation | ||
| sidebar: | ||
| group: | ||
| hideIndex: true | ||
| order: 5 | ||
| --- | ||
|
|
||
| import { DirectoryListing } from "~/components"; | ||
|
|
||
| <DirectoryListing /> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| --- | ||
| title: Get started | ||
| pcx_content_type: get-started | ||
| sidebar: | ||
| order: 2 | ||
| group: | ||
| badge: beta | ||
| head: | ||
| - tag: title | ||
| content: Get started with Cloudflare AI Audit | ||
| description: Learn how to set up AI Audit. | ||
|
|
||
| --- | ||
|
|
||
| import { Render, Steps } from "~/components"; | ||
|
|
||
Oxyjun marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ## Prerequisites | ||
|
|
||
| 1. Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up/). | ||
| 2. [Connect your domain to Cloudflare](/fundamentals/manage-domains/add-site/). | ||
Oxyjun marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## 1. Block all AI crawlers | ||
|
|
||
| To enable AI Audit: | ||
Oxyjun marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| <Steps> | ||
| 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. | ||
| 2. Go to **AI Audit**. | ||
| 3. From **AI Crawlers** tab, toggle **Block AI Crawlers**. | ||
| 4. From the **Bot traffic** page, under **Block AI Bots**, select **Enable**. | ||
| </Steps> | ||
|
|
||
| ## 2. Block specific AI crawlers (Enterprise plan only) | ||
|
|
||
| Customers on the Enterprise plan can choose to only block specific AI crawlers, while allowing others. | ||
|
|
||
| <Steps> | ||
| 1. Go to the **AI Crawlers** tab. | ||
| 2. Toggle the switch under the **Block** column for the AI crawlers you wish to block. | ||
| </Steps> | ||
|
|
||
| ## 3. Review detected AI crawlers | ||
|
|
||
| Review the AI crawlers detected on your site in the **Metrics** tab of the Cloudflare dashboard for key metrics. | ||
|
|
||
| Refer to [AI crawler detection](/ai-audit/ai-crawler-detection/) for more information. | ||
Oxyjun marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Oxyjun marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| --- | ||
| title: AI Audit | ||
| pcx_content_type: overview | ||
| sidebar: | ||
| order: 1 | ||
| badge: Beta | ||
| head: | ||
| - tag: title | ||
| content: Overview | ||
| description: AI Audit is a tool which allows you to analyze and control how third-party AI crawlers interact with your website. | ||
| --- | ||
|
|
||
| import { Description, Feature, FeatureTable, Plan, LinkButton } from "~/components"; | ||
|
|
||
| <Description> | ||
|
|
||
| Analyze and control third-party AI crawlers in your website. | ||
|
|
||
| </Description> | ||
|
|
||
| <Plan type="all" /> | ||
|
|
||
| AI Audit helps manage AI crawlers on your website by providing visibility on which crawlers are accessing your webpage, and allowing you to block unwanted crawlers. | ||
Oxyjun marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| <LinkButton href="/ai-audit/get-started/">Get started </LinkButton> | ||
|
|
||
| :::note[Beta phase] | ||
| AI Audit is currently only available as a beta product. | ||
| ::: | ||
|
|
||
| --- | ||
|
|
||
| ## Features | ||
|
|
||
| <Feature | ||
| header="AI crawler detection" | ||
| href="/ai-audit/features/ai-crawler-detection/" | ||
| cta="View AI crawlers" | ||
| > | ||
| Displays information about AI crawlers in your domains' pages. | ||
| </Feature> | ||
|
|
||
Oxyjun marked this conversation as resolved.
Show resolved
Hide resolved
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| --- | ||
| title: List of allowed AI crawlers | ||
| pcx_content_type: concept | ||
| sidebar: | ||
| order: 2 | ||
| --- | ||
|
|
||
| TBC | ||
Oxyjun marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| --- | ||
| title: Reference | ||
| pcx_content_type: navigation | ||
| sidebar: | ||
| group: | ||
| hideIndex: true | ||
| order: 10 | ||
| --- | ||
|
|
||
| import { DirectoryListing } from "~/components"; | ||
|
|
||
| <DirectoryListing /> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| --- | ||
| pcx_content_type: changelog | ||
| title: Release notes | ||
| release_notes_file_name: | ||
| - ai-audit | ||
| sidebar: | ||
| order: 100 | ||
| --- | ||
|
|
||
| import { ProductReleaseNotes } from "~/components"; | ||
Oxyjun marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| {/* <!-- Actual content lives in /src/content/release-notes/ai-audit.yaml. Update the file there for new entries to appear here. For more details, refer to https://developers.cloudflare.com/style-guide/documentation-content-strategy/content-types/changelog/#yaml-file --> */} | ||
|
|
||
| <ProductReleaseNotes /> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| name: AI Audit | ||
| product: | ||
| title: AI Audit | ||
| url: /ai-audit/ | ||
| group: Core platform | ||
| meta: | ||
| title: AI Audit | ||
| description: Analyze and control third-party AI crawlers in your website | ||
| author: "@cloudflare" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| link: "/ai-audit/changelog/" | ||
| productName: AI Audit | ||
| productLink: "/ai-audit/" | ||
| productArea: Core platform | ||
| productAreaLink: /fundamentals/reference/changelog/platform/ | ||
| entries: | ||
| - publish_date: "2025-06-09" | ||
| title: Documentation for AI Audit | ||
| description: There is now documentation for AI Audit | ||
|
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.