-
Notifications
You must be signed in to change notification settings - Fork 10k
[Bots] Signed agents #24668
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
[Bots] Signed agents #24668
Changes from 12 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
f6058c7
signed agents
patriciasantaana edf8436
update partial
patriciasantaana d87c38a
remove partial
patriciasantaana b1901c6
Merge branch 'production' into patricia/pcx18872-signed-agents
patriciasantaana 6dc5c00
fixes
patriciasantaana 753444a
redirects + naming
patriciasantaana b3fbe45
casing
patriciasantaana 896b08b
broken links
patriciasantaana ef4d6ef
broken links pt 2
patriciasantaana 1402a43
pm feedback
patriciasantaana 2271eb9
language fixes
patriciasantaana 375319e
misc edits
patriciasantaana a2b13f0
Apply suggestion from @marciocloudflare
patriciasantaana File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 41 additions & 0 deletions
41
src/content/docs/bots/concepts/bot/signed-agents/index.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| --- | ||
| pcx_content_type: overview | ||
| title: Signed agents | ||
| sidebar: | ||
| order: 3 | ||
| learning_center: | ||
| title: What is a bot? | ||
| link: https://www.cloudflare.com/learning/bots/what-is-a-bot/ | ||
|
|
||
| --- | ||
|
|
||
| A signed agent is controlled by an end user and a verified signature-agent from their Web Bot Auth implementation. | ||
|
|
||
| You can request for your agent to be added to Cloudflare's bots and agents directory by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard. | ||
|
|
||
| :::note | ||
| A bot cannot be registered as both a verified bot and a signed agent. Review Cloudflare's [verified bots](/bots/concepts/bot/verified-bots/) to determine how to identify your bot. | ||
| ::: | ||
|
|
||
| ## Signed agent requirement | ||
|
|
||
| For an agent to be recognized, it must meet the following requirements: | ||
|
|
||
| 1. The agent must follow the [signed agents policy](/bots/concepts/bot/signed-agents/policy/). | ||
| 2. The bot must be using [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/). | ||
|
|
||
| Once Cloudflare approves a signed agent, it should appear on [Cloudflare Radar's bots and agents directory](https://radar.cloudflare.com/verified-bots). | ||
|
|
||
| --- | ||
|
|
||
| ## Transient false negatives | ||
|
|
||
| Once Cloudflare lists an agent as a signed agent, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the agent for a defined period of time. | ||
|
|
||
| An agent can remain unlisted until Cloudflare sees traffic being sourced from the agent. When the agent is revalidated, it is listed as a signed agent again. | ||
|
|
||
| --- | ||
|
|
||
| ## Verification method | ||
|
|
||
| The bot must be verified using [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/). |
68 changes: 68 additions & 0 deletions
68
src/content/docs/bots/concepts/bot/signed-agents/policy.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| --- | ||
| pcx_content_type: reference | ||
| title: Signed agents policy | ||
| sidebar: | ||
| order: 3 | ||
| label: Policy | ||
|
|
||
| --- | ||
|
|
||
| In order to be listed by Cloudflare as a signed agent, your agent must conform to the below requirements. To provide the best possible protection to our customers, this policy may change in the future as we adapt to new bot behaviors. | ||
|
|
||
| ## Agent policy | ||
|
|
||
| ### Minimum traffic | ||
|
|
||
| An agent must have a minimum amount of traffic for Cloudflare to be able to find it in the sampled data. The minimum traffic should have more than 1,000 requests per day across multiple domains. | ||
|
|
||
| ### Minimum zones | ||
|
|
||
| Service must be made for a widespread use of zones. | ||
|
|
||
| #### Example | ||
|
|
||
| A bot crawling one site is not valid. | ||
|
|
||
| ### Agent identification | ||
|
|
||
| The user-agent field is optional as it is not required for Web Bot Authentication. | ||
|
|
||
| However, if you choose to provide a user-agent, it and the message signature must meet the following requirements: | ||
|
|
||
| - Have at least five characters. | ||
| - Must not contain special characters. | ||
| - Must not include the same user-agent of another verified service. | ||
|
|
||
| #### Example | ||
|
|
||
| `cloudflare-browser-rendering` is a valid message signature. | ||
|
|
||
| ### Service purpose | ||
|
|
||
| The purpose of the service should be benign or helpful to both the owner of a zone and the users of the service. The service cannot perform any of the following: | ||
|
|
||
| - Bot tooling | ||
| - Scalpers | ||
| - Credential-stuffing | ||
| - Directory-traversal scanning | ||
| - Excessive data scraping | ||
| - DDoS botnets | ||
|
|
||
| #### Example | ||
|
|
||
| Price scraping direct e-commerce competitors is not a valid use case. | ||
|
|
||
| ### Public documentation | ||
|
|
||
| The agent must have a publicly documented purpose and expected behavior. | ||
|
|
||
| --- | ||
|
|
||
| ## Breach of policy | ||
|
|
||
| If any of the requirements to validate are breached, a service will be removed from the signed agent list. | ||
|
|
||
| The following are examples of breaches of policy: | ||
|
|
||
| - The service has vulnerabilities that have not been patched. | ||
| - The disclosed purpose of the service does not reflect on the traffic. |
41 changes: 0 additions & 41 deletions
41
src/content/docs/bots/concepts/bot/verified-bots/categories.mdx
This file was deleted.
Oops, something went wrong.
79 changes: 72 additions & 7 deletions
79
src/content/docs/bots/concepts/bot/verified-bots/index.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,20 +1,85 @@ | ||
| --- | ||
| pcx_content_type: navigation | ||
| pcx_content_type: overview | ||
| title: Verified bots | ||
| sidebar: | ||
| group: | ||
| hideIndex: true | ||
| order: 2 | ||
| learning_center: | ||
| title: What is a bot? | ||
| link: https://www.cloudflare.com/learning/bots/what-is-a-bot/ | ||
|
|
||
| --- | ||
|
|
||
| import { Render, DirectoryListing } from "~/components"; | ||
| import { GlossaryTooltip } from "~/components" | ||
|
|
||
| <Render file="verified-bots" product="bots" /> | ||
| A verified bot is a bot which has been added to Cloudflare's list of <GlossaryTooltip term="verified bot">verified bots</GlossaryTooltip>. | ||
|
|
||
| Refer to the pages below for additional resources on verified bots. | ||
| You can request for your bot to be added to Cloudflare's bots and agents directory by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard. | ||
|
|
||
| <DirectoryListing /> | ||
| :::note | ||
| A bot cannot be registered as both a verified bot and a signed agent. Review Cloudflare's [signed agents](/bots/concepts/bot/signed-agents/) to determine how to identify your bot. | ||
| ::: | ||
|
|
||
| ## Verified bot requirement | ||
|
|
||
| For a bot to be verified, it must meet the following requirements: | ||
|
|
||
| 1. The bot must follow [verified bots policy](/bots/concepts/bot/verified-bots/policy/). | ||
| 2. The bot must be verified using one of the following verification methods: | ||
| - [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/) | ||
| - [IP validation](/bots/reference/bot-verification/ip-validation/) | ||
|
|
||
| Once Cloudflare approves a verified bot, it should appear on [Cloudflare Radar's bots and agents directory](https://radar.cloudflare.com/verified-bots). | ||
|
|
||
| --- | ||
|
|
||
| ## Transient false negatives | ||
|
|
||
| Once Cloudflare lists a bot as a verified bot, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the bot for a defined period of time. | ||
|
|
||
| It takes 24 hours for an inactive IP to be removed as a verified bot. | ||
|
|
||
| A bot can remain unlisted until Cloudflare sees traffic being sourced from the bot. When the bot is revalidated, it is listed as a verified bot again. | ||
|
|
||
| --- | ||
|
|
||
| ## Verification methods | ||
|
|
||
| The bot must be verified using one of the following validation methods: | ||
|
|
||
| - [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/) | ||
| - [IP validation](/bots/reference/bot-verification/ip-validation/) | ||
|
|
||
| --- | ||
|
|
||
| ## Categories | ||
|
|
||
| You can segment your verified bot traffic by its type and purpose by adding the Verified Bot Categories field `cf.verified_bot_category` as a filter criteria in [WAF Custom rules](/waf/custom-rules/), [Advanced Rate Limiting](/waf/rate-limiting-rules/), and Late Transform rules. | ||
|
|
||
| :::note | ||
|
|
||
patriciasantaana marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| The Verified Bot Categories field is not compatible with legacy Firewall rules. | ||
| ::: | ||
|
|
||
| | Name | String value | Example | | ||
| | -------------------------- | ---------------------------- | ----------------------------------------------------------------------- | | ||
| | Academic research | `Academic Research` | Library of Congress, TurnItInBot, Bibliothèque nationale de France | | ||
| | Accessibility | `Accessibility` | Accessible Web Bot | | ||
| | Advertising or marketing | `Advertising & Marketing` | Google Adsbot | | ||
| | Aggregators | `Aggregator` | Pinterest, Indeed Jobsbot | | ||
| | AI Assistant | `AI Assistant` | Perplexity-User, DuckAssistBot | | ||
| | AI Crawler | `AI Crawler` | Google Bard, ChatGPT bot | | ||
| | AI Search | `AI Search` | OAI-SearchBot | | ||
| | Archiver | `Archiver` | Internet Archive, CommonCrawl | | ||
| | Feed fetcher | `Feed Fetcher` | RSS or Podcast feed updaters | | ||
| | Monitoring or analytics | `Monitoring & Analytics` | Uptime Monitors | | ||
| | Page preview | `Page Preview` | Facebook, Slack, Twitter, or Discord Link Preview tools | | ||
| | Search engine crawler | `Search Engine Crawler` | Googlebot, Bingbot, Yandexbot, Baidubot | | ||
| | Search engine optimization | `Search Engine Optimization` | Google Lighthouse, GT Metrix, Pingdom, AddThis | | ||
| | Security | `Security` | Vulnerability Scanners, SSL Domain Control Validation (DCV) Check Tools | | ||
| | Social media marketing | `Social Media Marketing` | Brandwatch | | ||
| | Webhooks | `Webhooks` | Payment processors, WordPress Integration tools | | ||
| | Other | `Other` | | | ||
|
|
||
| ### Availability | ||
|
|
||
| Verified Bot Categories is available on all plans. | ||
37 changes: 0 additions & 37 deletions
37
src/content/docs/bots/concepts/bot/verified-bots/overview.mdx
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.