Skip to content

Commit 6dc5c00

Browse files
fixes
1 parent b1901c6 commit 6dc5c00

File tree

6 files changed

+32
-28
lines changed

6 files changed

+32
-28
lines changed

src/content/docs/bots/concepts/bot/signed-bots/index.mdx renamed to src/content/docs/bots/concepts/bot/signed-agents/index.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
pcx_content_type: overview
3-
title: Signed bots
3+
title: Signed agents
44
sidebar:
55
order: 3
66
learning_center:
@@ -23,7 +23,7 @@ A bot cannot be registered as both a verified bot and a signed agent. Review Clo
2323

2424
For an agent to be recognized, it must meet the following requirements:
2525

26-
1. The agent must follow the [signed agents policy](/bots/concepts/bot/signed-bots/policy/).
26+
1. The agent must follow the [signed agents policy](/bots/concepts/bot/signed-agents/policy/).
2727
2. The bot must be using [Web Bot Auth](/bots/reference/bot-verification/web-bot-auth/).
2828

2929
Once Cloudflare verifies a bot, it should appear on [Cloudflare Radar's list of verified bots](https://radar.cloudflare.com/verified-bots).
Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
pcx_content_type: reference
3-
title: Verified bots policy
3+
title: Signed agents policy
44
sidebar:
55
order: 3
66
label: Policy
@@ -10,35 +10,35 @@ sidebar:
1010

1111
In order to be listed by Cloudflare as a signed agent, your agent must conform to the below requirements. To provide the best possible protection to our customers, this policy may change in the future as we adapt to new bot behaviors.
1212

13-
## Agent Policy
13+
## Agent policy
1414

15-
### Minimum Traffic
15+
### Minimum traffic
1616

1717
An agent must have a minimum amount of traffic for Cloudflare to be able to find it in the sampled data. The minimum traffic should have more than 1,000 requests per day across multiple domains.
1818

19-
### Minimum Zones
19+
### Minimum zones
2020

2121
Service must be made for a widespread use of zones.
2222

2323
#### Example
2424

2525
A bot crawling one site is not valid.
2626

27-
### Agent Identification
27+
### Agent identification
2828

2929
The User-Agent field is optional as it is not required for Web Bot Authentication.
3030

3131
However, if you choose to provide a User-Agent, it and the message signature must meet the following requirements:
3232

33-
- Have at least 5 characters.
33+
- Have at least five characters.
3434
- Must not contain special characters.
3535
- Must not include the same user-agent of another verified service.
3636

3737
#### Example
3838

3939
`cloudflare-browser-rendering` is a valid message signature.
4040

41-
### Service Purpose
41+
### Service purpose
4242

4343
The purpose of the service should be benign or helpful to both the owner of a zone and the users of the service. The service cannot perform any of the following:
4444

@@ -53,15 +53,17 @@ The purpose of the service should be benign or helpful to both the owner of a zo
5353

5454
Mass price scraping direct e-commerce competitors is not a valid use case.
5555

56-
### Public Documentation
56+
### Public documentation
5757

5858
The agent must have a publicly documented purpose and expected behavior.
5959

60-
## Breach of Policy
60+
---
61+
62+
## Breach of policy
6163

6264
If any of the requirements to validate are breached, a service will be removed from the global allowlist.
6365

64-
#### Example breaches of policy
66+
The following are examples of breaches of policy:
6567

6668
- The service has vulnerabilities that have not been patched.
6769
- The disclosed purpose of the service does not reflect on the traffic.

src/content/docs/bots/concepts/bot/verified-bots/index.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ A verified bot is a bot which has been added to Cloudflare's list of <GlossaryTo
1616
You can request for your bot to be added to Cloudflare's bots and agents directory by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard.
1717

1818
:::note
19-
A bot cannot be registered as both a verified bot and a signed agent. Review Cloudflare's [signed bots](/bots/concepts/bot/signed-bots/) to determine how to identify your bot.
19+
A bot cannot be registered as both a verified bot and a signed agent. Review Cloudflare's [signed bots](/bots/concepts/bot/signed-agents/) to determine how to identify your bot.
2020
:::
2121

2222
## Verified bot requirement
@@ -80,6 +80,6 @@ The Verified Bot Categories field is not compatible with legacy Firewall rules.
8080
| Webhooks | `Webhooks` | Payment processors, WordPress Integration tools |
8181
| Other | `Other` | |
8282

83-
## Availability
83+
### Availability
8484

8585
Verified Bot Categories is available on all plans.

src/content/docs/bots/concepts/bot/verified-bots/policy.mdx

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -13,39 +13,39 @@ In order to be listed by Cloudflare as a <GlossaryTooltip term="verified bot" li
1313

1414
## Bot Policy
1515

16-
### Minimum Traffic
16+
### Minimum traffic
1717

1818
A bot or proxy must have a minimum amount of traffic for Cloudflare to be able to find it in the sampled data. The minimum traffic should have more than 1,000 requests per day across multiple domains.
1919

20-
### Minimum Zones
20+
### Minimum zones
2121

2222
Service must be made for a widespread use of zones.
2323

2424
#### Example
2525

2626
A bot crawling one site is not valid.
2727

28-
### Bot Identification
28+
### Bot identification
2929

3030
The user-agent or message signature with the following requirements:
3131

32-
- Have at least 5 characters.
32+
- Have at least five characters.
3333
- Must not contain special characters.
3434
- Must not include the same user-agent of another verified service.
3535

3636
#### Example
3737

38-
`GoogleBot/1.0` is a valid UA.
38+
`GoogleBot/1.0` is a valid user-agent.
3939

40-
### Domain Owner Consent
40+
### Domain owner consent
4141

4242
Domains should only be crawled with the explicit or implicit consent of the zone's owner or terms of use. Search engines crawlers must read the `robots.txt` to exclude paths to crawl from the owner.
4343

4444
#### Example
4545

4646
A tool trying to scalp inventories from different websites might be breaking terms of use while a search engine bot indexing websites but complying with `robots.txt` is a valid service.
4747

48-
### Service Purpose
48+
### Service purpose
4949

5050
The purpose of the service should be benign or helpful to both the owner of a zone and the users of the service. The service cannot perform any of the following:
5151

@@ -58,25 +58,27 @@ The purpose of the service should be benign or helpful to both the owner of a zo
5858

5959
#### Example
6060

61-
Price scraping direct ecommerce competitors is not a valid use case.
61+
Price scraping direct e-commerce competitors is not a valid use case.
6262

63-
### Crawling Etiquette
63+
### Crawling etiquette
6464

6565
The crawling etiquette should check `robots.txt` if crawling the whole website, and it should not attempt to crawl sensitive paths.
6666

6767
#### Example
6868

6969
If a search engine crawler skips `robots.txt`, it will be rejected.
7070

71-
### Public Documentation
71+
### Public documentation
7272

7373
The bot must have publicly documented expected behavior or user-agent format.
7474

75-
## Breach of Policy
75+
---
76+
77+
## Breach of policy
7678

7779
If any of the requirements to validate are breached, a service will be removed from the global allowlist.
7880

79-
#### Example breaches of policy
81+
The following are examples of breaches of policy:
8082

8183
- Adding a set of IPs that are not solely used by verified service.
8284
- The service IPs are breached by an attacker.

src/content/docs/bots/reference/bot-verification/index.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,6 @@ sidebar:
1010

1111
import { DirectoryListing } from "~/components"
1212

13-
Refer to the following pages for more information on Cloudflare's validation methods for [Verified] and [Signed]() bots.
13+
Refer to the following pages for more information on Cloudflare's validation methods for [Verified](/bots/concepts/bot/verified-bots/) and [Signed](/bots/concepts/bot/signed-agents/) bots.
1414

1515
<DirectoryListing />

src/content/docs/bots/reference/bot-verification/web-bot-auth.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ sidebar:
99

1010
import { GlossaryTooltip, Steps } from "~/components"
1111

12-
Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. Web Bot Auth is used as a verification method for [verified bots](/bots/concepts/bot/verified-bots/) and [signed bots](/bots/concepts/bot/signed-bots/).
12+
Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. Web Bot Auth is used as a verification method for [verified bots](/bots/concepts/bot/verified-bots/) and [signed bots](/bots/concepts/bot/signed-agents/).
1313

1414
It relies on two active IETF drafts: a [directory draft](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory) allowing the crawler to share their public keys, and a [protocol draft](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture) defining how these keys should be used to attach crawler's identity to HTTP requests.
1515

0 commit comments

Comments
 (0)