-
Notifications
You must be signed in to change notification settings - Fork 10k
[Bots] Web Bot Auth docs #23099
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
[Bots] Web Bot Auth docs #23099
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
c4232ad
Changes for WBA
Oxyjun ba0e6f2
Structure change
Oxyjun 699adac
Adding FAQ
Oxyjun 46dc052
Apply suggestions from code review
Oxyjun d49ba64
Fixing FAQs, adding more info on response headers
Oxyjun d22d2df
Review part 1 - inline suggestions
Oxyjun e3ae398
Applying review suggestions part 2
Oxyjun 02b4271
Update src/content/docs/bots/concepts/bot/verified-bots/overview.mdx
Oxyjun 14c8908
Squashed commit of the following:
Oxyjun d2ca445
Merge commit '0264d07c42c5107203aaabc038fd6efc1fb1134a' into jun/bots…
Oxyjun 2922787
Adding link to future blog
Oxyjun 3850658
Final round of fixes
Oxyjun e5c66c6
Adding WBA release note
Oxyjun dbaad35
Merge commit '3c081196eea0881a527e459e4b8eed4fabad4979' into jun/bots…
Oxyjun File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
25 changes: 25 additions & 0 deletions
25
src/content/docs/bots/concepts/bot/verified-bots/requirements.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| --- | ||
| pcx_content_type: concept | ||
| title: Verified bots requirements | ||
| sidebar: | ||
| order: 3 | ||
| label: Requirements | ||
|
|
||
| --- | ||
|
|
||
| import { GlossaryTooltip } from "~/components" | ||
|
|
||
| To add a bot to Cloudflare's list of <GlossaryTooltip term="verified bot">verified bots</GlossaryTooltip>, the bot must meet the following requirements: | ||
|
|
||
| 1. The bot must follow [verified bots policy](/bots/concepts/bot/verified-bots/policy/). | ||
| 2. The bot must be verified using one of the [verification methods](/bots/concepts/bot/verified-bots/verification/). | ||
|
|
||
| Once Cloudflare verifies a bot, it will appear on the [Cloudflare Radar's list of verified bots](https://radar.cloudflare.com/verified-bots). | ||
|
|
||
| ## Transient false negatives | ||
|
|
||
| Once Cloudflare lists a bot as a verified bot, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the bot for a defined period of time. | ||
|
|
||
| It takes 24 hours for an inactive IP to be removed as a verified bot. | ||
|
|
||
| A bot can remain unlisted until Cloudflare sees traffic being sourced from the bot. When the bot is revalidated, it is listed as a verified bot again. |
213 changes: 213 additions & 0 deletions
213
src/content/docs/bots/concepts/bot/verified-bots/verification.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,213 @@ | ||
| --- | ||
| pcx_content_type: concept | ||
| title: Verification methods | ||
| sidebar: | ||
| order: 7 | ||
| label: Verification methods | ||
|
|
||
| --- | ||
|
|
||
| import { GlossaryTooltip, Steps } from "~/components" | ||
|
|
||
| To submit a verified bot that Cloudflare is not [currently tracking](https://radar.cloudflare.com/verified-bots), fill out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard for the fastest possible results. Bot operators who prefer not to create a free Cloudflare account can do so using our [old form](https://docs.google.com/forms/d/e/1FAIpQLSdqYNuULEypMnp4i5pROSc-uP6x65Xub9svD27mb8JChA_-XA/viewform?usp=sf_link), but the waiting time is up to several weeks for verified bot requests to be evaluated. | ||
|
|
||
| Cloudflare can verify a bot in two ways: | ||
|
|
||
| - **Web Bot Auth**: An authentication method which leverages cryptographic signatures in HTTP messages to verify requests that come from an automated bot. | ||
| - **IP validation**: An authentication method which identifies a bot by their range of IP addresses. | ||
|
|
||
| ## Web Bot Auth | ||
|
|
||
| To authenticate a bot using Web Bot Auth, you need to: | ||
|
|
||
| 1. Generate a valid signing key. | ||
| 2. Publish and host a URL which contains the public key derived from your signing key. | ||
| 3. Register your key directory URL with Cloudflare. | ||
|
|
||
| ### 1. Generate a valid signing key | ||
|
|
||
| You need to generate a signing key which will be used to authenticate your bot's requests. | ||
|
|
||
| {/* prettier-ignore */} | ||
| <Steps> | ||
| 1. Generate a unique [Ed25519](https://ed25519.cr.yp.to/) private key to sign your requests. This example uses the [OpenSSL](https://openssl-library.org/) `genpkey` command: | ||
|
|
||
| ```sh | ||
| openssl genpkey -algorithm ed25519 -out private-key.pem | ||
| ``` | ||
| 2. Extract your public key. | ||
|
|
||
| ```sh | ||
| openssl pkey -in private-key.pem -pubout -out public-key.pem | ||
| ``` | ||
| 3. Convert the public key to JSON Web Key (JWK) using a tool of your choice. This example uses [`jwker`](https://github.com/jphastings/jwker) command line application. | ||
| ```sh | ||
| go install github.com/jphastings/jwker/cmd/jwker@latest | ||
| jwker public-key.pem public-key.jwk | ||
| ``` | ||
| </Steps> | ||
|
|
||
| By following these steps, you have generated a private key and a public key, then converted the public key to a JWK. | ||
|
|
||
| ### 2. Host a key directory | ||
|
|
||
| You need to host a key directory which creates a way for Cloudflare to authenticate your bot's requests. | ||
|
|
||
| <Steps> | ||
| 1. Host a key directory at a well known message signatures directory. The key directory should serve a JSON Web Key Set (JWKS) including the public key derived from your signing key. | ||
|
|
||
| An example directory would be: | ||
| ```txt | ||
| /.well-known/http-message-signatures-directory/ | ||
| ``` | ||
| 2. Serve the web page over HTTPS (not HTTP). | ||
| 3. Sign your HTTP response using the HTTP message signature specification by attaching one signature per key in your key directory. This ensures no one else can mirror your directory and attempt to register on your behalf. Your response must include the following headers: | ||
| - `Signature`: TBD | ||
| - `Signature-Input`: TBD | ||
|
|
||
| The following example shows the annotated request and response with required headers against `https://example.com`. | ||
| ```txt | ||
| GET /.well-known/http-message-signatures-directory HTTP/1.1 | ||
| Host: example.com | ||
| Accept: application/http-message-signatures-directory+json | ||
|
|
||
| HTTP/1.1 200 OK | ||
| Content-Type: application/http-message-signatures-directory+json | ||
| Signature: sig1=:TD5arhV1ved6xtx63cUIFCMONT248cpDeVUAljLgkdozbjMNpJGr/WAx4PzHj+WeG0xMHQF1BOdFLDsfjdjvBA==: | ||
| Signature-Input: sig1=("@authority");alg="ed25519";keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";nonce="ZO3/XMEZjrvSnLtAP9M7jK0WGQf3J+pbmQRUpKDhF9/jsNCWqUh2sq+TH4WTX3/GpNoSZUa8eNWMKqxWp2/c2g==";tag="http-message-signatures-directory";created=1750105829;expires=1750105839 | ||
| Cache-Control: max-age=86400 | ||
| { | ||
| "keys": [{ | ||
| "kty": "OKP", | ||
| "crv": "Ed25519", | ||
| "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs", // Base64 URL-encoded public key, with no padding | ||
| }] | ||
| } | ||
| ``` | ||
| </Steps> | ||
|
|
||
| :::note | ||
| This URL serves a standard JSON Web Key Set. Besides `x`, `crv`, and `kty`, you can include other standard JSON Web Key parameters, and you may publish non-Ed25519 keys as well. Multiple Ed25519 keys are acceptable as well. | ||
|
|
||
| Cloudflare will ignore all other key types and key parameters except those containing `kty`, `crv`, and `x` formatted above. Do not include information that would leak your private key, such as the `d` parameter. | ||
| ::: | ||
|
|
||
| You can use the Cloudflare-developed [`http-signature-directory` CLI tool](https://crates.io/crates/http-signature-directory) to assist you in validating your directory. | ||
|
|
||
| ### 3. Register your bot and key directory | ||
|
|
||
| You need to register your bot and its key directory to add your bot to the list of verified bots. | ||
|
|
||
| <Steps> | ||
| 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. | ||
| 2. Go to **Manage Account** > **Configurations**. | ||
| 3. Go to the **Verified Bots** tab. | ||
| 4. For **Verification Method**: select **Request Signature**. | ||
| 5. For **Validation Instructions**: enter the URL of your key directory. You can additionally supply User Agents values (and their match patterns) that will be sent by your bot. | ||
| 6. Select **Submit**. | ||
| </Steps> | ||
|
|
||
| Cloudflare accepts all valid Ed25519 keys found in your key directory. In the event a key already exists in Cloudflare's registered database, Cloudflare will work with you to supply a new key, or rotate your existing key. | ||
|
|
||
| :::note[Estimated review time] | ||
| The estimated review time is approximately 1 week. | ||
|
|
||
| After successful verification, you will be able to send verified requests. | ||
| ::: | ||
|
|
||
| ### 4. (After verification) Sign your requests | ||
|
|
||
| After your bot has been successfully verified, you need to sign your bot's requests. | ||
|
|
||
| <Steps> | ||
| 1. Choose a set of components to sign. A component is either an HTTP header, or any [derived components](https://www.rfc-editor.org/rfc/rfc9421#name-derived-components) in the HTTP Message Signatures specification. Cloudflare recommends the following: | ||
| - Choose at least the `@authority` derived component, which represents the domain you are sending requests to. For example, a request to `https://example.com` will be interpreted to have an `@authority` of `example.com`. | ||
| - Use components that only contain ASCII values. HTTP Message Signature specification disallows non-ASCII characters, which will result in failure to validate your bot's requests. | ||
|
|
||
| :::note[Use components with only ASCII values] | ||
| Cloudflare currently does not support `bs` or `sf` parameter designed to serialize non-ASCII values into ASCII equivalents. | ||
| ::: | ||
| - Add a `Content-Digest` header if you wish to sign your [message content](https://www.rfc-editor.org/rfc/rfc9421#name-message-content), then specify `Content-Digest` as a component to sign. | ||
| 2. [Calculate the base64 URL-encoded JWK thumbprint](https://www.rfc-editor.org/rfc/rfc8037.html#appendix-A.3) associated with your Ed25519 public key registered with Cloudflare. | ||
| 3. Construct a [`Signature-Input` header](https://www.rfc-editor.org/rfc/rfc9421#name-the-signature-input-http-fi) over your chosen components. The header must meet the following requirements. | ||
|
|
||
| | Required component parameter | Requirement | | ||
| | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | `tag` | This should be equal to `web-bot-auth`. | | ||
| | `alg` | This should be equal to `ed25519`. | | ||
| | `keyid` | This should be equal to the thumbprint computed in step 2. | | ||
| | `created` | This should be equal to a `Unix` timestamp associated with when the message was sent by your application. | | ||
| | `expires` | This should be equal to a `Unix` timestamp associated with when Cloudflare should no longer attempt to verify the message. A short `expires` reduces the likelihood of replay attacks, and Cloudflare recommends choosing suitable short-lived intervals. | | ||
| 4. Construct a [`Signature` header](https://www.rfc-editor.org/rfc/rfc9421#name-the-signature-http-field) over your chosen components. | ||
| 5. Construct a [`Signature-Agent` header](https://www.ietf.org/archive/id/draft-meunier-http-message-signatures-directory-00.html#name-header-field-definition) that points to your key directory. Note that Cloudflare will fail to verify a message if: | ||
| - The message includes a `Signature-Agent` header that is not an `https://`. | ||
| - The message includes a valid URI but do not enclose it in double quotes. | ||
| - The message has a valid `Signature-Agent` header, but does not include it in the component list in `Signature-Input`. | ||
| 6. Attach these three headers to your bot's requests. | ||
| </Steps> | ||
|
|
||
| An example request may look like this: | ||
|
|
||
| ```txt | ||
| Signature-Agent: "https://signature-agent.test" | ||
| Signature-Input: sig2=("@authority" "signature-agent") | ||
| ;created=1735689600 | ||
| ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U" | ||
| ;alg="ed25519" | ||
| ;expires=1735693200 | ||
| ;nonce="e8N7S2MFd/qrd6T2R3tdfAuuANngKI7LFtKYI/vowzk4lAZYadIX6wW25MwG7DCT9RUKAJ0qVkU0mEeLElW1qg==" | ||
| ;tag="web-bot-auth" | ||
| Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==: | ||
| ``` | ||
|
|
||
| ## IP validation | ||
|
|
||
| There are two type of IP validation: public IP list and reverse DNS. | ||
|
|
||
| ### Public IP List | ||
|
|
||
| To verify a bot using a public IP list, you need to provide: | ||
|
|
||
| - A fixed and limited set of IP addresses, which can be verified via publicly accessible plain-text, `JSON`, or `CSV`. | ||
| - IP addresses used solely by the bot owner. | ||
| - A user-agent match pattern. | ||
|
|
||
| ### Reverse DNS | ||
|
|
||
| To verify a bot using reverse DNS, you need to provide: | ||
|
|
||
| - A list of domain suffixes to validate DNS records. | ||
| - IP addresses should have PTR records set correctly. | ||
| - A user-agent match pattern. | ||
|
|
||
| ## Generic user-agents | ||
|
|
||
| User-agent patterns that match generic user-agents will be rejected by the Verified Bots API. When you add a user-agent pattern that is considered very common to the Verified Bot form, you may encounter an error message that will prompt you to correct the user-agent before you can submit again. | ||
|
|
||
| Generic user-agents include: | ||
|
|
||
| - `Dart` | ||
| - `Go-http-client` | ||
| - `GuzzleHttp` | ||
| - `Google Chrome` | ||
| - `Mozilla Firefox` | ||
| - `Safari` | ||
| - `Nessus` | ||
| - `Websocket++` | ||
| - `cloudflare-go` | ||
| - `fasthttp` | ||
| - `got` | ||
| - `nginx-ssl early hints` | ||
| - `node` | ||
| - `node-fetch` | ||
| - `okhttp` | ||
| - `python-requests` | ||
| - `uTorrent` | ||
|
|
||
|
|
||
| ## Additional resources | ||
Oxyjun marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| You may wish to refer to the following resources. | ||
|
|
||
| - Cloudflare's [`web-bot-auth` library in Rust](https://crates.io/crates/web-bot-auth). | ||
| - Cloudflare's [`web-bot-auth` npm package in Typescript](https://www.npmjs.com/package/web-bot-auth). | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.