diff --git a/public/__redirects b/public/__redirects index e86a8dbec137557..2f81ee6afd77a79 100644 --- a/public/__redirects +++ b/public/__redirects @@ -244,6 +244,7 @@ /bots/get-started/bm-subscription/ /bots/get-started/bot-management/ 301 /bots/get-started/pro/ /bots/get-started/super-bot-fight-mode/ 301 /bots/additional-configurations/javascript-detections/ /cloudflare-challenges/challenge-types/javascript-detections/ 301 +/bots/troubleshooting/frequently-asked-questions/ /bots/frequently-asked-questions/ 301 #browser-rendering /browser-rendering/get-started/browser-rendering-with-DO/ /browser-rendering/workers-bindings/browser-rendering-with-do/ 301 diff --git a/src/content/docs/bots/concepts/bot/index.mdx b/src/content/docs/bots/concepts/bot/index.mdx index 748e69507741dc7..6e428243917c9b1 100644 --- a/src/content/docs/bots/concepts/bot/index.mdx +++ b/src/content/docs/bots/concepts/bot/index.mdx @@ -23,6 +23,8 @@ For more background, refer to [What is a bot?](https://www.cloudflare.com/learni +For more information, refer to [Verified bots](/bots/concepts/bot/verified-bots/overview/). + :::note The method for allowing or blocking verified bots depends on [your plan](/bots/get-started/). diff --git a/src/content/docs/bots/concepts/bot/verified-bots/categories.mdx b/src/content/docs/bots/concepts/bot/verified-bots/categories.mdx index 8827292f5ad5ae7..cd6fb55dfd3cd26 100644 --- a/src/content/docs/bots/concepts/bot/verified-bots/categories.mdx +++ b/src/content/docs/bots/concepts/bot/verified-bots/categories.mdx @@ -2,8 +2,9 @@ pcx_content_type: reference title: Verified bot categories sidebar: - order: 3 - label: Categories + order: 20 + label: Categories + --- You can segment your verified bot traffic by its type and purpose by adding the Verified Bot Categories field `cf.verified_bot_category` as a filter criteria in [WAF Custom rules](/waf/custom-rules/), [Advanced Rate Limiting](/waf/rate-limiting-rules/), and Late Transform rules. diff --git a/src/content/docs/bots/concepts/bot/verified-bots/ip-validation.mdx b/src/content/docs/bots/concepts/bot/verified-bots/ip-validation.mdx new file mode 100644 index 000000000000000..92791b9070240e1 --- /dev/null +++ b/src/content/docs/bots/concepts/bot/verified-bots/ip-validation.mdx @@ -0,0 +1,57 @@ +--- +pcx_content_type: concept +title: IP validation +sidebar: + order: 7 + label: IP validation + +--- + +import { GlossaryTooltip, Steps } from "~/components" + +The IP validation method aims to identify all of the IP addresses that a bot may use to send requests. + +Cloudflare can achieve this in two ways: + +- **Using IP list provided by the bot owner**: The bot owner can host a public list of IP ranges (for example, [Googlebot's list](https://developers.google.com/static/search/apis/ipranges/googlebot.json)). Cloudflare fetches and uses this list directly for validation. +- **Using Domain-based reverse DNS**: The bot owner can provide a domain (or set of domains) that their bot requests originate from. Cloudflare collects the IP addresses observed in the requests with the bot's user agent, and performs reverse DNS lookups. If the reverse DNS of an IP resolves to one of the provided domains, Cloudflare considers it valid and stores it. + +## Public IP List + +To verify a bot using a public IP list, you need to provide: + +- A fixed and limited set of IP addresses, which can be verified via publicly accessible plain-text, `JSON`, or `CSV`. +- IP addresses used solely by the bot owner. +- A user-agent match pattern. + +## Reverse DNS + +To verify a bot using reverse DNS, you need to provide: + +- A list of domain suffixes to validate DNS records. +- IP addresses should have PTR records set correctly. +- A user-agent match pattern. + +## Generic user-agents + +User-agent patterns that match generic user-agents will be rejected by the Verified Bots API. When you add a user-agent pattern that is considered very common to the Verified Bot form, you may encounter an error message that will prompt you to correct the user-agent before you can submit again. + +Generic user-agents include: + +- `Dart` +- `Go-http-client` +- `GuzzleHttp` +- `Google Chrome` +- `Mozilla Firefox` +- `Safari` +- `Nessus` +- `Websocket++` +- `cloudflare-go` +- `fasthttp` +- `got` +- `nginx-ssl early hints` +- `node` +- `node-fetch` +- `okhttp` +- `python-requests` +- `uTorrent` \ No newline at end of file diff --git a/src/content/docs/bots/concepts/bot/verified-bots/overview.mdx b/src/content/docs/bots/concepts/bot/verified-bots/overview.mdx new file mode 100644 index 000000000000000..086e8d42a7382a9 --- /dev/null +++ b/src/content/docs/bots/concepts/bot/verified-bots/overview.mdx @@ -0,0 +1,37 @@ +--- +pcx_content_type: concept +title: Overview +sidebar: + order: 3 + label: Overview + +--- + +import { GlossaryTooltip } from "~/components" + +A **verified bot** is a bot which has been added to Cloudflare's list of verified bots. + +You can request for your bot to be added to Cloudflare's list of verified bots by filling out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard. + +## Verified bot requirement + +For a bot to be verified, it must meet the following requirements: + +1. The bot must follow [verified bots policy](/bots/concepts/bot/verified-bots/policy/). +2. The bot must be verified using one of the following verification methods: + - [Web Bot Auth](/bots/concepts/bot/verified-bots/web-bot-auth/) + - [IP validation](/bots/concepts/bot/verified-bots/ip-validation/) + +Once Cloudflare verifies a bot, it should appear on the [Cloudflare Radar's list of verified bots](https://radar.cloudflare.com/verified-bots). + +:::note +Bot operators who prefer not to create a free Cloudflare account can do so using our [old form](https://docs.google.com/forms/d/e/1FAIpQLSdqYNuULEypMnp4i5pROSc-uP6x65Xub9svD27mb8JChA_-XA/viewform?usp=sf_link), but the waiting time is up to several weeks for verified bot requests to be evaluated. +::: + +## Transient false negatives + +Once Cloudflare lists a bot as a verified bot, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the bot for a defined period of time. + +It takes 24 hours for an inactive IP to be removed as a verified bot. + +A bot can remain unlisted until Cloudflare sees traffic being sourced from the bot. When the bot is revalidated, it is listed as a verified bot again. diff --git a/src/content/docs/bots/concepts/bot/verified-bots/policy.mdx b/src/content/docs/bots/concepts/bot/verified-bots/policy.mdx index 8cabffae6c9bd2e..8024e2bd1411dbf 100644 --- a/src/content/docs/bots/concepts/bot/verified-bots/policy.mdx +++ b/src/content/docs/bots/concepts/bot/verified-bots/policy.mdx @@ -2,7 +2,7 @@ pcx_content_type: reference title: Verified bots policy sidebar: - order: 2 + order: 5 label: Policy --- @@ -27,7 +27,7 @@ A bot crawling one site is not valid. ### Bot Identification -The user-agent with the following requirements: +The user-agent or message signature with the following requirements: - Have at least 5 characters. - Must not contain special characters. @@ -72,22 +72,6 @@ If a search engine crawler skips `robots.txt`, it will be rejected. The bot must have publicly documented expected behavior or user-agent format. -## IP Validation - -A set of validation methods and requirements to gather set IP ranges for a verified service. - -### Public IP List - -- A fixed and limited set of IP addresses, which can be verified via publicly accessible plain-text, `JSON`, or `CSV`. -- IP addresses used solely by the bot owner. -- A user-agent match pattern. - -### Reverse DNS - -- A list of domain suffixes to validate DNS records. -- IP addresses should have PTR records set correctly. -- A user-agent match pattern. - ## Breach of Policy If any of the requirements to validate are breached, a service will be removed from the global allowlist. @@ -100,39 +84,3 @@ If any of the requirements to validate are breached, a service will be removed f - A block of IPs not briefed on onboarding is added to the list. - The disclosed purpose of the service does not reflect on the traffic. - An AI Crawler that does not respect the crawl-delay directive in robots.txt. - -## Online application - -To submit a verified bot that Cloudflare is not [currently tracking](https://radar.cloudflare.com/verified-bots), fill out an [online application](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots) in the Cloudflare dashboard for the fastest possible results. Bot operators who prefer not to create a free Cloudflare account can do so using our [old form](https://docs.google.com/forms/d/e/1FAIpQLSdqYNuULEypMnp4i5pROSc-uP6x65Xub9svD27mb8JChA_-XA/viewform?usp=sf_link), but the waiting time is up to several weeks for verified bot requests to be evaluated. - -### Generic user-agents - -User-agent patterns that match generic user-agents will be rejected by the Verified Bots API. When you add a user-agent pattern that is considered very common to the Verified Bot form, you may encounter an error message that will prompt you to correct the user-agent before you can submit again. - -Generic user-agents include: - -- `Dart` -- `Go-http-client` -- `GuzzleHttp` -- `Google Chrome` -- `Mozilla Firefox` -- `Safari` -- `Nessus` -- `Websocket++` -- `cloudflare-go` -- `fasthttp` -- `got` -- `nginx-ssl early hints` -- `node` -- `node-fetch` -- `okhttp` -- `python-requests` -- `uTorrent` - -## Transient false negatives - -Once Cloudflare lists a bot as a verified bot, this entry is cached and may get delisted if no traffic is seen in the Cloudflare network coming from the bot for a defined period of time. - -It takes 24 hours for an inactive IP to be removed as a verified bot. - -A bot can remain unlisted until Cloudflare sees traffic being sourced from the bot. When the bot is revalidated, it is listed as a verified bot again. diff --git a/src/content/docs/bots/concepts/bot/verified-bots/web-bot-auth.mdx b/src/content/docs/bots/concepts/bot/verified-bots/web-bot-auth.mdx new file mode 100644 index 000000000000000..81eef1eb36e7c03 --- /dev/null +++ b/src/content/docs/bots/concepts/bot/verified-bots/web-bot-auth.mdx @@ -0,0 +1,198 @@ +--- +pcx_content_type: concept +title: Web Bot Auth +sidebar: + order: 6 + label: Web Bot Auth + +--- + +import { GlossaryTooltip, Steps } from "~/components" + +Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. + +It relies on two active IETF drafts: a [directory draft](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory) allowing the crawler to share their public keys, and a [protocol draft](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture) defining how these keys should be used to attach crawler's identity to HTTP requests. + +This documentation goes over specific integration within Cloudflare. + +## 1. Generate a valid signing key + +You need to generate a signing key which will be used to authenticate your bot's requests. + +{/* prettier-ignore */} + +1. Generate a unique [Ed25519](https://ed25519.cr.yp.to/) private key to sign your requests. This example uses the [OpenSSL](https://openssl-library.org/) `genpkey` command: + + ```sh + openssl genpkey -algorithm ed25519 -out private-key.pem + ``` +2. Extract your public key. + + ```sh + openssl pkey -in private-key.pem -pubout -out public-key.pem + ``` +3. Convert the public key to JSON Web Key (JWK) using a tool of your choice. This example uses [`jwker`](https://github.com/jphastings/jwker) command line application. + ```sh + go install github.com/jphastings/jwker/cmd/jwker@latest + jwker public-key.pem public-key.jwk + ``` + + +By following these steps, you have generated a private key and a public key, then converted the public key to a JWK. + +:::note +You can also [generate a JavaScript key using WebCrypto API](https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey), which will produce a key in the correct JWK format. + +Many existing [JWK libraries](https://jwt.io/libraries) support WebCrypto API for generating JavaScript key. +::: + +## 2. Host a key directory + +You need to host a key directory which creates a way for your bot to authenticate its requests to Cloudflare. +This directory should follow the definition from the active IETF draft [draft-meunier-http-message-signatures-directory-01](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory-01). + + +1. Host a key directory at `/.well-known/http-message-signatures-directory` (note that this is a requirement). This key directory should serve a JSON Web Key Set (JWKS) including the public key derived from your signing key. +2. Serve the web page over HTTPS (not HTTP). +3. [Calculate the base64 URL-encoded JWK thumbprint](https://www.rfc-editor.org/rfc/rfc8037.html#appendix-A.3) associated with your Ed25519 public key. +4. Sign your HTTP response using the HTTP message signature specification by attaching one signature per key in your key directory. This ensures no one else can mirror your directory and attempt to register on your behalf. Your response must include the following headers: + - `Content-Type`: This header must have the value `application/http-message-signatures-directory+json`. + - `Signature`: Construct a [`Signature` header](https://www.rfc-editor.org/rfc/rfc9421#name-the-signature-http-field) over your chosen components. + - `Signature-Input`: Construct a [`Signature-Input` header](https://www.rfc-editor.org/rfc/rfc9421#name-the-signature-input-http-fi) over your chosen components. The header must meet the following requirements. + | Required component parameter | Requirement | + | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------- | + | `tag` | This should be equal to `http-message-signatures-directory`. | + | `keyid` | JWK thumbprint of the corresponding key in your directory. | + | `created` | This should be equal to a `Unix` timestamp associated with when the message was sent by your application. | + | `expires` | This should be equal to a `Unix` timestamp associated with when Cloudflare should no longer attempt to verify the message. | + + The following example shows the annotated request and response with required headers against `https://example.com`. + ```txt + GET /.well-known/http-message-signatures-directory HTTP/1.1 + Host: example.com + Accept: application/http-message-signatures-directory+json + + HTTP/1.1 200 OK + Content-Type: application/http-message-signatures-directory+json + Signature: sig1=:TD5arhV1ved6xtx63cUIFCMONT248cpDeVUAljLgkdozbjMNpJGr/WAx4PzHj+WeG0xMHQF1BOdFLDsfjdjvBA==: + Signature-Input: sig1=("@authority");alg="ed25519";keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";nonce="ZO3/XMEZjrvSnLtAP9M7jK0WGQf3J+pbmQRUpKDhF9/jsNCWqUh2sq+TH4WTX3/GpNoSZUa8eNWMKqxWp2/c2g==";tag="http-message-signatures-directory";created=1750105829;expires=1750105839 + Cache-Control: max-age=86400 + { + "keys": [{ + "kty": "OKP", + "crv": "Ed25519", + "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs", // Base64 URL-encoded public key, with no padding + }] + } + ``` + + +:::note +This URL serves a standard JSON Web Key Set. Besides `x`, `crv`, and `kty`, you can include other standard JSON Web Key parameters, and you may publish non-Ed25519 keys as well. Multiple Ed25519 keys are supported. Only those for which you provide a signature in the above format are going to be used. + +Cloudflare will ignore all other key types and key parameters except those containing `kty`, `crv`, and `x` formatted above. Do not include information that would leak your private key, such as the `d` parameter. +::: + +You can use the Cloudflare-developed [`http-signature-directory` CLI tool](https://crates.io/crates/http-signature-directory) to assist you in validating your directory. + +## 3. Register your bot and key directory + +You need to register your bot and its key directory to add your bot to the list of verified bots. + + +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/), and select your account and domain. +2. Go to **Manage Account** > **Configurations**. +3. Go to the **Verified Bots** tab. +4. For **Verification Method**: select **Request Signature**. +5. For **Validation Instructions**: enter the URL of your key directory. You can additionally supply User Agents values (and their match patterns) that will be sent by your bot. +6. Select **Submit**. + + +Cloudflare accepts all valid Ed25519 keys found in your key directory. In the event a key already exists in Cloudflare's registered database, Cloudflare will work with you to supply a new key, or rotate your existing key. + +:::note[Estimated review time] +The estimated review time is approximately one week. + +After successful verification, you will be able to send verified requests. +::: + +## 4. (After verification) Sign your requests + +After your bot has been successfully verified, your bot is ready to sign its requests. The signature protocol is defined in [draft-meunier-web-bot-auth-architecture-02](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture-02) + + +### 4.1. Choose a set of components to sign + +Choose a set of components to sign. + +A component is either an HTTP header, or any [derived components](https://www.rfc-editor.org/rfc/rfc9421#name-derived-components) in the HTTP Message Signatures specification. Cloudflare recommends the following: + - Choose at least the `@authority` derived component, which represents the domain you are sending requests to. For example, a request to `https://example.com` will be interpreted to have an `@authority` of `example.com`. + - Use components that only contain ASCII values. HTTP Message Signature specification disallows non-ASCII characters, which will result in failure to validate your bot's requests. + + :::note[Use components with only ASCII values] + Cloudflare currently does not support `bs` or `sf` parameter designed to serialize non-ASCII values into ASCII equivalents. + ::: + +:::caution[`Content-Digest` header] +If you wish to sign your [message content](https://www.rfc-editor.org/rfc/rfc9421#name-message-content) using a `Content-Digest` header, note that you should only do so if there is zero risk of a message being altered on the way to Cloudflare. + +For example, if the message is unencrypted and proxied to Cloudflare, you should not use `Content-Digest`. +::: + +### 4.2. Calculate the JWK thumbprint + +[Calculate the base64 URL-encoded JWK thumbprint](https://www.rfc-editor.org/rfc/rfc8037.html#appendix-A.3) from the public key you registered with Cloudflare. + +### 4.3. Construct the required headers + +Construct the three required headers for Web Bot Auth. + +#### `Signature-Input` header + +Construct a [`Signature-Input` header](https://www.rfc-editor.org/rfc/rfc9421#name-the-signature-input-http-fi) over your chosen components. The header must meet the following requirements. + +| Required component parameter | Requirement | +| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `tag` | This should be equal to `web-bot-auth`. | +| `keyid` | This should be equal to the thumbprint computed in step 2. | +| `created` | This should be equal to a `Unix` timestamp associated with when the message was sent by your application. | +| `expires` | This should be equal to a `Unix` timestamp associated with when Cloudflare should no longer attempt to verify the message. A short `expires` reduces the likelihood of replay attacks, and Cloudflare recommends choosing suitable short-lived intervals. | + +#### `Signature` header + +Construct a [`Signature` header](https://www.rfc-editor.org/rfc/rfc9421#name-the-signature-http-field) over your chosen components. + +#### `Signature-Agent` header + +Construct a [`Signature-Agent` header](https://www.ietf.org/archive/id/draft-meunier-http-message-signatures-directory-01.html#name-header-field-definition) that points to your key directory. Note that Cloudflare will fail to verify a message if: + - The message includes a `Signature-Agent` header that is not an `https://`. + - The message includes a valid URI but does not enclose it in double quotes. This is due to Signature-Agent being a structured field. + - The message has a valid `Signature-Agent` header, but does not include it in the component list in `Signature-Input`. + +### 4.4. Add the headers to your bot's requests + +Attach these three headers to your bot's requests. + +An example request may look like this: + +```txt +Signature-Agent: "https://signature-agent.test" +Signature-Input: sig2=("@authority" "signature-agent") + ;created=1735689600 + ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U" + ;alg="ed25519" + ;expires=1735693200 + ;nonce="e8N7S2MFd/qrd6T2R3tdfAuuANngKI7LFtKYI/vowzk4lAZYadIX6wW25MwG7DCT9RUKAJ0qVkU0mEeLElW1qg==" + ;tag="web-bot-auth" +Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==: +``` + +## Additional resources + +You may wish to refer to the following resources. + +- [Bots FAQs](/bots/frequently-asked-questions/). +- Cloudflare blog: [Message Signatures are now part of our Verified Bots Program](https://blog.cloudflare.com/verified-bots-with-cryptography). +- Cloudflare blog: [Forget IPs: using cryptography to verify bot and agent traffic](https://blog.cloudflare.com/web-bot-auth/). +- Cloudflare's [`web-bot-auth` library in Rust](https://crates.io/crates/web-bot-auth). +- Cloudflare's [`web-bot-auth` npm package in Typescript](https://www.npmjs.com/package/web-bot-auth). \ No newline at end of file diff --git a/src/content/docs/bots/troubleshooting/frequently-asked-questions.mdx b/src/content/docs/bots/frequently-asked-questions.mdx similarity index 71% rename from src/content/docs/bots/troubleshooting/frequently-asked-questions.mdx rename to src/content/docs/bots/frequently-asked-questions.mdx index 3ac181ae5ce05fe..bd20747a311386a 100644 --- a/src/content/docs/bots/troubleshooting/frequently-asked-questions.mdx +++ b/src/content/docs/bots/frequently-asked-questions.mdx @@ -3,32 +3,32 @@ pcx_content_type: faq title: FAQ structured_data: true sidebar: - order: 3 + order: 11 --- import { Render, RuleID } from "~/components"; ## Bots -## How does Cloudflare detect bots? +### How does Cloudflare detect bots? Cloudflare uses multiple methods to detect bots, but these vary by plan. For more details, refer to [Plans](/bots/plans). --- -## How do I know what is included in my plan? +### How do I know what is included in my plan? To know what's included in your plan, refer to our [Plans](/bots/plans). --- -## How do I set up my bot product? +### How do I set up my bot product? To learn how to set up your bot product, refer to [Get started](/bots/get-started). --- -## Yandex bot unexpectedly blocked by the WAF managed rule with ID `...f6cbb163` +### Yandex bot unexpectedly blocked by the WAF managed rule with ID `...f6cbb163` Yandex updates their bots very frequently, you may see more false positives while these changes are propagated. New and recently updated bots will occasionally be blocked by a Cloudflare WAF managed rule, as the IP list of Yandex bots has not yet synced with Yandex's most recent changes. @@ -45,7 +45,7 @@ Once the new Yandex IP is propagated to our system, the requests will not be blo --- -## How does machine learning work? +### How does machine learning work? Supervised machine learning takes certain variables (X) like gender and age and predicts another variable (Y) like income. @@ -55,7 +55,7 @@ Cloudflare uses data from millions of requests and re-train the system on a peri --- -## Why am I seeing a Managed Challenge action for WAF rules? +### Why am I seeing a Managed Challenge action for WAF rules? When you choose to challenge different bot categories with Bot Fight Mode or Super Bot Fight Mode, you will see Security Events with an **Action Taken** of **Managed Challenge**. @@ -65,13 +65,13 @@ This does not mean that your traffic was blocked. It is the challenge sent to yo To understand if the result of the challenge was a success or a failure, you can verify using [Logpush](/logs/about/). -## Does the WAF run before Super Bot Fight Mode? +### Does the WAF run before Super Bot Fight Mode? Yes. WAF rules are executed before Super Bot Fight Mode. If a WAF custom rule performs a [terminating action](/ruleset-engine/rules-language/actions/) such as _Block_, your Super Bot Fight Mode configuration will not be evaluated. --- -## What is cf.bot_management.verified_bot? +### What is cf.bot_management.verified_bot? A request's _cf.bot_management.verified_bot_ value is a boolean indicating whether such request comes from a Cloudflare allowed bot. @@ -83,13 +83,13 @@ To allow traffic from good bots, use the [Verified Bot](/ruleset-engine/rules-la --- -## Why might the ja3hash or JA4 be empty in HTTP logs? +### Why might the ja3hash or JA4 be empty in HTTP logs? --- -## I run a good bot and want for it to be added to the allowlist (cf.bot_management.verified_bot). What should I do? +### I run a good bot and want for it to be added to the allowlist (cf.bot_management.verified_bot). What should I do? Cloudflare maintains a sample list of verified bots in [Cloudflare Radar](https://radar.cloudflare.com/verified-bots). @@ -97,7 +97,7 @@ As a bot operator, in order to be listed by Cloudflare as a Verified Bot, your b --- -## What information do I need to troubleshoot my bot issues? +### What information do I need to troubleshoot my bot issues? If you are experiencing errors with your bot solution and need to submit a Support request, include the following information: @@ -124,7 +124,7 @@ Please follow instructions in the following questions on how to disable BFM and --- -## What should I do if I am getting False positives caused by Bot Fight Mode (BFM) or Super Bot Fight Mode (SBFM)? +### What should I do if I am getting False positives caused by Bot Fight Mode (BFM) or Super Bot Fight Mode (SBFM)? :::caution[Important considerations you need to be aware of before turning on BFM or SBFM] @@ -150,7 +150,7 @@ Bot Fight Mode can still trigger if you have IP Access rules, but it cannot trig --- -## Super Bot Fight Mode feature (SBFM) is still blocking requests even though the feature is turned off, why? +### Super Bot Fight Mode feature (SBFM) is still blocking requests even though the feature is turned off, why? This is a known issue the Bots team is working to resolve in the near future. In the meantime, there is a workaround to resolve such issue. You will need to run the following API command to check and remove the SBFM ruleset: @@ -171,3 +171,59 @@ This is a known issue the Bots team is working to resolve in the near future. In ``` Note that you need to replace `` with your own [API token](/fundamentals/api/get-started/create-token/). + +--- + +## Web Bot Auth + +### What key algorithms does Cloudflare support? + +Cloudflare supports Ed25519 key algorithm. + +--- + +### What `web-bot-auth` features from the IETF draft are not supported? + +The following derived components are not supported, and we will fail to verify a message if they are included: + +- `@query-params`: Cloudflare recommends signing the whole query using the `@query` component instead of signing an individual parameter. +- `@status`: This is not possible to include in the request path. + +The following component parameters defined in IETF RFC 9421 are not supported, and Cloudflare will fail to verify a message if they are included: + +- `sf` (for HTTP header fields) +- `bs` (for HTTP header fields) +- `key` (for HTTP header fields) +- `req` (for HTTP header fields or derived components) +- `name` (for `@query-param` support - this requires `@query-param` support) + +--- + +### Should I supply a `nonce` parameter in `Signature-Input`? + +The `nonce` parameter allows you to supply a `nonce` to prevent attackers from replaying past messages against a server. + +While Cloudflare recommends including it, there is currently no `nonce` validation, nor does Cloudflare guard against replay attacks using a database of seen `nonces`. + +Instead, Cloudflare recommends short `expires` as a protection against replay attacks. A minute is often sufficient. + +--- + +### How do I know my JSON Web Key set directory will be accepted? + +Cloudflare uses [`http-signature-directory` tool](https://crates.io/crates/http-signature-directory) to validate your directory. Please ensure this works against your directory before registering with us. + +--- + +### My message is failing validation. What could be the cause? + +- Ensure you have a [`Signature-Agent` header](/bots/concepts/bot/verified-bots/web-bot-auth/#signature-agent-header), and that its value is in double-quotes. +- Ensure you include `signature-agent` in the component list in your [`Signature-Input` header](/bots/concepts/bot/verified-bots/web-bot-auth/#signature-agent-header). +- Ensure your `expires` timestamp is not too short, such that, by the time it arrives at Cloudflare servers, it has already expired. A minute is often sufficient. +- Ensure you are not signing components containing non-ASCII values, or on the unsupported list. + +--- + +### I want to use HTTP message signatures / Web Bot Auth on my zone, and do not want Cloudflare's verification to intervene. What do I do? + +You can request the Web Bot Auth feature be disabled for your zone by contacting Cloudflare support. This will disable usage of Web Bot Auth specifically with Cloudflare, and verified bots will fallback to other modes to validate traffic. \ No newline at end of file diff --git a/src/content/release-notes/bots.yaml b/src/content/release-notes/bots.yaml index a5044b334912b97..8ea9947f42f371b 100644 --- a/src/content/release-notes/bots.yaml +++ b/src/content/release-notes/bots.yaml @@ -8,13 +8,16 @@ entries: - publish_date: "2025-07-02" title: Managed robots.txt will prepend existing files description: Cloudflare will prepend our managed `robots.txt` before your existing `robots.txt`, combining both into a single response. + - publish_date: "2025-06-26" + title: Web Bot Auth is now available for bot verification + description: Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. This provides a more robust way of verifying bots. - publish_date: "2025-05-14" title: Anomaly detection events now receive a bot score of 2 description: Events detected by the [anomaly detection engine](/bots/concepts/bot-detection-engines/#anomaly-detection-enterprise) are now given a bot score of 2. - publish_date: "2025-05-08" title: Machine Learning model v9 is now the default model description: |- - [Machine Learning model v9](/bots/reference/machine-learning-models/#model-versions-and-release-notes) is now the default model for all new zones and existing zones set to use the latest machine learning model. + [Machine Learning model v9](/bots/reference/machine-learning-models/#model-versions-and-release-notes) is now the default model for all new zones and existing zones set to use the latest machine learning model. - publish_date: "2025-04-28" title: Managed robots.txt is now available description: |-