Misc docs improvements (#23354)

Author: Oxyjun

src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/set-up-cloudflare-account.mdx

@@ -19,3 +19,10 @@ To begin using pay per crawl, set up your Cloudflare account.
 
 Sign up for a [Cloudflare account](https://dash.cloudflare.com/sign-up).
 
+:::note[Pay per crawl closed beta]
+Pay per crawl is currently in closed beta.
+
+To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer.
+
+To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
+:::

src/content/docs/ai-audit/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-up-cloudflare-account.mdx

@@ -31,3 +31,11 @@ AI Audit scopes pay per crawl to your zones.
 ## 3. Proxy traffic through Cloudflare
 
 Make sure your domain is [proxying traffic through Cloudflare](/fundamentals/concepts/how-cloudflare-works/#cloudflare-as-a-reverse-proxy).
+
+:::note[Pay per crawl closed beta]
+Pay per crawl is currently in closed beta.
+
+To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer.
+
+To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
+:::

src/content/docs/ai-audit/features/pay-per-crawl/what-is-pay-per-crawl.mdx

@@ -7,10 +7,12 @@ sidebar:
 
 import { Steps, GlossaryTooltip } from "~/components";
 
-:::note[Closed beta]
+:::note[Pay per crawl closed beta]
 Pay per crawl is currently in closed beta.
 
-To find out how to join the beta program and to learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
+To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer.
+
+To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
 :::
 
 AI crawlers often consume vast amounts of web content. Some provide mutual benefit to content owners by indexing content for search engines, but others engage in activities such as content scraping without permission.

src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

@@ -14,6 +14,10 @@ Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scr
 
 AI bots are expected to follow the `robots.txt` directives.
 
+:::note
+Respecting `robots.txt` is voluntary. If you want to prevent crawling, use AI Audit's [manage AI crawlers](/ai-audit/features/manage-ai-crawlers/) feature.
+:::
+
 ## Compatibility with existing `robots.txt` files
 
 Cloudflare will independently check whether your website has an existing `robots.txt` file and update the behavior of this feature based on your website.