Robots.txt From AI-Structured Web Administration.

[OsbornVentures]

Existing documentation URL(s)

https://medium.com/@bitsnbytesinc/robots-txt-isnt-for-you-it-s-for-them-06c33bd43d4b
https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
https://blog.cloudflare.com/introducing-pay-per-crawl/

What changes are you suggesting?

Your Aug 7 update (“instruct AI bots via robots.txt”) only influences the big, well-behaved crawlers.
Which doesn’t matter now: Perplexity and Anthropic often come in via third-party/masked crawlers after the recent pay-per-crawl shift.

Cat-and-mouse on user agents is unwinnable without degrading service. Saying “doesn’t affect SEO” is true—but the game moved to AEO/GEO Mid May 2025, Advising customers to rely on robots.txt encourages them to miss that shift. UA lists are dead.

Ask:

Encourage JSON endpoints (mirrors) for (swarm) crawlers instead of UA whack-a-mole.

Recommend lighter sites and paying for edge caching.

That’s it, practical, future-proof guidance over validation theater.

Additional information

Why the pivot from Pay Per Crawl , to Stop Crawl a month later?

[kodster28]

Howdy @OsbornVentures, going to close this out b/c it's not exactly relevant to the work we're doing here as a documentation team.

Not an expert, but you might have missed a related announcement for Web Bot Auth, which seems like it might help with your "UA whack-a-mole" comment.

I'll forward along to our Product team to see if it helps them with their future roadmap.