Skip to content

[Bots] Managed robots.txt #22026

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Apr 29, 2025
Merged

[Bots] Managed robots.txt #22026

merged 5 commits into from
Apr 29, 2025

Conversation

@patriciasantaana
Copy link
Contributor

@patriciasantaana patriciasantaana commented Apr 28, 2025

Summary

Add documentation on Managed robots.txt

Documentation checklist

@github-actions github-actions bot added product:bots Related to Bots product size/s labels Apr 28, 2025
@hyperlint-ai
Copy link
Contributor

hyperlint-ai bot commented Apr 28, 2025

Howdy and thanks for contributing to our repo. The Cloudflare team reviews new, external PRs within two (2) weeks. If it's been two weeks or longer without any movement, please tag the PR Assignees in a comment.

We review internal PRs within 1 week. If it's something urgent or has been sitting without a comment, start a thread in the Developer Docs space internally.

PR Change Summary

Added documentation for managing robots.txt to guide users in directing AI crawlers on their websites.

  • Introduced a new documentation page for Managed robots.txt
  • Provided instructions for implementing robots.txt based on different plans
  • Updated the sidebar order for static resource protection documentation

Modified Files

  • src/content/docs/bots/additional-configurations/static-resources.mdx

Added Files

  • src/content/docs/bots/additional-configurations/managed-robots-txt.mdx
  • src/content/partials/bots/enable-managed-robots-txt.mdx
How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 28, 2025 •
edited
Loading

Preview URL: https://17e6b30b.preview.developers.cloudflare.com
Preview Branch URL: https://patricia-pcx14359-robots-txt.preview.developers.cloudflare.com

Files with changes (up to 15)

Original Link Updated Link
https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ https://patricia-pcx14359-robots-txt.preview.developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
https://developers.cloudflare.com/security/settings/ https://patricia-pcx14359-robots-txt.preview.developers.cloudflare.com/security/settings/
https://developers.cloudflare.com/bots/additional-configurations/static-resources/ https://patricia-pcx14359-robots-txt.preview.developers.cloudflare.com/bots/additional-configurations/static-resources/

@patriciasantaana patriciasantaana marked this pull request as ready for review April 28, 2025 18:22
@patriciasantaana patriciasantaana requested a review from a team as a code owner April 28, 2025 18:22
marciocloudflare
marciocloudflare approved these changes Apr 29, 2025
View reviewed changes
patriciasantaana
patriciasantaana commented Apr 29, 2025
View reviewed changes
patriciasantaana
patriciasantaana commented Apr 29, 2025
View reviewed changes
patriciasantaana
patriciasantaana commented Apr 29, 2025
View reviewed changes
@patriciasantaana patriciasantaana merged commit 08266e6 into production Apr 29, 2025
13 checks passed
@patriciasantaana patriciasantaana deleted the patricia/pcx14359-robots-txt branch April 29, 2025 15:53
GregBrimble pushed a commit that referenced this pull request May 1, 2025
@patriciasantaana @marciocloudflare @GregBrimble
[Bots] Managed robots.txt (#22026)
030798a 
* managed robots.txt

* security settings

* release notes entry

* Update src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Co-authored-by: marciocloudflare <[email protected]>

* Apply suggestions from code review

---------

Co-authored-by: marciocloudflare <[email protected]>
daisyfaithauma pushed a commit that referenced this pull request May 7, 2025
@patriciasantaana @marciocloudflare @daisyfaithauma
[Bots] Managed robots.txt (#22026)
eaf35e8 
* managed robots.txt

* security settings

* release notes entry

* Update src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Co-authored-by: marciocloudflare <[email protected]>

* Apply suggestions from code review

---------

Co-authored-by: marciocloudflare <[email protected]>
daisyfaithauma pushed a commit that referenced this pull request May 8, 2025
@patriciasantaana @marciocloudflare @daisyfaithauma
[Bots] Managed robots.txt (#22026)
db549b8 
* managed robots.txt

* security settings

* release notes entry

* Update src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Co-authored-by: marciocloudflare <[email protected]>

* Apply suggestions from code review

---------

Co-authored-by: marciocloudflare <[email protected]>
Maddy-Cloudflare pushed a commit that referenced this pull request May 9, 2025
@patriciasantaana @marciocloudflare @Maddy-Cloudflare
[Bots] Managed robots.txt (#22026)
2250c9c 
* managed robots.txt

* security settings

* release notes entry

* Update src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Co-authored-by: marciocloudflare <[email protected]>

* Apply suggestions from code review

---------

Co-authored-by: marciocloudflare <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@marciocloudflare marciocloudflare marciocloudflare approved these changes

Assignees

@pedrosousa pedrosousa

@dcpena dcpena

@haleycode haleycode

Labels

product:bots Related to Bots product product:security size/s

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@patriciasantaana @marciocloudflare @pedrosousa @dcpena @haleycode