Skip to content

Conversation

@kathayl
Copy link
Contributor

@kathayl kathayl commented Apr 9, 2025

update to include BR respects robots.txt

Summary

Screenshots (optional)

Documentation checklist

  • The documentation style guide has been adhered to.
  • If a larger change - such as adding a new page- an issue has been opened in relation to any incorrect or out of date information that this PR fixes.
  • Files which have changed name or location have been allocated redirects.

update to include BR respects robots.txt
@hyperlint-ai
Copy link
Contributor

hyperlint-ai bot commented Apr 9, 2025

Howdy and thanks for contributing to our repo. The Cloudflare team reviews new, external PRs within two (2) weeks. If it's been two weeks or longer without any movement, please tag the PR Assignees in a comment.

We review internal PRs within 1 week. If it's something urgent or has been sitting without a comment, start a thread in the Developer Docs space internally.


PR Change Summary

Updated FAQ to clarify that Browser Rendering respects robots.txt and does not bypass Cloudflare's Bot Protection.

  • Clarified that Browser Rendering requests are identified as bots and do not bypass Bot Protection.
  • Added information that Browser Rendering respects the robots.txt protocol.
  • Updated guidance for scanning own zones with Bot Protection.

Modified Files

  • src/content/docs/browser-rendering/faq.mdx

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 9, 2025

@daisyfaithauma daisyfaithauma merged commit 2a12287 into production Apr 10, 2025
12 checks passed
@daisyfaithauma daisyfaithauma deleted the kathayl-patch-9 branch April 10, 2025 11:12
RebeccaTamachiro pushed a commit that referenced this pull request Apr 21, 2025
update to include BR respects robots.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants