Skip to content

Commit 50af477

Browse files
authored
[Support] Update references to Security app (#26021)
1 parent 2aaa134 commit 50af477

File tree

2 files changed

+19
-21
lines changed

2 files changed

+19
-21
lines changed

src/content/docs/support/troubleshooting/general-troubleshooting/troubleshooting-crawl-errors.mdx

Lines changed: 18 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,10 @@
11
---
22
pcx_content_type: troubleshooting
33
source: https://support.cloudflare.com/hc/en-us/articles/200169806-Troubleshooting-crawl-errors
4-
title: Troubleshooting crawl errors
5-
4+
title: Troubleshoot crawl errors
65
---
76

8-
## Overview
9-
10-
Cloudflare allows search engine crawlers and bots. If you observe crawl issues or Cloudflare challenges presented to the search engine crawler or bot, [contact Cloudflare support](/support/contacting-cloudflare-support/) with the information you gather when troubleshooting the crawl errors via the methods outlined in this guide.
7+
Cloudflare allows search engine crawlers and bots. If you observe crawl issues or Cloudflare challenges presented to the search engine crawler or bot, [contact Cloudflare support](/support/contacting-cloudflare-support/) with the information you gather when troubleshooting the crawl errors via the methods outlined in this guide.
118

129
***
1310

@@ -38,25 +35,26 @@ Review the following recommendations to prevent crawler errors:
3835
* [Monitor.Us](http://www.monitor.us/)
3936
* [Updown](https://updown.io/)
4037

41-
* Do not block Google crawler IP addresses via **custom rules** or **IP Access rules** within the **Security** app. If you are using [rate limiting rules](/waf/rate-limiting-rules/), make sure they do not apply to the Google crawler.
38+
* Do not block Google crawler IP addresses via [custom rules](/waf/custom-rules/) or [IP Access rules](/waf/tools/ip-access-rules/). If you are using [rate limiting rules](/waf/rate-limiting-rules/), make sure they do not apply to the Google crawler.
39+
40+
Confirm an IP address belongs to Google by consulting Google’s documentation on [verifying googlebot IP addresses](https://support.google.com/webmasters/bin/answer.py?answer=80553).
4241

43-
Confirm an IP address belongs to Google by consulting Google’s documentation on [verifying googlebot IP addresses](https://support.google.com/webmasters/bin/answer.py?answer=80553).
42+
* Do not block the United States via [custom rules](/waf/custom-rules/) or [IP Access rules](/waf/tools/ip-access-rules/).
4443

45-
* Do not block the United States via **custom rules** or **IP Access rules** within the **Security** app.
46-
* Do not block or User-Agents in your .htaccess, server configuration, [robots.txt](http://support.google.com/webmasters/bin/answer.py?answer=35303), or web application.
44+
* Do not block Google User-Agents in your `.htaccess` file, server configuration, [`robots.txt`](http://support.google.com/webmasters/bin/answer.py?answer=35303), or web application.
4745

48-
Google uses a [variety of User-Agents](https://support.google.com/webmasters/answer/1061943) to crawl your website. You can [test your robots.txt via Google](https://support.google.com/webmasters/answer/6062598?hl=en).
46+
Google uses a [variety of User-Agents](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) to crawl your website. You can [test your `robots.txt` via Google](https://support.google.com/webmasters/answer/6062598?hl=en).
4947

50-
* Do not allow crawling of files in the /cdn-cgi/ directory. This path is used internally by Cloudflare and Google encounters errors when crawling it. Disallow crawls of cdn-cgi via robots.txt:
48+
* Do not allow crawling of files in the `/cdn-cgi/` directory. This path is used internally by Cloudflare and Google encounters errors when crawling it. Disallow crawls of `cdn-cgi` via `robots.txt`:
5149

5250
`Disallow: /cdn-cgi/`
5351

5452
:::note
5553

56-
Errors for cdn-cgi do not impact site rankings.
54+
Errors for `cdn-cgi` do not impact site rankings.
5755
:::
5856

59-
* Ensure your [robots.txt file allows the AdSense crawler](http://support.google.com/webmasters/bin/answer.py?hl=en\&answer=1061943).
57+
* Ensure your [`robots.txt` file allows the AdSense crawler](http://support.google.com/webmasters/bin/answer.py?hl=en\&answer=1061943).
6058
* [Restore original visitor IP addresses](/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/) in your server logs.
6159

6260
***
@@ -67,15 +65,15 @@ Troubleshooting steps for the most commonly reported crawl errors are mentioned
6765

6866
### HTTP 4XX Errors
6967

70-
[HTTP 4XX errors](/support/troubleshooting/http-status-codes/4xx-client-error/) are the most common type of crawl error. Cloudflare delivers these errors from your web server to Google. These errors are caused for various reasons such as a missing page on your web server or a malformed link in your HTML. The solution depends upon the problem encountered.
68+
[HTTP 4XX errors](/support/troubleshooting/http-status-codes/4xx-client-error/) are the most common type of crawl error. Cloudflare delivers these errors from your web server to Google. These errors are caused for various reasons such as a missing page on your web server or a malformed link in your HTML. The solution depends upon the problem encountered.
7169

7270
### HTTP 5XX Errors
7371

7472
[HTTP 5XX errors](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/) indicate that either Cloudflare or your origin web server experienced an internal error. To correlate occurrences of crawl errors with site outages, monitor your origin web server's health. Monitoring your website health both through Cloudflare and directly to your origin web server IPs determines whether errors occurred due to Cloudflare or your origin web server.
7573

7674
### DNS Errors
7775

78-
Troubleshooting steps vary depending on whether your domain is on Cloudflare via a Full or CNAME setup. To verify which setup your domain uses, open a terminal and execute the following command (replace `www.example.com` with your Cloudflare domain):
76+
Troubleshooting steps vary depending on whether your domain is on Cloudflare via a Full or CNAME setup. To verify which setup your domain uses, open a terminal and execute the following command (replace `www.example.com` with your Cloudflare domain):
7977

8078
`dig +short SOA` `www.example.com`
8179

@@ -91,20 +89,20 @@ Once you’ve confirmed how your domain was setup with Cloudflare, proceed with
9189

9290
**CNAME**
9391

94-
Contact your hosting provider to investigate DNS errors and provide the date Google encountered DNS errors. Additionally, review the [Cloudflare System Status](http://www.cloudflare.com/system-status) page for any network outages on the date the errors were encountered by Google.
92+
Contact your hosting provider to investigate DNS errors and provide the date Google encountered DNS errors. Additionally, review the [Cloudflare System Status](http://www.cloudflare.com/system-status) page for any network outages on the date the errors were encountered by Google.
9593

9694
**Full**
9795

98-
[Contact Cloudflare support](/support/contacting-cloudflare-support/) and provide the date and time that Google observed the errors.
96+
[Contact Cloudflare support](/support/contacting-cloudflare-support/) and provide the date and time that Google observed the errors.
9997

10098
### Requesting troubleshooting assistance
10199

102-
If the above troubleshooting steps do not resolve your crawl errors, follow the steps below to export crawler errors as a .csv file from your Google Webmaster Tools Dashboard. Include this .csv file when [contacting Cloudflare Support](/support/contacting-cloudflare-support/).
100+
If the above troubleshooting steps do not resolve your crawl errors, follow the steps below to export crawler errors as a `.csv` file from your Google Webmaster Tools Dashboard. Include this `.csv` file when [contacting Cloudflare Support](/support/contacting-cloudflare-support/).
103101

104102
1. Log in to your Google Webmaster Tools account and navigate to the **Health** section of the affected domain.
105103
2. Click **Crawl Errors** in the left hand navigation.
106-
3. Click **Download** to export the list of errors as a .csv file.
107-
4. Provide the downloaded .csv file to Cloudflare support.
104+
3. Click **Download** to export the list of errors as a `.csv` file.
105+
4. Provide the downloaded `.csv` file to Cloudflare support.
108106

109107
***
110108

src/content/docs/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1010.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ A website owner blocked your request based on your client's web browser.
1414

1515
### Resolution
1616

17-
Notify the website owner of the blocking. If you cannot determine how to contact the website owner, lookup contact information for the domain via the [Whois database](https://lookup.icann.org/). Site owners disable **Browser** **Integrity Check** via the **Settings** tab of the **Security** app.
17+
Notify the website owner of the blocking. If you cannot determine how to contact the website owner, lookup contact information for the domain via the [Whois database](https://lookup.icann.org/). Site owners can [turn off Browser Integrity Check](/waf/tools/browser-integrity-check/#disable-browser-integrity-check) in the Security **Settings** page.
1818

1919
:::note
2020
Since the website owner performed the blocking, Cloudflare support cannot override a customer's security settings.

0 commit comments

Comments
 (0)