File tree Expand file tree Collapse file tree 1 file changed +4
-4
lines changed
specs/crawler/common/schemas Expand file tree Collapse file tree 1 file changed +4
-4
lines changed Original file line number Diff line number Diff line change @@ -78,11 +78,11 @@ Configuration:
7878 type : boolean
7979 description : |
8080 Whether to ignore the `noindex` robots meta tag.
81- If `true` pages with this meta tag _will_ be crawled.
81+ If `true`, pages with this meta tag _will_ be crawled.
8282 ignorePaginationAttributes :
8383 type : boolean
8484 description : |
85- Whether the crawler should follow `rel="prev"` and `rel="next"` pagination links in your `<head>` HTML.
85+ Whether the crawler should follow `rel="prev"` and `rel="next"` pagination links in the `<head>` section of an HTML page .
8686
8787 - If `true`, the crawler ignores the pagination links.
8888 - If `false`, the crawler follows the pagination links.
@@ -92,10 +92,10 @@ Configuration:
9292 description : |
9393 Query parameters to ignore while crawling.
9494
95- All URLs with the matching query parameters will be treated as identical.
95+ All URLs with the matching query parameters are treated as identical.
9696 This prevents indexing URLs that just differ by their query parameters.
9797
98- You can use wildcard characters to pattern match .
98+ Use wildcards to match multiple query parameters .
9999 maxItems : 9999
100100 example :
101101 - ref
You can’t perform that action at this time.
0 commit comments