Commit 2c0f0ee
Tom
Match UAs more generously, as per google spec
So previously the UA "googlebot" would not match "googlebot-news" as the RFC says "The robot must obey the first record in /robots.txt that contains a User-Agent line whose value contains the name token of the robot as a substring."
"googlebot-news" is not a substring of "googlebot" but the reverse is true, so we try both cases now.1 parent aeef897 commit 2c0f0ee
2 files changed
+15
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
96 | 109 | | |
0 commit comments