Skip to content

Commit 56c5de6

Browse files
V3.2020.03.1187 [ci skip]
1 parent a20e058 commit 56c5de6

File tree

28 files changed

+500
-80
lines changed

28 files changed

+500
-80
lines changed

.dev-tools/_htaccess_generator_files/bad-user-agents.list

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,12 @@ Alligator
1717
AllSubmitter
1818
AlphaBot
1919
Anarchie
20+
Ankit
2021
Apexoo
2122
archive\.org\_bot
2223
arquivo\.pt
2324
arquivo\-web\-crawler
25+
Aspeigel
2426
ASPSeek
2527
Asterias
2628
Attach
@@ -220,6 +222,7 @@ JustView
220222
Jyxobot
221223
Kenjin\ Spider
222224
Keyword\ Density
225+
Kinza
223226
Kozmosbot
224227
Lanshanbot
225228
Larbin
@@ -229,6 +232,7 @@ LexiBot
229232
Lftp
230233
LibWeb
231234
Libwhisker
235+
LieBaoFast
232236
Lightspeedsystems
233237
Likse
234238
Linkdexbot
@@ -262,6 +266,7 @@ Masscan
262266
Mass\ Downloader
263267
Mata\ Hari
264268
MauiBot
269+
Mb2345Browser
265270
meanpathbot
266271
Meanpathbot
267272
MeanPath\ Bot
@@ -270,6 +275,7 @@ mediawords
270275
MegaIndex\.ru
271276
Metauri
272277
MFC\_Tear\_Sample
278+
MicroMessenger
273279
Microsoft\ Data\ Access
274280
Microsoft\ URL\ Control
275281
MIDown\ tool
@@ -279,6 +285,7 @@ MJ12bot
279285
Mojeek
280286
Mojolicious
281287
Morfeus\ Fucking\ Scanner
288+
MQQBrowser
282289
Mr\.4x3
283290
MSFrontPage
284291
MSIECrawler
@@ -321,6 +328,7 @@ Openfind
321328
OpenLinkProfiler
322329
Openvas
323330
OpenVAS
331+
OPPO\ A33
324332
OrangeBot
325333
OrangeSpider
326334
OutclicksBot
@@ -348,6 +356,7 @@ PleaseCrawl
348356
plumanalytics
349357
Pockey
350358
POE\-Component\-Client\-HTTP
359+
polaris\ version
351360
Probethenet
352361
ProPowerBot
353362
ProWebWalker
@@ -556,6 +565,8 @@ zauba\.io
556565
Zermelo
557566
Zeus
558567
zgrab
568+
zh\_CN
569+
zh\-CN
559570
Zitebot
560571
ZmEu
561572
ZumBot

.dev-tools/_robots_generator_files/robots.list

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,12 @@ Alligator
1717
AllSubmitter
1818
AlphaBot
1919
Anarchie
20+
Ankit
2021
Apexoo
2122
archive.org_bot
2223
arquivo.pt
2324
arquivo-web-crawler
25+
Aspeigel
2426
ASPSeek
2527
Asterias
2628
Attach
@@ -220,6 +222,7 @@ JustView
220222
Jyxobot
221223
Kenjin Spider
222224
Keyword Density
225+
Kinza
223226
Kozmosbot
224227
Lanshanbot
225228
Larbin
@@ -229,6 +232,7 @@ LexiBot
229232
Lftp
230233
LibWeb
231234
Libwhisker
235+
LieBaoFast
232236
Lightspeedsystems
233237
Likse
234238
Linkdexbot
@@ -262,6 +266,7 @@ Masscan
262266
Mass Downloader
263267
Mata Hari
264268
MauiBot
269+
Mb2345Browser
265270
meanpathbot
266271
Meanpathbot
267272
MeanPath Bot
@@ -270,6 +275,7 @@ mediawords
270275
MegaIndex.ru
271276
Metauri
272277
MFC_Tear_Sample
278+
MicroMessenger
273279
Microsoft Data Access
274280
Microsoft URL Control
275281
MIDown tool
@@ -279,6 +285,7 @@ MJ12bot
279285
Mojeek
280286
Mojolicious
281287
Morfeus Fucking Scanner
288+
MQQBrowser
282289
Mr.4x3
283290
MSFrontPage
284291
MSIECrawler
@@ -321,6 +328,7 @@ Openfind
321328
OpenLinkProfiler
322329
Openvas
323330
OpenVAS
331+
OPPO A33
324332
OrangeBot
325333
OrangeSpider
326334
OutclicksBot
@@ -348,6 +356,7 @@ PleaseCrawl
348356
plumanalytics
349357
Pockey
350358
POE-Component-Client-HTTP
359+
polaris version
351360
Probethenet
352361
ProPowerBot
353362
ProWebWalker
@@ -556,6 +565,8 @@ zauba.io
556565
Zermelo
557566
Zeus
558567
zgrab
568+
zh_CN
569+
zh-CN
559570
Zitebot
560571
ZmEu
561572
ZumBot

0 commit comments

Comments
 (0)