Skip to content

Commit c9d57c3

Browse files
mitchellkrogzamitchellkrogza
committed
V3.2019.06.1119 [ci skip]
1 parent 035a71c commit c9d57c3

File tree

39 files changed

+125
-67
lines changed

39 files changed

+125
-67
lines changed

‎.dev-tools/_htaccess_generator_files/bad-user-agents.list‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ crawler\.feedback
8585
crawl\.sogou\.com
8686
CrazyWebCrawler
8787
Crescent
88+
CrunchBot
8889
CSHttp
8990
Curious
9091
Custo
@@ -132,6 +133,7 @@ ExtractorPro
132133
Extreme\ Picture\ Finder
133134
EyeNetIE
134135
Ezooms
136+
facebookscraper
135137
FDM
136138
FemtosearchBot
137139
FHscan
@@ -316,6 +318,7 @@ Page\ Analyzer
316318
PageGrabber
317319
page\ scorer
318320
PageScorer
321+
Pandalytics
319322
Panscient
320323
Papa\ Foto
321324
Pavuk
@@ -401,6 +404,7 @@ Snake
401404
Snapbot
402405
Snoopy
403406
SocialRankIOBot
407+
Sociscraper
404408
sogouspider
405409
Sogou\ web\ spider
406410
Sosospider
@@ -456,6 +460,7 @@ TwengaBot
456460
Twice
457461
Typhoeus
458462
UnisterBot
463+
Upflow
459464
URLy\ Warning
460465
URLy\.Warning
461466
Vacuum

‎.dev-tools/_robots_generator_files/robots.list‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ crawler.feedback
8585
crawl.sogou.com
8686
CrazyWebCrawler
8787
Crescent
88+
CrunchBot
8889
CSHttp
8990
Curious
9091
Custo
@@ -132,6 +133,7 @@ ExtractorPro
132133
Extreme Picture Finder
133134
EyeNetIE
134135
Ezooms
136+
facebookscraper
135137
FDM
136138
FemtosearchBot
137139
FHscan
@@ -316,6 +318,7 @@ Page Analyzer
316318
PageGrabber
317319
page scorer
318320
PageScorer
321+
Pandalytics
319322
Panscient
320323
Papa Foto
321324
Pavuk
@@ -401,6 +404,7 @@ Snake
401404
Snapbot
402405
Snoopy
403406
SocialRankIOBot
407+
Sociscraper
404408
sogouspider
405409
Sogou web spider
406410
Sosospider
@@ -456,6 +460,7 @@ TwengaBot
456460
Twice
457461
Typhoeus
458462
UnisterBot
463+
Upflow
459464
URLy Warning
460465
URLy.Warning
461466
Vacuum

‎.dev-tools/_test_results/_conf_files_2.2/globalblacklist.conf‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,10 @@
1616

1717
### Version Information #
1818
###################################################
19-
### Version: V3.2019.06.1118
20-
### Updated: Wed Jun 19 09:52:23 SAST 2019
19+
### Version: V3.2019.06.1119
20+
### Updated: Wed Jun 19 10:02:25 SAST 2019
2121
### Bad Referrer Count: 6693
22-
### Bad Bot Count: 544
22+
### Bad Bot Count: 550
2323
###################################################
2424
### Version Information ##
2525

@@ -257,6 +257,7 @@ BrowserMatchNoCase "^(.*?)(\bcrawler.feedback\b)(.*)$" bad_bot
257257
BrowserMatchNoCase "^(.*?)(\bcrawl.sogou.com\b)(.*)$" bad_bot
258258
BrowserMatchNoCase "^(.*?)(\bCrazyWebCrawler\b)(.*)$" bad_bot
259259
BrowserMatchNoCase "^(.*?)(\bCrescent\b)(.*)$" bad_bot
260+
BrowserMatchNoCase "^(.*?)(\bCrunchBot\b)(.*)$" bad_bot
260261
BrowserMatchNoCase "^(.*?)(\bCSHttp\b)(.*)$" bad_bot
261262
BrowserMatchNoCase "^(.*?)(\bCurious\b)(.*)$" bad_bot
262263
BrowserMatchNoCase "^(.*?)(\bCusto\b)(.*)$" bad_bot
@@ -304,6 +305,7 @@ BrowserMatchNoCase "^(.*?)(\bExtractorPro\b)(.*)$" bad_bot
304305
BrowserMatchNoCase "^(.*?)(\bExtreme\ Picture\ Finder\b)(.*)$" bad_bot
305306
BrowserMatchNoCase "^(.*?)(\bEyeNetIE\b)(.*)$" bad_bot
306307
BrowserMatchNoCase "^(.*?)(\bEzooms\b)(.*)$" bad_bot
308+
BrowserMatchNoCase "^(.*?)(\bfacebookscraper\b)(.*)$" bad_bot
307309
BrowserMatchNoCase "^(.*?)(\bFDM\b)(.*)$" bad_bot
308310
BrowserMatchNoCase "^(.*?)(\bFemtosearchBot\b)(.*)$" bad_bot
309311
BrowserMatchNoCase "^(.*?)(\bFHscan\b)(.*)$" bad_bot
@@ -488,6 +490,7 @@ BrowserMatchNoCase "^(.*?)(\bPage\ Analyzer\b)(.*)$" bad_bot
488490
BrowserMatchNoCase "^(.*?)(\bPageGrabber\b)(.*)$" bad_bot
489491
BrowserMatchNoCase "^(.*?)(\bpage\ scorer\b)(.*)$" bad_bot
490492
BrowserMatchNoCase "^(.*?)(\bPageScorer\b)(.*)$" bad_bot
493+
BrowserMatchNoCase "^(.*?)(\bPandalytics\b)(.*)$" bad_bot
491494
BrowserMatchNoCase "^(.*?)(\bPanscient\b)(.*)$" bad_bot
492495
BrowserMatchNoCase "^(.*?)(\bPapa\ Foto\b)(.*)$" bad_bot
493496
BrowserMatchNoCase "^(.*?)(\bPavuk\b)(.*)$" bad_bot
@@ -573,6 +576,7 @@ BrowserMatchNoCase "^(.*?)(\bSnake\b)(.*)$" bad_bot
573576
BrowserMatchNoCase "^(.*?)(\bSnapbot\b)(.*)$" bad_bot
574577
BrowserMatchNoCase "^(.*?)(\bSnoopy\b)(.*)$" bad_bot
575578
BrowserMatchNoCase "^(.*?)(\bSocialRankIOBot\b)(.*)$" bad_bot
579+
BrowserMatchNoCase "^(.*?)(\bSociscraper\b)(.*)$" bad_bot
576580
BrowserMatchNoCase "^(.*?)(\bsogouspider\b)(.*)$" bad_bot
577581
BrowserMatchNoCase "^(.*?)(\bSogou\ web\ spider\b)(.*)$" bad_bot
578582
BrowserMatchNoCase "^(.*?)(\bSosospider\b)(.*)$" bad_bot
@@ -628,6 +632,7 @@ BrowserMatchNoCase "^(.*?)(\bTwengaBot\b)(.*)$" bad_bot
628632
BrowserMatchNoCase "^(.*?)(\bTwice\b)(.*)$" bad_bot
629633
BrowserMatchNoCase "^(.*?)(\bTyphoeus\b)(.*)$" bad_bot
630634
BrowserMatchNoCase "^(.*?)(\bUnisterBot\b)(.*)$" bad_bot
635+
BrowserMatchNoCase "^(.*?)(\bUpflow\b)(.*)$" bad_bot
631636
BrowserMatchNoCase "^(.*?)(\bURLy.Warning\b)(.*)$" bad_bot
632637
BrowserMatchNoCase "^(.*?)(\bURLy\ Warning\b)(.*)$" bad_bot
633638
BrowserMatchNoCase "^(.*?)(\bVacuum\b)(.*)$" bad_bot

‎.dev-tools/_test_results/_conf_files_2.4/globalblacklist.conf‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,10 @@
1616

1717
### Version Information #
1818
###################################################
19-
### Version: V3.2019.06.1118
20-
### Updated: Wed Jun 19 09:52:23 SAST 2019
19+
### Version: V3.2019.06.1119
20+
### Updated: Wed Jun 19 10:02:25 SAST 2019
2121
### Bad Referrer Count: 6693
22-
### Bad Bot Count: 544
22+
### Bad Bot Count: 550
2323
###################################################
2424
### Version Information ##
2525

@@ -250,6 +250,7 @@ BrowserMatchNoCase "^(.*?)(\bcrawler.feedback\b)(.*)$" bad_bot
250250
BrowserMatchNoCase "^(.*?)(\bcrawl.sogou.com\b)(.*)$" bad_bot
251251
BrowserMatchNoCase "^(.*?)(\bCrazyWebCrawler\b)(.*)$" bad_bot
252252
BrowserMatchNoCase "^(.*?)(\bCrescent\b)(.*)$" bad_bot
253+
BrowserMatchNoCase "^(.*?)(\bCrunchBot\b)(.*)$" bad_bot
253254
BrowserMatchNoCase "^(.*?)(\bCSHttp\b)(.*)$" bad_bot
254255
BrowserMatchNoCase "^(.*?)(\bCurious\b)(.*)$" bad_bot
255256
BrowserMatchNoCase "^(.*?)(\bCusto\b)(.*)$" bad_bot
@@ -297,6 +298,7 @@ BrowserMatchNoCase "^(.*?)(\bExtractorPro\b)(.*)$" bad_bot
297298
BrowserMatchNoCase "^(.*?)(\bExtreme\ Picture\ Finder\b)(.*)$" bad_bot
298299
BrowserMatchNoCase "^(.*?)(\bEyeNetIE\b)(.*)$" bad_bot
299300
BrowserMatchNoCase "^(.*?)(\bEzooms\b)(.*)$" bad_bot
301+
BrowserMatchNoCase "^(.*?)(\bfacebookscraper\b)(.*)$" bad_bot
300302
BrowserMatchNoCase "^(.*?)(\bFDM\b)(.*)$" bad_bot
301303
BrowserMatchNoCase "^(.*?)(\bFemtosearchBot\b)(.*)$" bad_bot
302304
BrowserMatchNoCase "^(.*?)(\bFHscan\b)(.*)$" bad_bot
@@ -481,6 +483,7 @@ BrowserMatchNoCase "^(.*?)(\bPage\ Analyzer\b)(.*)$" bad_bot
481483
BrowserMatchNoCase "^(.*?)(\bPageGrabber\b)(.*)$" bad_bot
482484
BrowserMatchNoCase "^(.*?)(\bpage\ scorer\b)(.*)$" bad_bot
483485
BrowserMatchNoCase "^(.*?)(\bPageScorer\b)(.*)$" bad_bot
486+
BrowserMatchNoCase "^(.*?)(\bPandalytics\b)(.*)$" bad_bot
484487
BrowserMatchNoCase "^(.*?)(\bPanscient\b)(.*)$" bad_bot
485488
BrowserMatchNoCase "^(.*?)(\bPapa\ Foto\b)(.*)$" bad_bot
486489
BrowserMatchNoCase "^(.*?)(\bPavuk\b)(.*)$" bad_bot
@@ -566,6 +569,7 @@ BrowserMatchNoCase "^(.*?)(\bSnake\b)(.*)$" bad_bot
566569
BrowserMatchNoCase "^(.*?)(\bSnapbot\b)(.*)$" bad_bot
567570
BrowserMatchNoCase "^(.*?)(\bSnoopy\b)(.*)$" bad_bot
568571
BrowserMatchNoCase "^(.*?)(\bSocialRankIOBot\b)(.*)$" bad_bot
572+
BrowserMatchNoCase "^(.*?)(\bSociscraper\b)(.*)$" bad_bot
569573
BrowserMatchNoCase "^(.*?)(\bsogouspider\b)(.*)$" bad_bot
570574
BrowserMatchNoCase "^(.*?)(\bSogou\ web\ spider\b)(.*)$" bad_bot
571575
BrowserMatchNoCase "^(.*?)(\bSosospider\b)(.*)$" bad_bot
@@ -621,6 +625,7 @@ BrowserMatchNoCase "^(.*?)(\bTwengaBot\b)(.*)$" bad_bot
621625
BrowserMatchNoCase "^(.*?)(\bTwice\b)(.*)$" bad_bot
622626
BrowserMatchNoCase "^(.*?)(\bTyphoeus\b)(.*)$" bad_bot
623627
BrowserMatchNoCase "^(.*?)(\bUnisterBot\b)(.*)$" bad_bot
628+
BrowserMatchNoCase "^(.*?)(\bUpflow\b)(.*)$" bad_bot
624629
BrowserMatchNoCase "^(.*?)(\bURLy.Warning\b)(.*)$" bad_bot
625630
BrowserMatchNoCase "^(.*?)(\bURLy\ Warning\b)(.*)$" bad_bot
626631
BrowserMatchNoCase "^(.*?)(\bVacuum\b)(.*)$" bad_bot

‎.dev-tools/_test_results/_curl_tests_2.2/curltest1.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Last Tested: Wed Jun 19 09:55:06 SAST 2019
1+
Last Tested: Wed Jun 19 10:05:08 SAST 2019
22

33
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
44
<html><head>

‎.dev-tools/_test_results/_curl_tests_2.2/curltest10.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Last Tested: Wed Jun 19 09:55:06 SAST 2019
1+
Last Tested: Wed Jun 19 10:05:08 SAST 2019
22

33
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
44
<html><head>

‎.dev-tools/_test_results/_curl_tests_2.2/curltest11.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Last Tested: Wed Jun 19 09:55:06 SAST 2019
1+
Last Tested: Wed Jun 19 10:05:08 SAST 2019
22

33
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
44
<html><head>

‎.dev-tools/_test_results/_curl_tests_2.2/curltest12.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Last Tested: Wed Jun 19 09:55:06 SAST 2019
1+
Last Tested: Wed Jun 19 10:05:08 SAST 2019
22

33
<!DOCTYPE html>
44
<html>

‎.dev-tools/_test_results/_curl_tests_2.2/curltest2.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Last Tested: Wed Jun 19 09:55:06 SAST 2019
1+
Last Tested: Wed Jun 19 10:05:08 SAST 2019
22

33
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
44
<html><head>

‎.dev-tools/_test_results/_curl_tests_2.2/curltest3.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
Last Tested: Wed Jun 19 09:55:06 SAST 2019
1+
Last Tested: Wed Jun 19 10:05:08 SAST 2019
22

33
HTTP/1.1 403 Forbidden
4-
Date: Wed, 19 Jun 2019 07:55:07 GMT
4+
Date: Wed, 19 Jun 2019 08:05:09 GMT
55
Server: Apache/2.2.25 (Unix) DAV/2
66
Content-Type: text/html; charset=iso-8859-1
77

0 commit comments

Comments
 (0)