diff --git a/_generator_lists/bad-user-agents.list b/_generator_lists/bad-user-agents.list index af876deb409..e2ed41fdaaf 100755 --- a/_generator_lists/bad-user-agents.list +++ b/_generator_lists/bad-user-agents.list @@ -3,22 +3,22 @@ 404checker 404enemy 80legs -ADmantX -AIBOT -ALittle\ Client -ASPSeek Abonti Aboundex Aboundexbot Acunetix +ADmantX +adscanner AdsTxtCrawlerTP AfD-Verbotsverfahren AhrefsBot +AIBOT AiHitBot Aipbot Alexibot -AllSubmitter +ALittle\ Client Alligator +AllSubmitter AlphaBot Anarchie Anarchy @@ -26,32 +26,38 @@ Anarchy99 Ankit Anthill Apexoo +archive.org_bot +arquivo.pt +arquivo-web-crawler Aspiegel +ASPSeek Asterias Atomseobot Attach +autoemailspider AwarioRssBot AwarioSmartBot -BBBike -BDCbot -BDFetch -BLEXBot BackDoorBot -BackStreet -BackWeb Backlink-Ceck +backlink-check BacklinkCrawler +BackStreet +BackWeb Badass Bandit Barkrowler BatchFTP Battleztar\ Bazinga +BBBike +BDCbot +BDFetch BetaBot Bigfoot Bitacle -BlackWidow Black\ Hole Blackboard +BlackWidow +BLEXBot Blow BlowFish Boardreader @@ -67,49 +73,59 @@ Bullseye BunnySlippers BuzzSumo Bytespider -CATExplorador -CCBot -CODE87 -CSHttp +cah.io.community Calculon +CATExplorador CazoodleBot +CCBot Cegbfeieh CensysInspect -CheTeam +check1.exe CheeseBot CherryPicker +CheTeam ChinaClaw Chlooe Citoid Claritybot +clark-crawler Cliqzbot Cloud\ mapping +coccocbot Cocolyzebot +CODE87 Cogentbot +cognitiveseo Collector +com.plumanalytics Copier CopyRightCheck Copyscape Cosmos Craftbot +crawl.sogou.com +crawler.feedback +crawler4j Crawling\ at\ Home\ Project CrazyWebCrawler Crescent CrunchBot +CSHttp Curious Custo CyotekWebCopy -DBLBot -DIIbot -DSearch -DTS\ Agent -DataCha0s DatabaseDriverMysqli +DataCha0s +dataforseo.com +dataforseobot +DBLBot +demandbase-bot Demon Deusu Devil Digincore DigitalPebble +DIIbot Dirbuster Disco Discobot @@ -118,24 +134,29 @@ Dispatch DittoSpyder DnBCrawler-Analytics DnyzBot -DomCopBot DomainAppender DomainCrawler +Domains\ Project DomainSigmaCrawler +domainsproject.org DomainStatsBot -Domains\ Project +DomCopBot Dotbot Download\ Wonder Dragonfly Drip -ECCP/1.0 -EMail\ Siphon -EMail\ Wolf +DSearch +DTS\ Agent EasyDL Ebingbong +eCatch +ECCP/1.0 Ecxi EirGrabber +EMail\ Siphon +EMail\ Wolf EroCrawler +evc-batch Evil Exabot Express\ WebPictures @@ -145,9 +166,10 @@ ExtractorPro Extreme\ Picture\ Finder EyeNetIE Ezooms +facebookscraper FDM -FHscan FemtosearchBot +FHscan Fimap Firefox/7.0 FlashGet @@ -158,44 +180,45 @@ FrontPage Fuzz FyberSpider Fyrebot -G-i-g-a-b-o-t -GPTBot -GT::WWW GalaxyBot Genieo GermCrawler +Getintent GetRight GetWeb -Getintent Gigabot +G-i-g-a-b-o-t Go!Zilla Go-Ahead-Got-It -GoZilla +gopher Gotit -GrabNet +GoZilla +GPTBot Grabber +GrabNet Grafula GrapeFX GrapeshotCrawler GridBot -HEADMasterSEO -HMView -HTMLparser -HTTP::Lite -HTTrack +GT::WWW Haansoft HaosouSpider Harvest Havij +HEADMasterSEO +heritrix Heritrix Hloader +HMView HonoluluBot +HTMLparser +HTTP::Lite +HTTrack Humanlinks HybridBot -IDBTE4M -IDBot -IRLbot Iblog +IDBot +IDBTE4M Id-search IlseBot Image\ Fetch @@ -204,20 +227,26 @@ IndeedBot Indy\ Library InfoNaviRobot InfoTekies +instabid Intelliseek InterGET -InternetSeer Internet\ Ninja +InternetSeer +internetVista\ monitor +ips-agent Iria +IRLbot +isitwp.com Iskanie IstellaBot -JOC\ Web\ Spider +iubenda-radar JamesBOT Jbrofuzz JennyBot JetCar Jetty JikeSpider +JOC\ Web\ Spider Joomla Jorgee JustView @@ -227,8 +256,6 @@ Keybot\ Translation-Search-Machine Keyword\ Density Kinza Kozmosbot -LNSpiderguy -LWP::Simple Lanshanbot Larbin Leap @@ -241,12 +268,13 @@ Libwhisker LieBaoFast Lightspeedsystems Likse -LinkScan -LinkWalker Linkbot +linkdexbot LinkextractorPro LinkpadBot +LinkScan LinksManager +LinkWalker LinqiaMetadataDownloaderBot LinqiaRSSBot LinqiaScrapeBot @@ -254,21 +282,18 @@ Lipperhey Lipperhey\ Spider Litemage_walker Lmspider +LNSpiderguy Ltx71 -MFC_Tear_Sample -MIDown\ tool -MIIxpc -MJ12bot -MQQBrowser -MSFrontPage -MSIECrawler -MTRobot -Mag-Net +LWP::Simple +lwp-request +lwp-trivial Magnet +Mag-Net +magpie-crawler Mail.RU_Bot -Majestic-SEO -Majestic12 Majestic\ SEO +Majestic12 +Majestic-SEO MarkMonitor MarkWatch Mass\ Downloader @@ -277,89 +302,110 @@ Mata\ Hari MauiBot Mb2345Browser MeanPath\ Bot +meanpathbot Meanpathbot Mediatoolkitbot +mediawords MegaIndex.ru Metauri +MFC_Tear_Sample MicroMessenger Microsoft\ Data\ Access Microsoft\ URL\ Control +MIDown\ tool +MIIxpc Minefield Mister\ PiX +MJ12bot Moblie Safari Mojeek Mojolicious MolokaiBot Morfeus\ Fucking\ Scanner Mozlila +MQQBrowser Mr.4x3 +MSFrontPage +MSIECrawler Msrabot +MTRobot +muhstik-scan Musobot -NICErsPRO -NPbot +my-tiny-bot Name\ Intelligence Nameprotect Navroad NearSite Needle Nessus +Net\ Vampire NetAnts +Netcraft +netEstate\ NE\ Crawler NetLyzer NetMechanic NetSpider -NetZIP -Net\ Vampire -Netcraft Nettrack Netvibes +NetZIP NextGenSearchBot Nibbler +NICErsPRO Niki-bot Nikto NimbleCrawler Nimbostratus Ninja Nmap +NPbot Nuclei Nutch +oBot Octopus Offline\ Explorer Offline\ Navigator OnCrawl -OpenLinkProfiler -OpenVAS +openai +openai.com Openfind +OpenLinkProfiler Openvas +OpenVAS OrangeBot OrangeSpider OutclicksBot OutfoxBot -PECL::HTTP -PHPCrawl -POE-Component-Client-HTTP +Page\ Analyzer +page\ scorer PageAnalyzer PageGrabber PageScorer PageThing.com -Page\ Analyzer Pandalytics Panscient Papa\ Foto Pavuk +pcBrowser +PECL::HTTP PeoplePal Petalbot -Pi-Monster +PHPCrawl Picscout Picsearch PictureFinder Piepmatz Pimonster +Pi-Monster Pixray PleaseCrawl +plumanalytics Pockey +POE-Component-Client-HTTP +polaris\ version +probe-image-size +Probethenet ProPowerBot ProWebWalker -Probethenet Proximic Psbot Pu_iN @@ -368,7 +414,6 @@ PxBroker PyCurl QueryN\ Metasearch Quick-Crawler -RSSingBot Rainbot RankActive RankActiveLinkBot @@ -377,26 +422,25 @@ RankingBot RankingBot2 Rankivabot RankurBot -Re-re -ReGet RealDownload Reaper RebelMouse Recorder RedesScrapy +ReGet RepoMonkey +Re-re Ripper +ripz RocketCrawler Rogerbot -SBIder -SEOkicks -SEOkicks-Robot -SEOlyticsCrawler -SEOprofiler -SEOstats -SISTRIX -SMTBot +RSSingBot +s1z.ru SalesIntelligent +satoristudio.net +SBIder +scalaj-http +scan.lol ScanAlert Scanbot ScoutJet @@ -413,30 +457,47 @@ Semrush SemrushBot SentiBot SenutoBot -SeoSiteCheckup +seobility SeobilityBot +seocompany.store +SEOkicks +SEOkicks-Robot +SEOlyticsCrawler Seomoz +SEOprofiler +seoscanners +SeoSiteCheckup +seostar +SEOstats +serpstatbot +sexsearcher Shodan Siphon +SISTRIX +Site\ Sucker +Sitebeam +sitechecker.pro SiteCheckerBotCrawler SiteExplorer +Siteimprove SiteLockSpider +siteripz SiteSnagger SiteSucker -Site\ Sucker -Sitebeam -Siteimprove Sitevigil SlySearch SmartDownload +SMTBot Snake Snapbot Snoopy SocialRankIOBot Sociscraper Sogou\ web\ spider +sogouspider Sosospider Sottopop +sp_auditbot SpaceBison Spammen SpankBot @@ -444,6 +505,7 @@ Spanner Spbot Spinn3r SputnikBot +spyfu Sqlmap Sqlworm Sqworm @@ -457,16 +519,18 @@ Surfbot SurveyBot Suzuran Swiftbot +sysscan Szukacz T0PHackTeam T8Abot +tAkeOut Teleport TeleportPro Telesoft Telesphoreo Telesphorep -TheNomad The\ Intraformant +TheNomad Thumbor TightTwatBot TinyTestBot @@ -475,6 +539,8 @@ Toata Toweyabot Tracemyfile Trendiction +trendiction.com +trendiction.de Trendictionbot True_Robot Turingos @@ -483,15 +549,16 @@ TurnitinBot TwengaBot Twice Typhoeus -URLy.Warning -URLy\ Warning +ubermetrics-technologies.com UnisterBot Upflow -V-BOT -VB\ Project -VCI +URLy.Warning +URLy\ Warning Vacuum Vagabondo +VB\ Project +V-BOT +VCI VelenPublicWebCrawler VeriCiteCrawler VidibleScraper @@ -499,51 +566,50 @@ Virusdie VoidEYE Voil Voltron -WASALive-Bot -WBSearchBot -WEBDAV -WISENutbot -WPScan -WWW-Collector-E -WWW-Mechanize -WWW::Mechanize -WWWOFFLE +voyagerx.com Wallpapers Wallpapers/3.0 WallpapersHD -WeSEE +WASALive-Bot +WBSearchBot +Web\ Auto +Web\ Collage +Web\ Enhancer +Web\ Fetch +Web\ Fuck +Web\ Pix +Web\ Sauger +Web\ Sucker +Webalta WebAuto WebBandit WebCollage WebCopier +WEBDAV WebEnhancer WebFetch WebFuck +webgains-bot WebGo\ IS WebImageCollector WebLeacher +WebmasterWorldForumBot +webmeup-crawler WebPix +webpros.com +webprosbot WebReaper WebSauger -WebStripper -WebSucker -WebWhacker -WebZIP -Web\ Auto -Web\ Collage -Web\ Enhancer -Web\ Fetch -Web\ Fuck -Web\ Pix -Web\ Sauger -Web\ Sucker -Webalta -WebmasterWorldForumBot Webshag +Website\ Quester WebsiteExtractor WebsiteQuester -Website\ Quester Webster +WebStripper +WebSucker +WebWhacker +WebZIP +WeSEE Whack Whacker Whatweb @@ -551,97 +617,32 @@ Who.is\ Bot Widow WinHTTrack WiseGuys\ Robot +WISENutbot Wonderbot Woobot Wotbox Wprecon +WPScan +WWW::Mechanize +WWW-Collector-E +WWW-Mechanize +WWWOFFLE +x09Mozilla +x22Mozilla Xaldon\ WebSpider Xaldon_WebSpider Xenu +xpymep1.exe YoudaoBot Zade Zauba +zauba.io Zermelo Zeus +zgrab Zitebot ZmEu ZoomBot ZoominfoBot ZumBot ZyBorg -adscanner -archive.org_bot -arquivo-web-crawler -arquivo.pt -autoemailspider -backlink-check -cah.io.community -check1.exe -clark-crawler -coccocbot -cognitiveseo -com.plumanalytics -crawl.sogou.com -crawler.feedback -crawler4j -dataforseo.com -dataforseobot -demandbase-bot -domainsproject.org -eCatch -evc-batch -facebookscraper -gopher -heritrix -instabid -internetVista\ monitor -ips-agent -isitwp.com -iubenda-radar -linkdexbot -lwp-request -lwp-trivial -magpie-crawler -meanpathbot -mediawords -muhstik-scan -netEstate\ NE\ Crawler -oBot -openai -openai.com -page\ scorer -pcBrowser -plumanalytics -polaris\ version -probe-image-size -ripz -s1z.ru -satoristudio.net -scalaj-http -scan.lol -seobility -seocompany.store -seoscanners -seostar -serpstatbot -sexsearcher -sitechecker.pro -siteripz -sogouspider -sp_auditbot -spyfu -sysscan -tAkeOut -trendiction.com -trendiction.de -ubermetrics-technologies.com -voyagerx.com -webgains-bot -webmeup-crawler -webpros.com -webprosbot -x09Mozilla -x22Mozilla -xpymep1.exe -zauba.io -zgrab