New minor release 0.11.7

Author: mrnfrancesco

CHANGELOG

@@ -4,9 +4,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.11.7] - 2022-09-18
 ### Added
-- Test cases for plugins creation and running
+- DuckDuckGo plugin
+
+### Changed
+- `--debugging` CLI parameter is now `--debug`
+
+### Fixed
+- False negatives due to whitespaces in subdomain extracted text (e.g. "\n     some.subdomain.com  \n")
+- Subdomain selectors for Yahoo, Ask, Baidu and StartPage
+- WebCrawler query parameters
+- Baidu requests delay to avoid anti-bot checks
+
+### Removed
+- WebCrawler plugin because of `403: Forbidden` HTTP error
+- StartPage and Exaled plugin because of CAPTCHA
+- Teoma plugin since it now redirects to Ask
+
 
 ## [0.9.2] - 2019-08-01
 ### Added

Dockerfile

@@ -2,7 +2,7 @@ FROM python:3.7.4-alpine3.10
 
 RUN apk --no-cache add libxml2 libxml2-dev libxslt-dev gcc musl-dev wget ca-certificates
 
-ARG YASS_VERSION="0.9.2"
+ARG YASS_VERSION="0.11.7"
 
 WORKDIR /yass
 RUN wget "https://github.com/mrnfrancesco/yass/archive/v${YASS_VERSION}.zip" -O yass.zip && \

README

@@ -10,8 +10,8 @@ YASS - Yet Another Subdomainer Software
    :alt: Code Climate
 .. image:: https://img.shields.io/badge/python-3.7-blue.svg
     :target: https://www.python.org/downloads/release/python-374/
-.. image:: https://img.shields.io/badge/release-v0.9.2-green.svg
-    :target: https://github.com/mrnfrancesco/yass/releases/tag/v0.9.2
+.. image:: https://img.shields.io/badge/release-v0.11.7-green.svg
+    :target: https://github.com/mrnfrancesco/yass/releases/tag/v0.11.7
 
 YASS is a plugin-powered search engine based subdomainer.
 Its goal is to give you a tool to query whatever search engine you like and parse HTML response writing *less than 10 lines of code*.
@@ -29,7 +29,7 @@ USAGE
 
     Output arguments:
       -g, --grepable        output results in grepable format (default: False)
-      -d, --debugging       set output format to a more verbose one, for debugging
+      -d, --debug           set output format to a more verbose one, for debugging
                             purpose (default: False)
       -l {debug,info,warning,error,critical}, --level {debug,info,warning,error,critical}
                             set output verbosity (default: info)
@@ -48,7 +48,7 @@ Example output
 
         ________________________________________________
 
-        YASS - Yet Another Subdomainer Software (v0.9.2)
+        YASS - Yet Another Subdomainer Software (v0.11.7)
             by Francesco Marano (@mrnfrancesco)
         ________________________________________________
 
@@ -146,4 +146,4 @@ Or you can use Docker:
     $ docker run --rm -ti mrnfrancesco/yass:latest /bin/sh
 
     /yass # yass --version
-    YASS v0.9.2
+    YASS v0.11.7

README.md

@@ -3,7 +3,7 @@
 [![Build Status](https://travis-ci.org/mrnfrancesco/yass.svg?branch=master)](https://travis-ci.org/mrnfrancesco/yass)
 [![Code Climate](https://codeclimate.com/github/mrnfrancesco/yass/badges/gpa.svg)](https://codeclimate.com/github/mrnfrancesco/yass)
 [![Python 3.7](https://img.shields.io/badge/python-3.7-blue.svg)](https://www.python.org/downloads/release/python-374/)
-[![Release](https://img.shields.io/badge/release-v0.9.2-green.svg)](https://github.com/mrnfrancesco/yass/releases/tag/v0.9.2)
+[![Release](https://img.shields.io/badge/release-v0.11.7-green.svg)](https://github.com/mrnfrancesco/yass/releases/tag/v0.11.7)
 
 YASS is a plugin-powered search engine based subdomainer.
 Its goal is to give you a tool to query whatever search engine you like and parse HTML response writing *less than 10 lines of code*.
@@ -20,7 +20,7 @@ Its goal is to give you a tool to query whatever search engine you like and pars
 
     Output arguments:
       -g, --grepable        output results in grepable format (default: False)
-      -d, --debugging       set output format to a more verbose one, for debugging
+      -d, --debug           set output format to a more verbose one, for debugging
                             purpose (default: False)
       -l {debug,info,warning,error,critical}, --level {debug,info,warning,error,critical}
                             set output verbosity (default: info)
@@ -40,7 +40,7 @@ $ yass microsoft.com
 
     ________________________________________________
 
-    YASS - Yet Another Subdomainer Software (v0.9.2)
+    YASS - Yet Another Subdomainer Software (v0.11.7)
         by Francesco Marano (@mrnfrancesco)
     ________________________________________________
 
@@ -141,5 +141,5 @@ $ docker build -t mrnfrancesco/yass:latest .
 $ docker run --rm -ti mrnfrancesco/yass:latest /bin/sh
 
 /yass # yass --version
-YASS v0.9.2
+YASS v0.11.7
 ```

bin/yass

@@ -63,7 +63,7 @@ def parse(arguments=None):
     output = parser.add_argument_group('Output arguments')
     output.add_argument('-g', '--grepable', help="output results in grepable format (default: %(default)s)",
                         default=False, action="store_true")
-    output.add_argument('-d', '--debugging', default=False, action="store_true",
+    output.add_argument('-d', '--debug', default=False, action="store_true",
                         help="set output format to a more verbose one, for debugging purpose (default: %(default)s)")
     output.add_argument('-l', '--level', help="set output verbosity (default: %(default)s)",
                         choices=['debug', 'info', 'warning', 'error', 'critical'], default='info')
@@ -99,13 +99,13 @@ def banner():
 
 
 if __name__ == '__main__':
-    if len(sys.argv) is 1:
+    if len(sys.argv) == 1:
         sys.argv.append('--usage')
 
     options = parse(arguments=None)  # Use sys.argv[1:] as default arguments
     if options.grepable:
         options.level = None
-    logger_config(options.colored, options.debugging, options.level)
+    logger_config(options.colored, options.debug, options.level)
 
     if not options.grepable:
         print(banner())

yass/__init__.py

@@ -1,7 +1,7 @@
 __lname__ = "yass"
 __uname__ = "YASS"
 __acronym__ = "Yet Another Subdomainer Software"
-__version__ = "0.8.0"
+__version__ = "0.11.7"
 __author__ = "Francesco Marano (@mrnfrancesco)"
 __author_email__ = "francesco.mrn24@gmail.com"
 __source_url__ = "https://github.com/mrnfrancesco/yass"

yass/logger.py

@@ -69,7 +69,7 @@ def logger(depth=3):
     return caller.f_locals[lname]
 
 
-def config(colored=True, debugging=False, level='info'):
+def config(colored=True, debug=False, level='info'):
     level = str(level)  # to avoid error if level is None
 
     dictConfig({
@@ -90,7 +90,7 @@ def config(colored=True, debugging=False, level='info'):
         },
         'handlers': {
             'custom': {
-                'formatter': 'debugging' if debugging else 'default',
+                'formatter': 'debugging' if debug else 'default',
                 'class': 'yass.logger._ColorStreamHandler' if colored else 'logging.StreamHandler',
                 'stream': 'ext://sys.stdout'
             }

yass/models.py

@@ -15,6 +15,7 @@
 
 
 import re
+import string
 import time
 from typing import List, Optional
 
@@ -122,7 +123,7 @@ def extract(elements) -> List[str]:
         :type elements: list[Element]
         :return: extracted data
         """
-        return [element.text_content() for element in elements]
+        return [element.text_content().translate(str.maketrans('', '', string.whitespace)) for element in elements]
 
     def clean(self, urls: List[str]) -> List[str]:
         """
@@ -132,7 +133,7 @@ def clean(self, urls: List[str]) -> List[str]:
         :return: cleaned subdomains URLs
         """
         subdomains = []
-        regexp = re.compile(r'(.+://)?(?P<subdomain>(.+)\.{domain})([/\?].*)?'.format(domain=self.domain))
+        regexp = re.compile(r'(.+://)?(?P<subdomain>(.+)\.{domain})([/?].*)?'.format(domain=self.domain))
         for url in urls:
             match = regexp.match(url)
             if match and match.group('subdomain'):

yass/plugins.py

@@ -26,14 +26,15 @@ class Meta:
 class Ask(PluginBase):
     class Meta:
         search_url = 'https://www.ask.com/web'
-        subdomains_selector = 'p.PartialSearchResults-item-url'
+        subdomains_selector = 'div.PartialSearchResults-item-url'
 
 
 class Baidu(PluginBase):
     class Meta:
         search_url = 'https://www.baidu.com/s'
         query_param = 'wd'
-        subdomains_selector = 'span.g'
+        subdomains_selector = 'div.result div.c-row span'
+        request_delay = 1
 
 
 class Bing(PluginBase):
@@ -42,43 +43,43 @@ class Meta:
         subdomains_selector = '#b_results li.b_algo cite'
 
 
-# Anti-bot tricks make yass fails with Google
-#
+class DuckDuckGo(PluginBase):
+    class Meta:
+        search_url = 'https://html.duckduckgo.com/html/'
+        subdomains_selector = 'a.result__url'
+
+
+# Blocked by CAPTCHA
 # class Google(PluginBase):
 #     class Meta:
 #         search_url = 'https://www.google.com/search'
 #         subdomains_selector = '#search div.g cite'
 #         request_delay = 1
 
 
-class StartPage(PluginBase):
-    class Meta:
-        search_url = 'https://startpage.com/do/search'
-        subdomains_selector = 'span.search-item__url'
+# Blocked by CAPTCHA
+# class StartPage(PluginBase):
+#     class Meta:
+#         search_url = 'https://startpage.com/do/search'
+#         subdomains_selector = 'a.w-gl__result-url'
 
 
-class WebCrawler(PluginBase):
-    class Meta:
-        search_url = 'https://www.webcrawler.com/serp'
-        subdomains_selector = 'div.resultDisplayUrl'
-        include_param = ''
-        exclude_param = '-'
+# Blocked by "403: Forbidden" HTTP error
+# class WebCrawler(PluginBase):
+#     class Meta:
+#         search_url = 'https://www.webcrawler.com/serp'
+#         subdomains_selector = 'div.resultDisplayUrl'
 
 
 class Yahoo(PluginBase):
     class Meta:
         search_url = 'https://search.yahoo.com/search'
         query_param = 'p'
-        subdomains_selector = 'span.fz-ms.fw-m.fc-12th.wr-bw.lh-17'
+        subdomains_selector = 'ol.searchCenterMiddle li div > span'
 
 
-class Teoma(PluginBase):
-    class Meta:
-        search_url = 'https://www.teoma.com/web'
-        subdomains_selector = '.algo-display-url'
-
-
-class Exalead(PluginBase):
-    class Meta:
-        search_url = 'https://www.exalead.com/search/web/results'
-        subdomains_selector = 'li.media a.ellipsis'
+# Blocked by CAPTCHA
+# class Exalead(PluginBase):
+#     class Meta:
+#         search_url = 'https://www.exalead.com/search/web/results'
+#         subdomains_selector = 'li.media a.ellipsis'