Introduction of PyFunceble v4.1.0 (Blue Duckling: Ixora)!

Fixed:
  * Fatal error when a DNS Name longer that 255 octets is given.
  * URL with scheme and port are no longer `INVALID`.
  * Encoding format of CSV. We now enforce UTF-8. (#291)
  * Restrict `python-box` version. (#294)

New:
  * The abilitiy to use proxies. (#132)
    This include the usage of `socks5` for `.onion` test.
  * The `--max-http-retries` argument and logic.
  * The `--*-lookup-only` argument and logic. (#276)
  * The `--chancy-test` mode. It is a mode that break the
    multiprocessing safety in place to use the full potential of the
    machine.
    **WARNING**: This is dangerous. Use at your own risk.
  * When testing URL we now do a DNS lookup first.
  * The DNS Lookup Tool has a max depth of 60. This avoid infinite loop.
  * Domain registrars are now extracted from the WHOIS record and
    provided as part of the output datasets. (#272)
  * It is now possible to display the Top N registrar. (#272)
  * Multiprocessing-Safe HTTP/HTTPS requests.
  * SPECIAL rules:
      - *.dr.ag
      -
  * Workaround Y2K28 - until database vendors fix it.
  * A way to automate  CLI demos.

Improvement:
  * IPv6 format of DNS server was improved to accept brackets.
  * Regex of the WHOIS "decoder" are now case insensitive.
  * The way workers/processes are spawned and started. It is now
    possible to differ the startup of a process to a time in the future.
  * We now run against the inactive and autocontinue dataset when it
    is really necessary.
  * Our "self-healing" method to avoid false positive is now
    stronger. (#282)
  * The syntax checker does not only follow RFCs, it now tries to accept
    real world practices. (#282)
  * Documentation (thanks to @spirillen).

Maintenance:
  * Deployment of AUR packages is now safer.
  * Added missing documentation of SPECIAL rules.
  * Fix typo.
  * It's 2022.

Contributors:
  * @avatartw
  * @cdgriffith
  * @mitchellkrogza
  * @ryanbr
  * @spirillen
  * @veracioux

Author: funilrys

.github/workflows/dev.yml

@@ -359,13 +359,8 @@ jobs:
 
       - name: Install dependencies
         run: |
-          curl -LO "https://repo.archlinuxcn.org/x86_64/${patched_glibc}" && \
-          bsdtar -C / -xvf "$patched_glibc" && \
-          rm ${patched_glibc} && \
           pacman -Syu --needed --noconfirm && \
-          pacman -S --needed --noconfirm git binutils pacman-contrib openssh rsync python python-setuptools
-        env:
-          patched_glibc: glibc-linux4-2.33-4-x86_64.pkg.tar.zst
+          pacman -S --needed --noconfirm git binutils pacman-contrib openssh rsync python python-setuptools jq
 
       - uses: actions/checkout@v2
         name: Clone repository
@@ -380,6 +375,15 @@ jobs:
 
           latest=$(python3 setup.py --version)
 
+          if [[ $(python3 setup.py --name) == "PyFunceble-dev" ]]
+          then
+            licenseURL="https://raw.githubusercontent.com/funilrys/PyFunceble/dev/LICENSE"
+            newURL=$(curl -s https://pypi.org/pypi/PyFunceble-dev/json | jq -r ".releases.\"${latest}\"[] | select(.filename | strings | test(\".tar\")).url")
+          else
+            licenseURL="https://raw.githubusercontent.com/funilrys/PyFunceble/master/LICENSE"
+            newURL=$(curl -s https://pypi.org/pypi/PyFunceble/json | jq -r ".releases.\"${latest}\"[] | select(.filename | strings | test(\".tar\")).url")
+          fi
+
           git config --global user.name ${{ github.actor }}
           git config --global user.email ${{ secrets.GIT_EMAIL }}
           git clone ssh://[email protected]/pyfunceble-dev.git ${aurRepoDir}
@@ -398,6 +402,7 @@ jobs:
 
           sed "/^pkgver=/s/.*/pkgver=${latest}/" -i PKGBUILD
           sed "/^pkgrel=/s/.*/pkgrel=1/" -i PKGBUILD
+          sed "s,^source=(.*),source=('${newURL}' '${licenseURL}')," -i PKGBUILD
 
           su makepkg -c "updpkgsums"
           su makepkg -c "makepkg --printsrcinfo" > .SRCINFO

.github/workflows/stable.yml

@@ -269,9 +269,8 @@ jobs:
 
       - name: Install dependencies
         run: |
-          mkdir -p /var/lib/pacman/
-          pacman -Syu --noconfirm
-          pacman -S --needed --noconfirm git binutils pacman-contrib openssh rsync python python-setuptools
+          pacman -Syu --needed --noconfirm && \
+          pacman -S --needed --noconfirm git binutils pacman-contrib openssh rsync python python-setuptools jq
 
       - uses: actions/checkout@v2
         name: Clone repository
@@ -286,6 +285,15 @@ jobs:
 
           latest=$(python3 setup.py --version)
 
+          if [[ $(python3 setup.py --name) == "PyFunceble-dev" ]]
+          then
+            licenseURL="https://raw.githubusercontent.com/funilrys/PyFunceble/dev/LICENSE"
+            newURL=$(curl -s https://pypi.org/pypi/PyFunceble-dev/json | jq -r ".releases.\"${latest}\"[] | select(.filename | strings | test(\".tar\")).url")
+          else
+            licenseURL="https://raw.githubusercontent.com/funilrys/PyFunceble/master/LICENSE"
+            newURL=$(curl -s https://pypi.org/pypi/PyFunceble/json | jq -r ".releases.\"${latest}\"[] | select(.filename | strings | test(\".tar\")).url")
+          fi
+
           git config --global user.name ${{ github.actor }}
           git config --global user.email ${{ secrets.GIT_EMAIL }}
           git clone ssh://[email protected]/pyfunceble.git ${aurRepoDir}
@@ -299,13 +307,12 @@ jobs:
           echo "Previous version: ${current}"
           echo "New version: ${latest}"
 
-          [[ "${current}" == "${latest}" ]] && exit 0
-
           useradd makepkg
           chown makepkg:root . -R
 
           sed "/^pkgver=/s/.*/pkgver=${latest}/" -i PKGBUILD
           sed "/^pkgrel=/s/.*/pkgrel=1/" -i PKGBUILD
+          sed "s,^source=(.*),source=('${newURL}' '${licenseURL}')," -i PKGBUILD
 
           su makepkg -c "updpkgsums"
           su makepkg -c "makepkg --printsrcinfo" > .SRCINFO

LICENSE

@@ -175,7 +175,7 @@
 
    END OF TERMS AND CONDITIONS
 
-   Copyright 2017, 2018, 2019, 2020, 2021 Nissar Chababy
+   Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

PyFunceble/__init__.py

@@ -34,7 +34,7 @@
 ::
 
 
-    Copyright 2017, 2018, 2019, 2020, 2021 Nissar Chababy
+    Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

PyFunceble/checker/__init__.py

@@ -35,7 +35,7 @@
 ::
 
 
-    Copyright 2017, 2018, 2019, 2020, 2021 Nissar Chababy
+    Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

PyFunceble/checker/availability/__init__.py

@@ -35,7 +35,7 @@
 ::
 
 
-    Copyright 2017, 2018, 2019, 2020, 2021 Nissar Chababy
+    Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

PyFunceble/checker/availability/base.py

@@ -35,7 +35,7 @@
 ::
 
 
-    Copyright 2017, 2018, 2019, 2020, 2021 Nissar Chababy
+    Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
@@ -513,12 +513,23 @@ def subject_propagator(self) -> "CheckerBase":
     def should_we_continue_test(self, status_post_syntax_checker: str) -> bool:
         """
         Checks if we are allowed to continue a standard testing.
+
+        Rules:
+            1. No status available yet. Continue to next test method.
+            2. Status is/still INVALID. Continue to next test method.
+            3. Above are False. Not allowed to continue to next test method.
         """
 
-        return bool(
-            not self.status.status
-            or status_post_syntax_checker == PyFunceble.storage.STATUS.invalid
-        )
+        if not self.status.status:
+            return True
+
+        if (
+            status_post_syntax_checker == PyFunceble.storage.STATUS.invalid
+            and self.status.status == PyFunceble.storage.STATUS.invalid
+        ):
+            return True
+
+        return False
 
     def guess_and_set_use_extra_rules(self) -> "AvailabilityCheckerBase":
         """
@@ -663,6 +674,9 @@ def query_syntax_checker(self) -> "AvailabilityCheckerBase":
     def query_dns_record(self) -> Optional[Dict[str, Optional[List[str]]]]:
         """
         Tries to query the DNS record(s) of the given subject.
+
+        .. versionchanged:: 4.1.0b8.dev
+           Lookup order relative to actual subject.
         """
 
         PyFunceble.facility.Logger.info(
@@ -672,14 +686,16 @@ def query_dns_record(self) -> Optional[Dict[str, Optional[List[str]]]]:
 
         result = {}
 
-        if self.status.subdomain_syntax:
+        if self.status.idna_subject != self.dns_query_tool.subject:
+            self.domain_syntax_checker.set_subject(self.dns_query_tool.subject)
+            self.ip_syntax_checker.set_subject(self.dns_query_tool.subject)
+
+        if self.domain_syntax_checker.is_valid_subdomain():
             lookup_order = ["NS", "A", "AAAA", "CNAME", "DNAME"]
-        elif self.status.domain_syntax:
-            lookup_order = ["NS", "CNAME", "A", "AAAA", "DNAME"]
-        elif self.status.ip_syntax:
+        elif self.ip_syntax_checker.is_valid():
             lookup_order = ["PTR"]
         else:
-            lookup_order = []
+            lookup_order = ["NS", "CNAME", "A", "AAAA", "DNAME"]
 
         if lookup_order:
             for record_type in lookup_order:
@@ -692,6 +708,11 @@ def query_dns_record(self) -> Optional[Dict[str, Optional[List[str]]]]:
 
                     break
 
+        if self.status.idna_subject != self.dns_query_tool.subject:
+            # Switch back subject because we don't want to break subsequential calls.
+            self.domain_syntax_checker.set_subject(self.status.idna_subject)
+            self.ip_syntax_checker.set_subject(self.status.idna_subject)
+
         PyFunceble.facility.Logger.debug("DNS Record:\n%r", result)
 
         PyFunceble.facility.Logger.info(
@@ -741,9 +762,8 @@ def try_to_query_status_from_whois(
             if not known_record:
                 # We assume that expired dataset are never saved into the
                 # dataset.
-                self.status.expiration_date = (
-                    self.whois_query_tool.get_expiration_date()
-                )
+                self.status.expiration_date = self.whois_query_tool.expiration_date
+                self.status.registrar = self.whois_query_tool.lookup_record.registrar
                 self.status.whois_record = self.whois_query_tool.lookup_record.record
 
                 if (
@@ -766,9 +786,11 @@ def try_to_query_status_from_whois(
                     )
             else:
                 self.status.expiration_date = known_record["expiration_date"]
+                self.status.registrar = known_record["registrar"]
                 self.status.whois_record = None
         else:
-            self.status.expiration_date = self.whois_query_tool.get_expiration_date()
+            self.status.expiration_date = self.whois_query_tool.expiration_date
+            self.status.registrar = self.whois_query_tool.lookup_record.registrar
             self.status.whois_record = self.whois_query_tool.lookup_record.record
 
         if self.status.expiration_date:
@@ -790,11 +812,14 @@ def try_to_query_status_from_whois(
     def try_to_query_status_from_dns(self) -> "AvailabilityCheckerBase":
         """
         Tries to query the status from the DNS lookup.
+
+        .. versionchanged:: 4.1.0b7.dev
+           Logging return the correct subject.
         """
 
         PyFunceble.facility.Logger.info(
             "Started to try to query the status of %r from: DNS Lookup",
-            self.status.idna_subject,
+            self.dns_query_tool.subject,
         )
 
         lookup_result = self.query_dns_record()
@@ -806,12 +831,12 @@ def try_to_query_status_from_dns(self) -> "AvailabilityCheckerBase":
 
             PyFunceble.facility.Logger.info(
                 "Could define the status of %r from: DNS Lookup",
-                self.status.idna_subject,
+                self.dns_query_tool.subject,
             )
 
         PyFunceble.facility.Logger.info(
             "Finished to try to query the status of %r from: DNS Lookup",
-            self.status.idna_subject,
+            self.dns_query_tool.subject,
         )
 
         return self

PyFunceble/checker/availability/domain.py

@@ -35,7 +35,7 @@
 ::
 
 
-    Copyright 2017, 2018, 2019, 2020, 2021 Nissar Chababy
+    Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

PyFunceble/checker/availability/domain_and_ip.py

@@ -35,7 +35,7 @@
 ::
 
 
-    Copyright 2017, 2018, 2019, 2020, 2021 Nissar Chababy
+    Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.

PyFunceble/checker/availability/extra_rules.py

@@ -35,7 +35,7 @@
 ::
 
 
-    Copyright 2017, 2018, 2019, 2020, 2021 Nissar Chababy
+    Copyright 2017, 2018, 2019, 2020, 2022 Nissar Chababy
 
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
@@ -85,9 +85,11 @@ def __init__(self, status: Optional[AvailabilityCheckerStatus] = None) -> None:
                 (self.__switch_to_down_if, 410),
             ],
             r"\.24\.eu$": [(self.__switch_to_down_if, 503)],
+            r"\.altervista\.org$": [(self.__switch_to_down_if, 403)],
             r"\.angelfire\.com$": [(self.__switch_to_down_if, 404)],
             r"\.blogspot\.": [self.__handle_blogspot],
             r"\.canalblog\.com$": [(self.__switch_to_down_if, 404)],
+            r"\.dr\.ag$": [(self.__switch_to_down_if, 503)],
             r"\.fc2\.com$": [self.__handle_fc2_dot_com],
             r"\.github\.io$": [(self.__switch_to_down_if, 404)],
             r"\.godaddysites\.com$": [(self.__switch_to_down_if, 404)],