Port sqlite operations to APSW (#1076)

* porting sqlite operations to APSW and making necessary changes with unittests for db.py

* ruff fixes

* ruff fixes 2.0

* improved test_db.py

* ruff fixes

* fix for subdomains

* ruff

* updated poetry lock and merged master

* fixing test_db.py, the other one is still failing because of some mocking issues

* fixing tests

* poetry lock updates

* bot suggestions

* update documentation for databases

* spelling fix

* docstring update

* added logic to toggle between APSW and SQLAlchemy for sqlite databases for backward compatibility

* updated docs

* bug fixing

* coderabbit suggested changes and test fixes

* coderabbit suggested changes

* closing connections if no targets are present

* added new parameter for retires in config and ensured closure of all connections

* triple quotes preserve whitespaces which causes indentation issues in tests and make them more fragile

* some bug fixes

* removed unncessary file

* error handling

* Update code

* some formatting changes

* removing report

* used context managers for file opens in db.py and updated test to handle __enter__ (as earlier we didn't have to)

* fix: removing duplicate python instance

* fixed lock file

---------

Signed-off-by: Achintya Jai <153343775+pUrGe12@users.noreply.github.com>
Co-authored-by: Arkadii Yakovets <arkadii.yakovets@owasp.org>

Author: pUrGe12

docs/Usage.md

@@ -646,58 +646,71 @@ python nettacker.py --start-api --api-access-log --api-port 8080 --api-debug-mod
 ![](https://github.com/aman566/DiceGameJS/blob/master/Screencast-from-Tuesday-09-June-2020-02-32-32-IST-_online-video-cutter.com_.gif)
 
 # Database
-OWASP Nettacker, currently supports two databases:
-- SQLite
+
+OWASP Nettacker, currently supports three databases:
+
 - MySQL
+- PostgreSQL
+- SQLite
+
 The default database is SQLite. You can, however, configure the db to your liking.
+
 ## SQLite configuration
-The SQLite database can be configured in `core/config.py` file under the `_database_config()` function. Here is a sample configuration:
-```
-return {
-        "DB": "sqlite",
-        "DATABASE":  _paths()["home_path"] + "/nettacker.db", # This is the location of your db
-        "USERNAME": "",
-        "PASSWORD": "",
-        "HOST": "",
-        "PORT": ""
-    }
+
+The configurations below are for a SQLite wrapper called **APSW** (Another Python SQLite Wrapper). The configurations can be found inside `nettacker/config.py` file under the `DBConfig` class. 
+
+```python
+    engine = "sqlite"
+    name = str(CWD / ".nettacker/data/nettacker.db")
+    host = ""
+    port = ""
+    username = ""
+    password = ""
+    ssl_mode = "disable"
+    journal_mode = "WAL"
+    synchronous_mode = "NORMAL"
 ```
+
+These are the default and recommended settings. Feel free to play around and change them according to need. To use SQLite database, ensure that the `engine` value is set to `sqlite` and the `name` is the path to your database. The `journal_mode` and `synchronous_mode` are chosen to be optimal for multithreaded I/O operations.
+
+> Note: You can choose to use a lite wrapper for Sqlite called APSW by setting the `use_apsw_for_sqlite` parameter inside config to True for performance enhancements.
+
 ## MySQL configuration:
-The MySQL database can be configured in `core/config.py` file under the `_database_config()` function. Here is a sample configuration:
-```
-return {
-        "DB": "mysql",
-        "DATABASE": "nettacker", # This is the name of your db
-        "USERNAME": "username",
-        "PASSWORD": "password",
-        "HOST": "localhost or some other host",
-        "PORT": "3306 or some other custom port"
-    }
-```
-After this configuration:
-1. Open the configuration file of mysql(`/etc/mysql/my.cnf` in case of linux) as a sudo user
-2. Add this to the end of the file :
-``` 
-[mysqld]  
-sql_mode = "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
+
+The MySQL database can be configured in `nettacker/config.py` file under the `DBConfig` class. Here is a sample configuration:
+
+```python
+    engine = "mysql"
+    name = "nettacker"
+    host = "localhost"
+    port = 3306
+    username = "root"
+    password = "some-password"
+    ssl_mode = "disable"
+    journal_mode = "WAL"
+    synchronous_mode = "NORMAL"
 ```
-3.  Restart MySQL
+
+Only the relevant fields will be considered and you don't need to update/change/remove the irrelevant ones (`ssl_mode`, `journal_mode` and `synchronous_mode` aren't relevant in this case).
 
 ## Postgres Configuration
 
-The Postgres database can be configured in core/config.py file under the _database_config() function. Here is a sample configuration:
-`
-return {
-        "DB": "postgreas",
-        "DATABASE": "nettacker" # Name of db
-        "USERNAME": "username",
-        "PASSWORD": "password",
-        "HOST": "localhost or some other host",
-        "PORT": "5432 or some other custom port"
-    }
-`
-After this configuration please comment out the following line in database/db.py   `connect_args={'check_same_thread': False}` 
+The Postgres database can be configured in `nettacker/config.py` file under the `DBConfig` class. Here is a sample configuration:
+
+```python
+    engine = "postgres"
+    name = "nettacker"
+    host = "localhost"
+    port = 5432
+    username = "root"
+    password = "some-password"
+    ssl_mode = "disable"
+    journal_mode = "WAL"
+    synchronous_mode = "NORMAL"
+```
 
+In this case the irrelevant fields are `journal_mode` and `synchronous_mode`. You don't have to update/change/remove them.
 
+**Note**: If you want encryption, then set `ssl_mode` to `require`.
 
 Let me know if you have any more questions.

nettacker/api/core.py

@@ -4,8 +4,8 @@
 
 from nettacker.config import Config
 from nettacker.core.app import Nettacker
-from nettacker.core.messages import messages as _
 from nettacker.core.messages import get_languages
+from nettacker.core.messages import messages as _
 
 
 def get_value(flask_request, key):

nettacker/api/engine.py

@@ -8,39 +8,38 @@
 from threading import Thread
 from types import SimpleNamespace
 
-from flask import Flask, jsonify
+from flask import Flask, Response, abort, jsonify, make_response, render_template
 from flask import request as flask_request
-from flask import render_template, abort, Response, make_response
 from werkzeug.serving import WSGIRequestHandler
 from werkzeug.utils import secure_filename
 
 from nettacker import logger
 from nettacker.api.core import (
-    get_value,
+    api_key_is_valid,
     get_file,
-    mime_types,
-    scan_methods,
-    profiles,
+    get_value,
     graphs,
     languages_to_country,
-    api_key_is_valid,
+    mime_types,
+    profiles,
+    scan_methods,
 )
 from nettacker.api.helpers import structure
 from nettacker.config import Config
 from nettacker.core.app import Nettacker
 from nettacker.core.die import die_failure
 from nettacker.core.graph import create_compare_report
 from nettacker.core.messages import messages as _
-from nettacker.core.utils.common import now, generate_compare_filepath
+from nettacker.core.utils.common import generate_compare_filepath, now
 from nettacker.database.db import (
     create_connection,
     get_logs_by_scan_id,
-    select_reports,
     get_scan_result,
     last_host_logs,
+    logs_to_report_html,
     logs_to_report_json,
     search_logs,
-    logs_to_report_html,
+    select_reports,
 )
 from nettacker.database.models import Report
 

nettacker/config.py

@@ -3,7 +3,7 @@
 from pathlib import Path
 
 from nettacker import version
-from nettacker.core.utils.common import now, generate_random_token
+from nettacker.core.utils.common import generate_random_token, now
 
 CWD = Path.cwd()
 PACKAGE_PATH = Path(__file__).parent
@@ -82,7 +82,11 @@ class DbConfig(ConfigBase):
     For sqlite database:
         fill the name of the DB as sqlite,
         DATABASE as the name of the db user wants
-        other details can be left empty
+        Set the journal_mode (default="WAL") and
+        synchronous_mode (default="NORMAL"). Rest
+        of the fields can be left empty
+        This is the default database:
+        str(CWD / ".nettacker/data/nettacker.db")
     For mysql users:
         fill the ENGINE name of the DB as mysql
         NAME as the name of the database you want to create
@@ -104,6 +108,8 @@ class DbConfig(ConfigBase):
     username = ""
     password = ""
     ssl_mode = "disable"
+    journal_mode = "WAL"
+    synchronous_mode = "NORMAL"
 
 
 class PathConfig:
@@ -142,6 +148,9 @@ class DefaultSettings(ConfigBase):
     parallel_module_scan = 1
     passwords = None
     passwords_list = None
+    # Setting to toggle between APSW and SQLAlchemy for sqlite databases.
+    use_apsw_for_sqlite = False
+
     ping_before_scan = False
     ports = None
     profiles = None
@@ -151,6 +160,9 @@ class DefaultSettings(ConfigBase):
         random_chars=generate_random_token(10),
     )
     retries = 1
+    max_retries = 3
+    max_submit_query_retry = 100
+    retry_delay = 0.1
     scan_ip_range = False
     scan_subdomains = False
     selected_modules = None

nettacker/core/app.py

@@ -12,16 +12,16 @@
 from nettacker.config import Config, version_info
 from nettacker.core.arg_parser import ArgParser
 from nettacker.core.die import die_failure
-from nettacker.core.graph import create_report, create_compare_report
+from nettacker.core.graph import create_compare_report, create_report
 from nettacker.core.ip import (
-    get_ip_range,
     generate_ip_range,
-    is_single_ipv4,
-    is_ipv4_range,
+    get_ip_range,
     is_ipv4_cidr,
-    is_single_ipv6,
-    is_ipv6_range,
+    is_ipv4_range,
     is_ipv6_cidr,
+    is_ipv6_range,
+    is_single_ipv4,
+    is_single_ipv6,
 )
 from nettacker.core.messages import messages as _
 from nettacker.core.module import Module
@@ -158,9 +158,7 @@ def expand_targets(self, scan_id):
 
             for target in copy.deepcopy(self.arguments.targets):
                 for row in find_events(target, "subdomain_scan", scan_id):
-                    for sub_domain in json.loads(row.json_event)["response"]["conditions_results"][
-                        "content"
-                    ]:
+                    for sub_domain in json.loads(row)["response"]["conditions_results"]["content"]:
                         if sub_domain not in self.arguments.targets:
                             self.arguments.targets.append(sub_domain)
         # icmp_scan

nettacker/core/arg_parser.py

@@ -5,16 +5,16 @@
 import yaml
 
 from nettacker import all_module_severity_and_desc
-from nettacker.config import version_info, Config
+from nettacker.config import Config, version_info
 from nettacker.core.die import die_failure, die_success
 from nettacker.core.ip import (
-    is_single_ipv4,
-    is_single_ipv6,
+    generate_ip_range,
     is_ipv4_cidr,
-    is_ipv6_range,
-    is_ipv6_cidr,
     is_ipv4_range,
-    generate_ip_range,
+    is_ipv6_cidr,
+    is_ipv6_range,
+    is_single_ipv4,
+    is_single_ipv6,
 )
 from nettacker.core.messages import messages as _
 from nettacker.core.template import TemplateLoader

nettacker/core/graph.py

@@ -9,17 +9,17 @@
 
 import texttable
 
-from nettacker import logger, all_module_severity_and_desc
+from nettacker import all_module_severity_and_desc, logger
 from nettacker.config import Config, version_info
 from nettacker.core.die import die_failure
 from nettacker.core.messages import messages as _
 from nettacker.core.utils.common import (
+    generate_compare_filepath,
     merge_logs_to_list,
     now,
     sanitize_path,
-    generate_compare_filepath,
 )
-from nettacker.database.db import get_logs_by_scan_id, submit_report_to_db, get_options_by_scan_id
+from nettacker.database.db import get_logs_by_scan_id, get_options_by_scan_id, submit_report_to_db
 
 log = logger.get_logger()
 nettacker_path_config = Config.path
@@ -86,7 +86,7 @@ def build_text_table(events):
     table_headers = ["date", "target", "module_name", "port", "logs"]
     _table.add_rows([table_headers])
     for event in events:
-        log = merge_logs_to_list(json.loads(event["json_event"]), [])
+        log = merge_logs_to_list(event, [])
         _table.add_rows(
             [
                 table_headers,
@@ -252,15 +252,15 @@ def create_report(options, scan_id):
         )
         index = 1
         for event in all_scan_logs:
-            log_list = merge_logs_to_list(json.loads(event["json_event"]), [])
+            log_list = merge_logs_to_list(event, [])
             html_table_content += log_data.table_items.format(
                 event["date"],
                 event["target"],
                 event["module_name"],
                 event["port"],
                 "<br>".join(log_list) if log_list else "Detected",  # event["event"], #log
                 index,
-                html.escape(event["json_event"]),
+                html.escape(json.dumps(event)),
             )
             index += 1
         html_table_content += (

nettacker/core/lib/base.py

@@ -10,8 +10,8 @@
 from nettacker.config import Config
 from nettacker.core.messages import messages as _
 from nettacker.core.utils.common import merge_logs_to_list, remove_sensitive_header_keys
-from nettacker.database.db import find_temp_events, submit_temp_logs_to_db, submit_logs_to_db
-from nettacker.logger import get_logger, TerminalCodes
+from nettacker.database.db import find_temp_events, submit_logs_to_db, submit_temp_logs_to_db
+from nettacker.logger import TerminalCodes, get_logger
 
 log = get_logger()
 
@@ -52,7 +52,7 @@ def get_dependent_results_from_database(self, target, module_name, scan_id, even
             while True:
                 event = find_temp_events(target, module_name, scan_id, event_name)
                 if event:
-                    events.append(json.loads(event.event)["response"]["conditions_results"])
+                    events.append(json.loads(event)["response"]["conditions_results"])
                     break
                 time.sleep(0.1)
         return events

nettacker/core/lib/http.py

@@ -11,10 +11,10 @@
 
 from nettacker.core.lib.base import BaseEngine
 from nettacker.core.utils.common import (
-    replace_dependent_response,
-    reverse_and_regex_condition,
     get_http_header_key,
     get_http_header_value,
+    replace_dependent_response,
+    reverse_and_regex_condition,
 )
 
 asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

nettacker/core/lib/socket.py

@@ -11,7 +11,7 @@
 import time
 
 from nettacker.core.lib.base import BaseEngine, BaseLibrary
-from nettacker.core.utils.common import reverse_and_regex_condition, replace_dependent_response
+from nettacker.core.utils.common import replace_dependent_response, reverse_and_regex_condition
 
 log = logging.getLogger(__name__)