Enhance Shodan2DB: Improve database schema, optimize JSON parsing, and revamp HTML report template

- Updated database schema in shodan2db.py for better performance and integrity, including new indexes and constraints.
- Refactored JSON parsing logic to minimize memory usage and improve error handling.
- Introduced a new report template with enhanced styling and layout for better readability and user experience.
- Added support for dynamic filtering of hosts in the HTML report.
- Improved error messages and validation for file paths and database connections.

Author: atao

README.md

@@ -1,25 +1,29 @@
 # Shodan2DB
 
-🔌 Shodan export to SQLite database and generate an HTML report.
+🔌 Utility designed to parse Shodan JSON exports, store them into an optimized SQLite database, and generate comprehensive HTML vulnerability exposure reports.
 
 ## Purpose
 
-The purpose of this tool is to parse Shodan export files and put them into a SQLite database.
+The primary objective of this tool is to centralize Shodan export data into a structured SQLite environment for advanced querying and infrastructure monitoring.
 
-Exports bypass the restriction on "**vuln**" and "**tag**" tags, which are only available with Small Business, Corporate or Enterprise accounts. These data are included present in Shodan exports.
+**Account Restriction Bypass :** Shodan JSON exports natively include valuable fields like `"vulns"` and `"tags"` regardless of your tier. This tool extracts these premium attributes—normally locked behind expensive Small Business, Corporate, or Enterprise API accounts—and surfaces them directly for analysis.
 
-Once in the database, it's easier to analyze the data and extract a list of machines with CVEs.
+## Key Features
 
-Generate a report of found CVEs with HTML template.
-Customize the report in `templates/` folder using jinja2 and Bulma CSS.
+- **Blazing Fast Imports:** Leverages SQLite WAL (Write-Ahead Logging) and atomic transaction batching, reducing disk I/O bottlenecks. Perfect for running on lightweight hardware.
+- **Robust Exception Handling:** Implements defensive data extraction wrappers preventing common `KeyError` crashes caused by incomplete Shodan geographic metadata.
+- **Dynamic HTML Indexing:** Generates responsive, elegant dashboards styled with **Bulma CSS** and **Bootstrap Icons**, complete with client-side filtering and real-time live search.
+- **Strict Data Integrity:** Enforces database-level composite unique constraints (`UNIQUE(ip, cveid)`) preventing redundant storage overhead during overlapping historical imports.
 
 ## Requirements
 
+Ensure your local execution environment satisfies the necessary dependencies:
+
 ```bash
 pip install -r requirements.txt
 ```
 
-## Usage and options
+## Usage and Options
 
 ```bash
 Usage: shodan2db.py [OPTIONS] COMMAND [ARGS]...
@@ -32,7 +36,7 @@ Commands:
   parse   Parse the Shodan JSON export file and store data in the database.
 ```
 
-- *Command parse*
+### Command : parse
 
 ```bash
 Usage: shodan2db.py parse [OPTIONS]
@@ -41,69 +45,87 @@ Usage: shodan2db.py parse [OPTIONS]
 
 Options:
   -i, --input-file PATH  JSON export file from Shodan.  [required]
-  -d, --database TEXT    Database name.  [required]
+  -d, --database TEXT    Database name or path.         [required]
   -v, --verbose          Verbose mode.
   -h, --help             Show this message and exit.
 ```
 
-- *Command export*
+### Command : export
 
 ```bash
 Usage: shodan2db.py export [OPTIONS]
 
   Generate an HTML report from the data in the database.
 
 Options:
-  -d, --database PATH       Path to the SQLite database file.  [required]
-  -o, --report-file PATH    Output path for the HTML report file.  [default:
-                            shodan.html]
-  -t, --template-file PATH  Template used for the report.  [default:
-                            report.html]
+  -d, --database PATH       Path to the SQLite database file.         [required]
+  -o, --report-file PATH    Output path for the HTML report file.    [default: shodan.html]
+  -t, --template-file PATH  Path to the Jinja2 template file.        [default: templates/report.html]
   -v, --verbose             Verbose mode.
   -h, --help                Show this message and exit.
 ```
 
 ## Quickstart
 
-Do a search and click on "**Download Results**".
+1. Query Shodan via the web interface and click on **"Download Results"**.
 
 <img src="img/Shodan Export.png">
 
-Select the number of results to download.
+2. Select the number of results to download.
 
 <img src="img/Shodan Results.png">
 
-Download your results.
+3. Download your results.
 
 <img src="img/Shodan Download.png">
 
-Then import the results into the database using the command :
+4. Import your results and compile your threat report utilizing the CLI sequences:
 
 ```bash
-python .\shodan2db.py parse -i .\example_shodan.json -d .\example_database.db -v
-python .\shodan2db.py export -d .\example_database.db -o .\example_report.html -v
+# Step 1: Parse and seed your structured SQLite layer
+python shodan2db.py parse -i ./example_shodan.json -d ./example_database.db -v
+
+# Step 2: Extract analytics and output your HTML dashboard
+python shodan2db.py export -d ./example_database.db -o ./example_report.html -v
 ```
 
+5. See report file :
+
+- Dataset summary
+
+  <img src="img/report.png">
+
+- Host details
+
+  <img src="img/report2.png">
+
 **Tags** and **vulns** are visible directly in the **Summary** table.
 
 <img src="img/Summary.png">
 
-Example of report :
+## Database Architecture
 
-<img src="img/report.png">
+Upon initialization, the tool optimizes SQLite pragmas and automatically structures the underlying relational objects:
 
-## Templates
+- **`services`**: Houses core operational network service logs, banners, geo-coordinates, ISP allocations, and metadata.
+- **`vulnerabilities`**: Stores indexed granular mappings of CVE IDs alongside their verified CVSS scores and threat summary descriptions.
+- **`Summary` (View)**: An internal prioritized virtual evaluation layer sorting network entities by physical exposure risk (`nbvulns DESC`).
 
-Customize the report template in [templates/report.html](templates/report.html)
+## Templates Customization
 
-The current one uses [Bulma CSS](https://bulma.io/).
+The presentation architecture is entirely modular. You can seamlessly customize the structural layout or interface themes inside `templates/report.html`. 
+
+The default layout leverages **Bulma CSS** to render high-contrast, professional cybersecurity matrices with custom conditional coloring for CVSS threat scales.
 
 ## Development
 
-Use Shodan2DB as python class
+You can import `Shodan2DB` directly into third-party automated playbooks or continuous integration loops as a native Python class:
 
 ```python
-import shodan2db as sh
+from shodan2db import Shodan2DB
 
-sh.Shodan2DB.export(verbose=True, exportfile="test.html", database="test.db")
-```
+# Programmatically trigger parsing and reporting
+Shodan2DB.prepare_database(verbose=True, database="production_audit")
+Shodan2DB.parser(verbose=True, inputfile="raw_shodan.json", database="production_audit")
+Shodan2DB.export(verbose=True, exportfile="exposure.html", database="production_audit", template_file="templates/report.html")
+```