Add --user-agent presets, fix version handling, improve HTML export and docs

- Add __version__ constant, fix hardcoded "2.0.0" in JSON export
- Add --user-agent flag with 13 presets (stealth, chrome, firefox, safari, edge, android, iphone, googlebot)
- New HTML export CSS: dark theme matching MetaDetective site, responsive, stats header (files/fields/identities)
- Add valid_output_directory: -o creates the directory automatically if it does not exist
- README: document display modes (singular vs all), --format constraint, export formats, --user-agent, depth warning for scraping

Author: franckferman

README.md

@@ -17,6 +17,8 @@
 [![PyPI][pypi-shield]](https://pypi.org/project/MetaDetective/)
 [![Docker][docker-shield]](https://hub.docker.com/r/franckferman/metadetective)
 [![License][license-shield]](https://github.com/franckferman/MetaDetective/blob/stable/LICENSE)
+![Python Stdlib](https://img.shields.io/badge/Python%20deps-stdlib%20only-green?style=for-the-badge)
+![ExifTool](https://img.shields.io/badge/Requires-ExifTool-blue?style=for-the-badge)
 
 <a href="https://github.com/franckferman/MetaDetective">
   <img src="https://raw.githubusercontent.com/franckferman/MetaDetective/stable/docs/github/graphical_resources/Logo-Without_background-MetaDetective.png" alt="MetaDetective" width="340">
@@ -190,37 +192,157 @@ jq '.unique.Author' MetaDetective_Export-*.json
 
 ### Web scraping
 
+MetaDetective can crawl a target website, discover downloadable files (PDF, DOCX, XLSX, images, etc.), and download them for local metadata analysis.
+
+**Two scraping modes:**
+
+- **`--download-dir`** - Download files to a local directory for analysis. This is the primary mode.
+- **`--scan`** - Preview only: list discovered files and stats without downloading. Useful for scoping before a full download.
+
+> `--scan` and `--download-dir` are mutually exclusive.
+
+**The `--depth` flag is critical.** By default, depth is **0**: MetaDetective only looks at the URL you provide. Most interesting files (reports, presentations, internal documents) are linked from subpages, not the homepage. **Always set `--depth 1` or higher for real engagements.**
+
+| Depth | Behavior |
+|-------|----------|
+| `0` (default) | Only the target URL. Finds files directly linked on that single page. |
+| `1` | Target URL + all pages linked from it. Covers most site structures. |
+| `2+` | Follows links N levels deep. Broader coverage, more requests, slower. |
+
+**Download (primary workflow):**
+
 ```bash
-# Scan target site, list files found
-python3 MetaDetective.py --scraping --scan --url https://target.com/
+# Standard download with depth 1 (recommended starting point)
+python3 MetaDetective.py --scraping --url https://target.com/ \
+  --download-dir ~/loot/ --depth 1
 
-# Filter by extension
-python3 MetaDetective.py --scraping --scan --url https://target.com/ --extensions pdf docx xlsx
+# Target specific file types
+python3 MetaDetective.py --scraping --url https://target.com/ \
+  --download-dir ~/loot/ --depth 2 --extensions pdf docx xlsx pptx
 
-# Download files (depth 2, 8 threads)
+# Parallel download (8 threads, 10 req/s)
 python3 MetaDetective.py --scraping --url https://target.com/ \
-  --download-dir ~/loot/ --extensions pdf docx --depth 2 --threads 8
+  --download-dir ~/loot/ --depth 2 --threads 8 --rate 10
 
-# Control request rate (requests/sec)
+# Follow external links (CDN, subdomain, partner sites)
 python3 MetaDetective.py --scraping --url https://target.com/ \
-  --download-dir ~/loot/ --rate 5
+  --download-dir ~/loot/ --depth 1 --follow-extern
 
-# Follow external links
+# Stealth: realistic User-Agent + low rate
 python3 MetaDetective.py --scraping --url https://target.com/ \
-  --download-dir ~/loot/ --follow-extern
+  --download-dir ~/loot/ --depth 2 --user-agent stealth --rate 2
+```
+
+**Scan (preview):**
+
+```bash
+# Quick preview: how many files are reachable?
+python3 MetaDetective.py --scraping --scan --url https://target.com/ --depth 1
+
+# Filter preview by extension
+python3 MetaDetective.py --scraping --scan --url https://target.com/ \
+  --depth 2 --extensions pdf docx
+```
+
+**Full pipeline (scrape + analyze + export):**
+
+```bash
+# Step 1: download files
+python3 MetaDetective.py --scraping --url https://target.com/ \
+  --download-dir ~/loot/ --depth 2 --extensions pdf docx xlsx
+
+# Step 2: analyze and export
+python3 MetaDetective.py -d ~/loot/ -e html -o ~/results/
+```
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--url` | required | Target URL |
+| `--download-dir` | - | Download destination (created if needed) |
+| `--scan` | - | Preview mode (no download) |
+| `--depth` | `0` | Link depth to follow. **Set to 1+ for real use.** |
+| `--extensions` | all supported | Filter by file type |
+| `--threads` | `4` | Concurrent download threads (1-100) |
+| `--rate` | `5` | Max requests per second (1-1000) |
+| `--follow-extern` | off | Follow links to external domains |
+| `--user-agent` | `MetaDetective/<ver>` | Custom or preset UA string |
+
+### Display modes
+
+MetaDetective offers two display modes that control how results are structured:
+
+**`--display singular`** (default) - Aggregates all unique values per field across every file. Best for OSINT: "who touched these documents?" at a glance.
+
+```bash
+# Default: deduplicated singular view
+python3 MetaDetective.py -d ./loot/
+
+# With formatted style (vertical list with markers)
+python3 MetaDetective.py -d ./loot/ --format formatted
+
+# With concise style (comma-separated on one line)
+python3 MetaDetective.py -d ./loot/ --format concise
+```
+
+**`--display all`** - One block per file with its individual metadata. Best for forensic analysis: examine each document's properties independently.
+
+```bash
+python3 MetaDetective.py -d ./loot/ --display all
+```
+
+> `--format` only works with `--display singular`. Using `--format` with `--display all` produces an error.
+
+### Export formats
+
+Three export formats are available. All respect the current `--display` mode.
+
+```bash
+# HTML report with dark theme, stats bar, and responsive layout
+python3 MetaDetective.py -d ./loot/ -e html
+
+# HTML per-file view
+python3 MetaDetective.py -d ./loot/ --display all -e html
+
+# Plain text
+python3 MetaDetective.py -d ./loot/ -e txt
+
+# JSON (structured, pipe into jq)
+python3 MetaDetective.py -d ./loot/ -e json
+
+# Custom output directory (created automatically if it does not exist)
+python3 MetaDetective.py -d ./loot/ -e html -o ~/results/
+
+# Custom filename suffix
+python3 MetaDetective.py -d ./loot/ -e json -c pentest-corp -o ~/results/
+```
+
+The HTML export includes a summary header showing total files analyzed, total metadata fields extracted, and unique identities found (from Author, Creator, and Last Modified By fields).
+
+### User-Agent (scraping)
+
+When scraping, MetaDetective identifies itself as `MetaDetective/<version>` by default. Use `--user-agent` to change this:
+
+```bash
+# Use a preset
+python3 MetaDetective.py --scraping --scan --url https://target.com/ --user-agent stealth
+
+# Available presets
+#   stealth, chrome-win, chrome-mac, chrome-linux,
+#   firefox-win, firefox-mac, firefox-linux,
+#   safari-mac, edge-win, android, iphone, googlebot
+
+# Custom string
+python3 MetaDetective.py --scraping --scan --url https://target.com/ \
+  --user-agent 'Mozilla/5.0 (compatible; MyScanner/1.0)'
 ```
 
-### Filtering and display options
+### Filtering
 
 | Flag | Description |
 |------|-------------|
 | `-t pdf docx` | Restrict to file types |
 | `-i admin anonymous` | Ignore values matching pattern (regex supported) |
 | `--parse-only Author Creator` | Extract only specified fields |
-| `--display all` | Show metadata per file |
-| `--display singular` | Deduplicated view across all files (default) |
-| `--format formatted` | Decorated output |
-| `--format concise` | Compact output |
 
 ### Supported formats