Add MkDocs documentation

Author: aleff-github

docs/api.md

@@ -0,0 +1,78 @@
+# Internal API reference (for contributors)
+
+> This is not a public library API. It is provided to help contributors and maintainers.
+
+## `wayparam.http`
+
+### `HttpConfig`
+Fields:
+- `timeout_s: float` (default: 30.0)
+- `retries: int` (default: 4)
+- `backoff_base_s: float`
+- `max_backoff_s: float`
+- `user_agent: Optional[str]`
+- `proxy: Optional[str]`
+
+### `get_text(client, url, params, config) -> str`
+Performs a GET request and returns response text, with retry/backoff behavior.
+
+Raises `RuntimeError` after retries with a message like:
+- `HTTP request failed after retries (status=503): ...`
+- `HTTP request failed after retries (no-status): ...`
+
+## `wayparam.wayback`
+
+### `CdxOptions`
+Fields:
+- `include_subdomains: bool`
+- `collapse: str | None` (default: `urlkey`)
+- `from_ts: str | None`
+- `to_ts: str | None`
+- `limit: int`
+- `filters: list[str] | None`
+
+### `iter_original_urls(domain, client, http_config, rate_limiter, opt) -> AsyncIterator[str]`
+Yields “original” URLs from the CDX API, handling paging/resumeKey.
+
+## `wayparam.normalize`
+
+### `NormalizeOptions`
+Fields:
+- `placeholder: str`
+- `keep_values: bool`
+- `only_params: bool`
+- `drop_tracking: bool`
+- `drop_empty: bool`
+- `sort_params: bool`
+
+### `canonicalize_url(url, opt) -> str | None`
+Returns a canonicalized URL or `None` if filtered out or invalid.
+
+## `wayparam.filters`
+
+### `FilterOptions`
+Fields:
+- `ext_blacklist: set[str]`
+- `ext_whitelist: set[str] | None`
+- `path_exclude_regex: list[re.Pattern] | None`
+
+### `is_boring(url, opt) -> bool`
+Returns True if the URL should be filtered out as “boring”.
+
+## `wayparam.output`
+
+### `UrlRecord`
+Fields:
+- `domain: str`
+- `url: str`
+- `source: str` (default: `wayback`)
+- `fetched_at: str | None`
+
+### `write_record(fh, rec, fmt)`
+Writes one record to a file handle.
+
+### `print_record_stdout(rec, fmt)`
+Prints one record to stdout.
+
+### `print_hint_stderr(message)`
+Prints diagnostics to stderr.

docs/architecture.md

@@ -0,0 +1,29 @@
+# Architecture
+
+wayparam is intentionally modular. Each module has a single responsibility, which makes the tool easier to test, audit, and package.
+
+## High-level data flow
+
+1. **cli.py**
+   - Parses args
+   - Builds option objects
+   - Orchestrates concurrency
+2. **wayback.py**
+   - Builds CDX query parameters
+   - Handles pagination/resumeKey
+3. **http.py**
+   - Makes resilient HTTP requests (retries, backoff)
+4. **filters.py**
+   - Drops “boring” URLs (static assets) early
+5. **normalize.py**
+   - Canonicalizes and normalizes URLs (stable output)
+6. **output.py**
+   - Writes records to files and/or stdout (txt/jsonl)
+7. **ratelimit.py**
+   - Global RPS limiter (optional)
+
+## Why this structure matters
+
+- unit tests focus on pure logic (`normalize.py`, `filters.py`, parsing)
+- integration tests mock HTTP at the transport layer (httpx MockTransport)
+- CLI stays pipeline-friendly: stdout is clean and predictable

docs/cli/examples.md

@@ -0,0 +1,76 @@
+# Practical CLI examples
+
+This page contains practical, copy/paste-friendly examples.
+
+---
+
+## Basic collection
+
+```bash
+wayparam -d example.com
+```
+
+Writes results to:
+- `results/example.com.txt` (default format `txt`)
+
+---
+
+## Multiple domains
+
+```bash
+wayparam -l domains.txt
+```
+
+---
+
+## Streaming (pipelines)
+
+### Deduplicate and save
+```bash
+wayparam -d example.com --stdout --no-files | sort -u > urls.txt
+```
+
+### Filter by keyword
+```bash
+wayparam -d example.com --stdout --no-files | grep -i "redirect"
+```
+
+### JSONL + jq
+```bash
+wayparam -d example.com --stdout --no-files --format jsonl | jq -r '.url' | sort -u
+```
+
+---
+
+## Subdomains + time filters
+
+```bash
+wayparam -d example.com --include-subdomains --from 2018 --to 2021
+```
+
+---
+
+## Focus on “dynamic” endpoints
+
+Exclude typical static paths:
+```bash
+wayparam -d example.com --exclude-path-regex "^/static/" --exclude-path-regex "^/assets/"
+```
+
+---
+
+## Being polite to Wayback (recommended)
+
+```bash
+wayparam -l domains.txt --rps 1 --concurrency 2
+```
+
+---
+
+## Keep parameter values (careful)
+
+```bash
+wayparam -d example.com --keep-values
+```
+
+Use only if you understand the privacy implications.