Release

Author: aleff-github

.github/workflows/ci.yml

@@ -0,0 +1,22 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install
+        run: |
+          python -m pip install -U pip
+          pip install -e '.[dev]'
+      - name: Lint
+        run: ruff check .
+      - name: Tests
+        run: pytest -q

.gitignore

@@ -0,0 +1,12 @@
+# ---------- Outputs / runtime artifacts ----------
+results/
+
+# ---------- Python bytecode / caches ----------
+__pycache__/
+.pytest_cache/
+
+# ---------- Virtual environments ----------
+.venv/
+
+# ---------- Secrets ----------
+.env

CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+## 0.2.3
+- Added httpx-level integration tests using MockTransport (no network)
+
+## 0.2.2
+- Fixed pyproject.toml table ordering (project.urls) for editable installs
+
+## 0.2.1
+- Added man page (man/wayparam.1)
+
+## 0.2.0
+- Added output formats (txt/jsonl) and safer diagnostics to stderr
+- Added per-domain stats (optional)
+- Improved HTTP error messages with status/no-status
+- Added basic tests and CI config
+
+## 0.1.0
+- Initial release

CODE_OF_CONDUCT.md

@@ -0,0 +1,3 @@
+# Code of Conduct
+
+Be respectful. Assume good intent. No harassment.

CONTRIBUTING.md

@@ -0,0 +1,16 @@
+# Contributing
+
+## Dev setup
+```bash
+python -m venv .venv
+# Windows: .venv\Scripts\activate
+# macOS/Linux: source .venv/bin/activate
+pip install -U pip
+pip install -e '.[dev]'
+pytest
+ruff check .
+```
+
+## Guidelines
+- Keep stdout machine-readable (URLs / JSONL). Put diagnostics on stderr.
+- Add unit tests for parsing/normalization changes.

README.md

@@ -0,0 +1,232 @@
+# wayparam
+
+**wayparam** is a modern, cross-platform CLI tool to **fetch historical URLs from the Internet Archive Wayback CDX API**, filter out “boring” URLs (static assets), and **normalize query parameters** so you can focus on endpoints that actually matter.
+
+This project is **inspired by ParamSpider** (same overall goal, completely rewritten with a more robust architecture, modern async I/O, better filtering, and production-friendly output behavior).
+
+> OSINT tool: **wayparam does not crawl targets**. It only queries the Wayback CDX API.
+
+---
+
+## Key features
+
+- **Wayback CDX API** URL collection (single domain or list)
+- **Async + concurrency** for speed on multiple domains
+- **Rate limiting** (`--rps`) to be polite with Wayback/CDX
+- **Retry + backoff** and clearer error messages
+- **CDX pagination** (resumeKey) when available
+- Filters “boring” URLs by:
+  - extension blacklist/whitelist
+  - optional path regex exclusion
+- **Canonicalization & normalization**
+  - drop fragments
+  - normalize host/ports
+  - sort parameters
+  - mask parameter values (default placeholder: `FUZZ`)
+  - optional tracking parameter removal (utm_*, gclid, fbclid, …)
+- Output:
+  - per-domain files (default)
+  - **stdout streaming** for pipelines (`--stdout`)
+  - `txt` or `jsonl` output (`--format`)
+
+---
+
+## Installation
+
+### From source (recommended for now)
+
+```bash
+python -m venv .venv
+# Windows: .venv\Scripts\activate
+# macOS/Linux: source .venv/bin/activate
+python -m pip install -U pip
+pip install -e .
+````
+
+### Development install (tests + lint)
+
+```bash
+pip install -e ".[dev]"
+```
+
+---
+
+## Quick start
+
+### 1) Single domain (writes to `results/`)
+
+```bash
+wayparam -d example.com
+```
+
+### 2) List of domains
+
+```bash
+wayparam -l domains.txt
+```
+
+### 3) Stream to stdout (for piping), no files
+
+```bash
+wayparam -d example.com --stdout --no-files
+```
+
+### 4) JSONL output (great for tooling)
+
+```bash
+wayparam -d example.com --stdout --no-files --format jsonl
+```
+
+### 5) Include subdomains + be polite to Wayback
+
+```bash
+wayparam -d example.com --include-subdomains --rps 1 --concurrency 2
+```
+
+### 6) Customize filtering (extensions + path regex)
+
+```bash
+wayparam -d example.com --ext-blacklist ".png,.jpg,.css,.js" --exclude-path-regex "^/static/"
+```
+
+---
+
+## How it works (under the hood)
+
+1. **Input parsing**
+
+   * `-d/--domain` for a single host
+   * `-l/--list` for multiple hosts (one per line, supports comments and basic normalization)
+
+2. **Query the Wayback CDX API**
+
+   * Requests are sent to the CDX endpoint (Wayback Machine)
+   * Uses `matchType=host` by default, or `matchType=domain` when `--include-subdomains` is enabled
+   * Uses pagination (resumeKey) when the API provides it
+
+3. **Filter “boring” URLs**
+
+   * Drops URLs that look like static assets (by extension), with optional whitelist mode
+   * Optional regex filters can exclude paths (e.g., `/static/`, `/assets/`, …)
+
+4. **Canonicalize + normalize**
+
+   * Removes fragments (`#...`)
+   * Normalizes default ports (`:80`, `:443`)
+   * Parses query string and:
+
+     * replaces values with a placeholder (default `FUZZ`)
+     * optionally drops tracking parameters
+     * sorts parameters for stable output
+   * Deduplicates results
+
+5. **Output**
+
+   * By default writes per-domain results into `results/`
+   * `--stdout` streams machine-readable output
+   * Diagnostics (hints, logs, stats) go to **stderr** (safe for pipelines)
+
+---
+
+## Output behavior (important for pipelines)
+
+* **stdout**: only results (URLs or JSONL) when `--stdout` is enabled
+* **stderr**: logs, errors, hints (VPN/proxy), optional stats
+
+This means you can safely do:
+
+```bash
+wayparam -d example.com --stdout --no-files | sort -u > urls.txt
+```
+
+---
+
+## Common options
+
+### Wayback/CDX
+
+* `--include-subdomains`
+* `--from 2019` / `--to 2021` (or full timestamps like `20190101000000`)
+* `--filter statuscode:200` (repeatable)
+* `--no-collapse` (more duplicates, more data)
+
+### Normalization
+
+* `--placeholder X`
+* `--keep-values` (not recommended if you share logs)
+* `--drop-tracking` / `--no-drop-tracking`
+* `--all-urls` (include URLs without query parameters)
+
+### Filtering
+
+* `--ext-blacklist ".png,.jpg,.css,.js"`
+* `--ext-whitelist ".php,.asp,.aspx"`
+* `--exclude-path-regex "regex"` (repeatable)
+
+### Performance / network
+
+* `--concurrency 8`
+* `--rps 1` (recommended when using VPNs / noisy networks)
+* `--timeout 30`
+* `--retries 4`
+* `--proxy http://127.0.0.1:8080`
+
+---
+
+## Troubleshooting: VPN / Proxy issues (Wayback CDX)
+
+If you see errors like “failed after retries” against the CDX endpoint, it often means:
+
+* the VPN/proxy exit node is **blocked** or **rate-limited** by Wayback
+* your VPN does TLS filtering or networking policies that break automated requests
+
+Try:
+
+* disconnecting VPN/proxy and rerunning
+* switching to a different VPN server
+* lowering `--concurrency` and setting `--rps 1`
+
+wayparam will print a **human-readable hint in English** to stderr when it detects this pattern.
+
+---
+
+## Man page
+
+A manual page is included:
+
+```bash
+man ./man/wayparam.1
+```
+
+---
+
+## Testing
+
+Install dev dependencies and run:
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+```
+
+The test suite includes **httpx-level integration tests** using `httpx.MockTransport` (no network).
+
+---
+
+## License
+
+wayparam is **free software** released under the **GNU General Public License v3 (GPLv3)**.
+See the `LICENSE` file for details.
+
+---
+
+## Acknowledgements
+
+* Inspired by **ParamSpider** (same objective: fetch Wayback URLs, filter noise, focus on parameterized endpoints).
+* Thanks to the OSINT / security community for patterns and workflows around URL collection and parameter discovery.
+
+---
+
+## Disclaimer
+
+Use responsibly and lawfully. This tool queries the Internet Archive and does not actively scan targets, but your downstream usage of collected URLs may have legal and ethical implications depending on context.

SECURITY.md

@@ -0,0 +1,5 @@
+# Security Policy
+
+Report security issues via the project issue tracker (private disclosure if possible).
+
+This tool queries the Internet Archive (Wayback CDX). It does not crawl targets.