Skip to content

Commit c366607

Browse files
crivetimihaicrivetimihai
authored
256 fuzz testing (#760)
* Implement comprehensive fuzz testing automation (#256) - Add property-based testing with Hypothesis for JSON-RPC, JSONPath, and schema validation - Add coverage-guided fuzzing with Atheris for deep code path exploration - Add API endpoint fuzzing with Schemathesis for contract validation - Add security-focused testing for vulnerability discovery (SQL injection, XSS, etc.) - Add complete Makefile automation with fuzz-all, fuzz-quick, fuzz-extended targets - Add optional [fuzz] dependency group in pyproject.toml for clean installation - Add comprehensive reporting with JSON/Markdown outputs and executive summaries - Add complete developer documentation with examples and troubleshooting guides - Exclude fuzz tests from main test suite to prevent auth failures - Found multiple real bugs in JSON-RPC validation during development Signed-off-by: Mihai Criveti <[email protected]> * Update fuzz testing Signed-off-by: Mihai Criveti <[email protected]> * Update fuzz testing Signed-off-by: Mihai Criveti <[email protected]> --------- Signed-off-by: Mihai Criveti <[email protected]>
1 parent 4fc483f commit c366607

19 files changed

+3710
-13
lines changed

‎.github/workflows/pytest.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ jobs:
7575
- name: 🧪 Run pytest
7676
run: |
7777
pytest \
78+
--ignore=tests/fuzz \
7879
--cov=mcpgateway \
7980
--cov-report=xml \
8081
--cov-report=html \

‎.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,11 @@ FIXMEs
4444
*.old
4545
logs/
4646
*.log
47+
48+
# Fuzzing artifacts and reports
49+
reports/
50+
corpus/
51+
tests/fuzz/fuzzers/results/
4752
.venv
4853
mcp.db
4954
public/

‎.pre-commit-config.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -368,7 +368,7 @@ repos:
368368
description: Verifies test files in tests/ directories start with `test_`.
369369
language: python
370370
files: (^|/)tests/.+\.py$
371-
exclude: ^tests/.*/(pages|helpers)/.*\.py$ # Exclude page object and helper files
371+
exclude: ^tests/.*/(pages|helpers|fuzzers|scripts)/.*\.py$ # Exclude page object, helper, fuzzer, and script files
372372
args: [--pytest-test-first] # `test_.*\.py`
373373

374374
# - repo: https://github.com/pycqa/flake8

‎Makefile

Lines changed: 110 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ test:
234234
@test -d "$(VENV_DIR)" || $(MAKE) venv
235235
@/bin/bash -c "source $(VENV_DIR)/bin/activate && \
236236
python3 -m pip install -q pytest pytest-asyncio pytest-cov && \
237-
python3 -m pytest --maxfail=0 --disable-warnings -v"
237+
python3 -m pytest --maxfail=0 --disable-warnings -v --ignore=tests/fuzz"
238238

239239
coverage:
240240
@test -d "$(VENV_DIR)" || $(MAKE) venv
@@ -4355,3 +4355,112 @@ pre-commit-check-headers: ## 🪝 Check headers for pre-commit hooks
43554355
pre-commit-fix-headers: ## 🪝 Fix headers for pre-commit hooks
43564356
@echo "🪝 Fixing headers for pre-commit..."
43574357
@python3 .github/tools/fix_file_headers.py --fix-all
4358+
4359+
# ==============================================================================
4360+
# 🎯 FUZZ TESTING - Automated property-based and security testing
4361+
# ==============================================================================
4362+
# help: 🎯 FUZZ TESTING - Automated property-based and security testing
4363+
# help: fuzz-install - Install fuzzing dependencies (hypothesis, schemathesis, etc.)
4364+
# help: fuzz-all - Run complete fuzzing suite (hypothesis + atheris + api + security)
4365+
# help: fuzz-hypothesis - Run Hypothesis property-based tests for core validation
4366+
# help: fuzz-atheris - Run Atheris coverage-guided fuzzing (requires clang/libfuzzer)
4367+
# help: fuzz-api - Run Schemathesis API fuzzing (requires running server)
4368+
# help: fuzz-restler - Run RESTler API fuzzing instructions (stateful sequences)
4369+
# help: fuzz-restler-auto - Run RESTler via Docker automatically (requires Docker + server)
4370+
# help: fuzz-security - Run security-focused vulnerability tests (SQL injection, XSS, etc.)
4371+
# help: fuzz-quick - Run quick fuzzing for CI/PR validation (50 examples)
4372+
# help: fuzz-extended - Run extended fuzzing for nightly testing (1000+ examples)
4373+
# help: fuzz-report - Generate comprehensive fuzzing reports (JSON + Markdown)
4374+
# help: fuzz-clean - Clean fuzzing artifacts and generated reports
4375+
4376+
fuzz-install: ## 🔧 Install all fuzzing dependencies
4377+
@echo "🔧 Installing fuzzing dependencies..."
4378+
@test -d "$(VENV_DIR)" || $(MAKE) venv
4379+
@/bin/bash -c "source $(VENV_DIR)/bin/activate && \
4380+
pip install -e .[fuzz]"
4381+
@echo "✅ Fuzzing tools installed"
4382+
4383+
fuzz-hypothesis: fuzz-install ## 🧪 Run Hypothesis property-based tests
4384+
@echo "🧪 Running Hypothesis property-based tests..."
4385+
@/bin/bash -c "source $(VENV_DIR)/bin/activate && \
4386+
python3 -m pytest tests/fuzz/ -v \
4387+
--hypothesis-show-statistics \
4388+
--hypothesis-profile=dev \
4389+
-k 'not (test_sql_injection or test_xss_prevention or test_integer_overflow or test_rate_limiting)' \
4390+
|| true"
4391+
4392+
fuzz-atheris: ## 🎭 Run Atheris coverage-guided fuzzing
4393+
@echo "🎭 Running Atheris coverage-guided fuzzing..."
4394+
@echo "⚠️ Atheris requires clang/libfuzzer - skipping for now"
4395+
@mkdir -p corpus tests/fuzz/fuzzers/results reports
4396+
@echo "✅ Atheris setup completed (requires manual clang installation)"
4397+
4398+
fuzz-api: ## 🌐 Run Schemathesis API fuzzing
4399+
@echo "🌐 Running Schemathesis API fuzzing..."
4400+
@echo "⚠️ API fuzzing requires running server - skipping automated server start"
4401+
@echo "💡 To run manually:"
4402+
@echo " 1. make dev (in separate terminal)"
4403+
@echo " 2. source $(VENV_DIR)/bin/activate && schemathesis run http://localhost:4444/openapi.json --checks all --auth admin:changeme"
4404+
@mkdir -p reports
4405+
@echo "✅ API fuzzing setup completed"
4406+
4407+
fuzz-restler: ## 🧪 Run RESTler API fuzzing (instructions)
4408+
@echo "🧪 Running RESTler API fuzzing (via Docker or local install)..."
4409+
@echo "⚠️ RESTler is not installed by default; using instructions only"
4410+
@mkdir -p reports/restler
4411+
@echo "💡 To run with Docker (recommended):"
4412+
@echo " 1) make dev # in another terminal"
4413+
@echo " 2) curl -sSf http://localhost:4444/openapi.json -o reports/restler/openapi.json"
4414+
@echo " 3) docker run --rm -v $$PWD/reports/restler:/workspace ghcr.io/microsoft/restler restler compile --api_spec /workspace/openapi.json"
4415+
@echo " 4) docker run --rm -v $$PWD/reports/restler:/workspace ghcr.io/microsoft/restler restler test --grammar_dir /workspace/Compile --no_ssl --time_budget 5"
4416+
@echo " # Artifacts will be under reports/restler"
4417+
@echo "💡 To run with local install (RESTLER_HOME):"
4418+
@echo " export RESTLER_HOME=/path/to/restler && \\"
4419+
@echo " $$RESTLER_HOME/restler compile --api_spec reports/restler/openapi.json && \\"
4420+
@echo " $$RESTLER_HOME/restler test --grammar_dir Compile --no_ssl --time_budget 5"
4421+
@echo "✅ RESTler instructions emitted"
4422+
4423+
fuzz-restler-auto: ## 🤖 Run RESTler via Docker automatically (server must be running)
4424+
@echo "🤖 Running RESTler via Docker against a running server..."
4425+
@if ! command -v docker >/dev/null 2>&1; then \
4426+
echo "🐳 Docker not found; skipping RESTler fuzzing (fuzz-restler-auto)."; \
4427+
echo " Hint: Install Docker or use 'make fuzz-restler' for manual steps."; \
4428+
exit 0; \
4429+
fi
4430+
@test -d "$(VENV_DIR)" || $(MAKE) venv
4431+
@/bin/bash -c "source $(VENV_DIR)/bin/activate && \
4432+
python3 tests/fuzz/scripts/run_restler_docker.py"
4433+
4434+
fuzz-security: fuzz-install ## 🔐 Run security-focused fuzzing tests
4435+
@echo "🔐 Running security-focused fuzzing tests..."
4436+
@echo "⚠️ Security tests require running application with auth - they may fail in isolation"
4437+
@/bin/bash -c "source $(VENV_DIR)/bin/activate && \
4438+
HYPOTHESIS_PROFILE=dev python3 -m pytest tests/fuzz/test_security_fuzz.py -v \
4439+
|| true"
4440+
4441+
fuzz-quick: fuzz-install ## ⚡ Run quick fuzzing for CI
4442+
@echo "⚡ Running quick fuzzing for CI..."
4443+
@/bin/bash -c "source $(VENV_DIR)/bin/activate && \
4444+
HYPOTHESIS_PROFILE=ci python3 -m pytest tests/fuzz/ -v \
4445+
-k 'not (test_very_large or test_sql_injection or test_xss_prevention or test_integer_overflow or test_rate_limiting)' \
4446+
|| true"
4447+
4448+
fuzz-extended: fuzz-install ## 🕐 Run extended fuzzing for nightly runs
4449+
@echo "🕐 Running extended fuzzing suite..."
4450+
@/bin/bash -c "source $(VENV_DIR)/bin/activate && \
4451+
HYPOTHESIS_PROFILE=thorough python3 -m pytest tests/fuzz/ -v \
4452+
--durations=20 || true"
4453+
4454+
fuzz-report: fuzz-install ## 📊 Generate fuzzing report
4455+
@echo "📊 Generating fuzzing report..."
4456+
@mkdir -p reports
4457+
@/bin/bash -c "source $(VENV_DIR)/bin/activate && \
4458+
python3 tests/fuzz/scripts/generate_fuzz_report.py"
4459+
4460+
fuzz-clean: ## 🧹 Clean fuzzing artifacts
4461+
@echo "🧹 Cleaning fuzzing artifacts..."
4462+
@rm -rf corpus/ tests/fuzz/fuzzers/results/ reports/schemathesis-report.json
4463+
@rm -f reports/fuzz-report.json
4464+
4465+
fuzz-all: fuzz-hypothesis fuzz-atheris fuzz-api fuzz-security fuzz-report ## 🎯 Run complete fuzzing suite
4466+
@echo "🎯 Complete fuzzing suite finished"

0 commit comments

Comments
 (0)