pySCG: Add a testing framework for Python code examples and documentation

[tommcd]

Summary

Add a pytest-based testing framework to validate Python code examples and Markdown documentation structure across the Secure Coding Guide for Python.

Motivation

Currently, there is no automated validation of:

• Python code syntax and deprecation warnings in examples
• Markdown documentation structure conformance to the template
• Internal links and code file references
• Consistency across 100+ CWE articles

This leads to potential issues with broken examples, missing documentation sections, and broken links that could confuse contributors and readers.

Solution

Implement a comprehensive testing framework that validates:

1. Python Code Quality

• Syntax validation using ast.parse()
• Deprecation warning detection via subprocess execution
• Expected output validation against documented examples
• Support for expected failure markers (intentional issues)
• Automatic cleanup of test artifacts

2. Markdown Documentation Structure

• Required sections validation (title, introduction, Non-Compliant, Compliant, Automated Detection, Related Guidelines, Bibliography)
• Code file reference validation (compliant01.py, noncompliant01.py exist)
• Table structure validation (Automated Detection, Related Guidelines, Bibliography)
• Section order validation

3. Link Validation

• Internal link validation in README.md files
• Index table link validation
• Integration with lychee for comprehensive link checking (internal only in CI, external available locally)

4. CI/CD Integration

• GitHub Actions workflow for automated testing on PRs
• Smart testing: Only tests changed CWE directories on PRs (fast, ~1 min)
• Full suite: Runs all tests on pushes to main (~3 min)
• Multi-version testing (Python 3.9-3.14)
• Coverage reporting to Codecov
• Fast execution with uv package manager

5. Local Development Support

• Easy setup with uv sync --group test
• Multi-version testing with tox
• Parallel execution support with pytest-xdist
• Clean issue reporting with generate_issue_report.py
• Comprehensive documentation in tests/README.md

Implementation Details

• Uses pytest with parametrized tests for all Python and Markdown files
• Organized test utilities in tests/utils/ for maintainability
• PEP 735 dependency groups for clean dependency management
• Automatic cleanup of test artifacts (database files, temp files, etc.)
• Cross-platform Python-based issue report generator

Benefits

• Ensures all code examples are syntactically valid and run without warnings
• Validates documentation consistency across all CWE articles
• Catches broken links and missing files before merge
• Provides clear, actionable error messages for contributors
• Reduces review burden by automating quality checks
• Supports Python 3.9-3.14 compatibility testing
• Fast CI/CD with smart testing (only changed files on PRs)

Current State

The framework identifies issues in 44 files (documented in KNOWN_ISSUES.md):

• 31 Documentation files with issues:
  • Missing required sections (Bibliography, Automated Detection, etc.)
  • Missing required tables
  • Section order issues
  • Missing code file references
• 13 Python code files with issues:
  • 11 files with execution timeouts (intentional infinite loops/blocking operations)
  • 1 file with DeprecationWarning
  • 1 file with output mismatch

These pre-existing issues can be addressed in follow-up PRs. The testing framework will prevent new issues from being introduced.

Files Added/Modified

New Files:

• .github/workflows/python-tests.yml - CI/CD workflow with smart testing
• docs/Secure-Coding-Guide-for-Python/tests/ - Complete test framework
• docs/Secure-Coding-Guide-for-Python/pyproject.toml - Project configuration
• docs/Secure-Coding-Guide-for-Python/tox.ini - Multi-version testing config
• docs/Secure-Coding-Guide-for-Python/.gitignore - Ignore test artifacts
• docs/Secure-Coding-Guide-for-Python/generate_test_summary.sh - Summary script
• docs/Secure-Coding-Guide-for-Python/KNOWN_ISSUES.md - Reference of existing issues

Modified Files:

• docs/Secure-Coding-Guide-for-Python/CONTRIBUTING.md - Added testing instructions
• Fixed typo: Renamed compliants02.py → compliant02.py

Testing

All 765 tests collected, 531 passed, 80 failures (pre-existing issues), 154 skipped.
Tests run successfully across Python 3.9-3.14 in CI/CD.

[tommcd]

Update: Added Inlined Code Validation Test

Added a critical new test that validates inlined code in README files matches the actual Python files.

What it validates:

• README files contain inlined code blocks that reference Python files like:
  markdown *[noncompliant01.py](noncompliant01.py):* \\\python ... code ... \\\
• The inlined code must match the actual file content (after stripping SPDX headers and test markers)

What it catches:

• Outdated documentation: Code file updated but README not updated
• SPDX headers in docs: Headers incorrectly included in inlined code
• Test markers in docs: EXPECTED_TIMEOUT/EXPECTED_FAILURE appearing in documentation

Current findings:

• Identified 18 files with inlined code mismatches (pre-existing issues)
• These are real documentation problems documented in KNOWN_ISSUES.md
• Will be fixed in follow-up PRs

This ensures documentation stays in sync with actual code, preventing documentation drift.