Skip to content

Commit bea804d

Browse files
tommcdtommcd
committed
feat(pySCG): add comprehensive testing framework for code examples and docs
Add pytest-based testing framework with 765 test cases to validate Python code examples and Markdown documentation structure. Features: - Python validation: syntax, deprecation warnings, expected output - Markdown validation: required sections, tables, section order, code refs - Link validation: internal links, index table, code file references - Smart CI/CD: tests only changed CWE directories on PRs (~1 min) - Full suite: runs all tests on pushes to main (~3 min) - Multi-version: Python 3.9-3.14 testing with tox - Clean reporting: generate_issue_report.py for actionable summaries - Auto cleanup: removes test artifacts after test runs CI/CD Integration: - GitHub Actions workflow with matrix strategy - Lychee link checker (internal links in CI, external available locally) - Coverage reporting to Codecov - Fast execution with uv package manager Documentation: - Comprehensive tests/README.md with usage examples - Updated CONTRIBUTING.md with testing workflow - KNOWN_ISSUES.md documents 44 files with pre-existing issues The framework identifies issues in 44 files (31 documentation files with missing sections/tables/order issues, 13 Python files with timeouts/warnings) documented in KNOWN_ISSUES.md. These pre-existing issues can be addressed in follow-up PRs. All new contributions will be validated automatically. Closes #1010 Signed-off-by: tommcd <[email protected]>
1 parent 2b6b325 commit bea804d

28 files changed

+3898
-0
lines changed

‎.github/workflows/python-tests.yml‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
###############################################
3+
###############################################
4+
## Python Testing Framework GitHub Actions ##
5+
###############################################
6+
###############################################
7+
name: Python Tests
8+
9+
#############################
10+
# Start the job on all push #
11+
#############################
12+
on:
13+
push:
14+
branches: [main]
15+
paths:
16+
- 'docs/Secure-Coding-Guide-for-Python/**'
17+
- '.github/workflows/python-tests.yml'
18+
pull_request:
19+
branches: [main]
20+
paths:
21+
- 'docs/Secure-Coding-Guide-for-Python/**'
22+
- '.github/workflows/python-tests.yml'
23+
workflow_dispatch: # Allow manual trigger for full test suite
24+
25+
###############
26+
# Set the Job #
27+
###############
28+
jobs:
29+
test:
30+
# Name the Job
31+
name: Run Python Tests (Python ${{ matrix.python-version }})
32+
# Set the agent to run on
33+
runs-on: ubuntu-latest
34+
35+
# Matrix strategy for multiple Python versions
36+
strategy:
37+
matrix:
38+
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13', '3.14']
39+
fail-fast: false
40+
41+
##################
42+
# Load all steps #
43+
##################
44+
steps:
45+
##########################
46+
# Checkout the code base #
47+
##########################
48+
- name: Checkout Code
49+
uses: actions/checkout@v4
50+
51+
######################
52+
# Install uv package #
53+
######################
54+
- name: Install uv
55+
uses: astral-sh/setup-uv@v3
56+
with:
57+
enable-cache: true
58+
cache-dependency-glob: "docs/Secure-Coding-Guide-for-Python/pyproject.toml"
59+
60+
########################
61+
# Set up Python version #
62+
########################
63+
- name: Set up Python ${{ matrix.python-version }}
64+
run: uv python install ${{ matrix.python-version }}
65+
66+
##########################
67+
# Install dependencies #
68+
##########################
69+
- name: Install Dependencies
70+
working-directory: docs/Secure-Coding-Guide-for-Python
71+
run: uv sync --group test
72+
73+
##########################
74+
# Get changed files #
75+
##########################
76+
- name: Get Changed Files
77+
id: changed-files
78+
uses: tj-actions/changed-files@v44
79+
with:
80+
files: |
81+
docs/Secure-Coding-Guide-for-Python/CWE-*/**
82+
docs/Secure-Coding-Guide-for-Python/Intro_*/**
83+
84+
##########################
85+
# Run pytest tests #
86+
##########################
87+
- name: Run Tests on Changed Files (PR)
88+
if: github.event_name == 'pull_request' && steps.changed-files.outputs.any_changed == 'true'
89+
working-directory: docs/Secure-Coding-Guide-for-Python
90+
env:
91+
CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
92+
run: |
93+
# Extract unique CWE directories from changed files
94+
CHANGED_DIRS=$(echo "$CHANGED_FILES" | tr ' ' '\n' | grep -E 'CWE-|Intro_' | sed 's|docs/Secure-Coding-Guide-for-Python/||' | sed 's|/.*||' | sort -u | tr '\n' ' ')
95+
96+
if [ -n "$CHANGED_DIRS" ]; then
97+
echo "::notice::Testing changed directories: $CHANGED_DIRS"
98+
# Build pytest -k filter for changed directories
99+
FILTER=$(echo "$CHANGED_DIRS" | sed 's/ / or /g')
100+
uv run pytest tests/ --tb=line -q -k "$FILTER" || {
101+
echo "::error::Tests failed. See details above."
102+
exit 1
103+
}
104+
else
105+
echo "::notice::No CWE directories changed, skipping tests"
106+
fi
107+
108+
- name: Run Full Test Suite (Push to main or manual trigger)
109+
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
110+
working-directory: docs/Secure-Coding-Guide-for-Python
111+
run: |
112+
uv run pytest tests/ --tb=line -q || {
113+
echo "::error::Tests failed. Review the output above for specific issues."
114+
exit 1
115+
}
116+
117+
##########################
118+
# Link checking #
119+
##########################
120+
- name: Link Checking with Lychee
121+
if: matrix.python-version == '3.12'
122+
uses: lycheeverse/lychee-action@v2
123+
with:
124+
args: --offline --verbose --no-progress 'docs/Secure-Coding-Guide-for-Python/**/*.md'
125+
fail: true
126+
127+
##########################
128+
# Upload coverage report #
129+
##########################
130+
- name: Upload Coverage
131+
if: matrix.python-version == '3.12'
132+
uses: codecov/codecov-action@v4
133+
with:
134+
files: ./docs/Secure-Coding-Guide-for-Python/reports/coverage/coverage.xml
135+
flags: python-tests
136+
token: ${{ secrets.CODECOV_TOKEN }}

0 commit comments

Comments
 (0)