Commit 69d8bb4
feat(ruleset): add Python security rules with remote ruleset CLI support (#466)
* feat: reorganize rules into bundle-based directory structure
- Create bundle directories: security, best-practice, performance, networking
- Move 47 Python rule files to appropriate bundles based on severity
- Add manifest.json files for global and per-category metadata
Directory structure:
rules/
├── manifest.json (global registry)
├── docker/
│ ├── manifest.json
│ ├── security/ (5 rules: CRITICAL/HIGH)
│ ├── best-practice/ (30 rules: MEDIUM/LOW/INFO)
│ └── performance/ (2 rules: caching-related)
└── docker-compose/
├── manifest.json
├── security/ (6 rules: CRITICAL/HIGH)
└── networking/ (4 rules: MEDIUM/LOW)
Backwards compatibility:
- RuleLoader still walks directories recursively (tested)
- Rule decorators unchanged
- All existing tests pass (go test ./dsl/... ✓)
- File history preserved via git mv
Enables:
- PR-02: R2 bundle upload by directory
- PR-04: Website submodule integration
- User-friendly ruleset names: docker/security
Total files: 47 (before) → 47 (after) ✓
Refs: pathfinder-rules-architecture tech spec
Implements: PR-01-reorganize-rules.md
* feat: add R2 upload tooling and GitHub workflow for rule distribution
This commit implements PR-02, creating infrastructure to process rule bundles
into distributable zip files with checksums and automatically upload them to
Cloudflare R2 CDN.
**New files:**
- tools/process_rules_for_r2.py: Python script to process rule bundles
* Walks bundle directories and creates zip files per bundle
* Calculates SHA256 checksums for security verification
* Updates manifests with computed metadata (file_count, zip_size, checksum, download_url)
* Supports --dry-run mode for testing
* Generates dist/rules/ output structure ready for R2 upload
- tools/upload_rules_to_r2.sh: Bash script for R2 upload
* Uploads zip bundles to s3://code-pathfinder-assets/rules/
* Sets appropriate cache headers (24h for zips, 1h for manifests)
* Verifies uploads by checking manifest accessibility
* Ubuntu/GitHub Actions compatible (uses portable path operations)
* Supports non-interactive mode via UPLOAD_CONFIRMED=yes
- .github/workflows/deploy-rules.yml: GitHub Actions workflow
* Triggers on rule changes (rules/**/*.py, rules/**/manifest.json)
* Processes rules and uploads to R2 on every merge to main
* Supports manual trigger via workflow_dispatch
* Uploads artifacts for debugging
* Placeholder for Vercel deploy hook (PR-07)
**Technical details:**
- Script uses parameter expansion (${path#$prefix}) instead of realpath for better portability
- Uses -mindepth 2 to correctly filter category manifests vs global manifest
- Proper IFS handling and quoting for files with spaces
- set -euo pipefail for strict error handling in bash
- SHA256 checksums written in standard format for verification
**Output structure:**
dist/rules/
├── manifest.json (global manifest with categories_info and base_url)
├── docker/
│ ├── manifest.json (bundles with checksum/file_count metadata)
│ ├── security.zip (5 rules, 8.0 KB)
│ ├── security.zip.sha256
│ ├── best-practice.zip (30 rules, 30.7 KB)
│ ├── best-practice.zip.sha256
│ ├── performance.zip (2 rules, 6.3 KB)
│ └── performance.zip.sha256
└── docker-compose/
├── manifest.json
├── security.zip (6 rules, 13.9 KB)
├── security.zip.sha256
├── networking.zip (4 rules, 7.5 KB)
└── networking.zip.sha256
**Integration:**
- Depends on: PR-01 (bundle directory structure)
- Enables: PR-03 (CLI remote download), PR-07 (Vercel deploy)
- Tested: Dry-run mode, actual processing, checksum validation
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat: add CLI remote ruleset download with multi-source support
This commit implements PR-03, enabling remote ruleset downloads from R2 with
support for multiple rulesets and merging with local rules for maximum flexibility.
**New ruleset package** (sast-engine/ruleset/):
- types.go: Core data structures (RulesetSpec, Manifest, Bundle, CacheEntry)
- resolver.go: Ruleset spec parsing (docker/security → Category + Bundle)
- manifest.go: HTTP manifest loading from R2 with timeout handling
- cache.go: Local filesystem cache with TTL, checksum validation, SHA256 verification
- downloader.go: HTTP download with retry logic, zip extraction, zip-slip prevention
**Enhanced scan command** (sast-engine/cmd/scan.go):
- NEW: --ruleset flag (can be specified multiple times for multi-source)
- NEW: --refresh-rules flag to force cache invalidation
- NEW: prepareRules() merges local + remote rules into temp directory
**Features:**
- Multiple ruleset support + local/remote mixing
- 24h cache with checksum validation
- Zip-slip prevention + SHA256 verification
- Retry logic with exponential backoff
- Platform-specific cache directories
- Automatic cleanup of temp directories
**Test coverage: 79%** (24/24 tests passing)
- resolver_test.go, manifest_test.go, cache_test.go, downloader_test.go
**Quality:** ✅ buildGo ✅ testGo 1 parent 225bb8d commit 69d8bb4
File tree
148 files changed
+5651
-599
lines changed- .github/workflows
- python-sdk
- codepathfinder
- cli
- examples
- rules
- python
- deserialization
- django
- flask
- injection
- tests
- rules
- docker-compose
- networking
- security
- docker
- best-practice
- performance
- security
- sast-engine
- cmd
- ruleset
- test/fixtures
- clean_project
- rules
- vulnerable_project
- tools
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
148 files changed
+5651
-599
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | | - | |
| 12 | + | |
| 13 | + | |
12 | 14 | | |
13 | 15 | | |
14 | 16 | | |
| |||
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
31 | | - | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
32 | 37 | | |
33 | 38 | | |
34 | 39 | | |
35 | 40 | | |
36 | 41 | | |
37 | 42 | | |
38 | | - | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
39 | 51 | | |
40 | 52 | | |
41 | 53 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
| 72 | + | |
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
79 | | - | |
| 79 | + | |
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
| |||
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
89 | | - | |
90 | | - | |
91 | | - | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | | - | |
102 | | - | |
| 101 | + | |
| 102 | + | |
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
110 | | - | |
| 110 | + | |
111 | 111 | | |
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
116 | | - | |
| 116 | + | |
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
131 | | - | |
| 131 | + | |
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
139 | | - | |
| 139 | + | |
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
| |||
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
156 | | - | |
| 156 | + | |
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
160 | | - | |
161 | | - | |
| 160 | + | |
| 161 | + | |
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
| |||
192 | 192 | | |
193 | 193 | | |
194 | 194 | | |
195 | | - | |
| 195 | + | |
196 | 196 | | |
197 | 197 | | |
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
201 | 201 | | |
202 | | - | |
| 202 | + | |
203 | 203 | | |
204 | 204 | | |
205 | 205 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
| 25 | + | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| |||
0 commit comments