fix(python/sast): add Python 3.9 fallback for stdlib module discovery #464
+68
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The stdlib registry generation was failing for Python 3.9 because sys.stdlib_module_names is only available in Python 3.10+. This commit adds a fallback mechanism that uses pkgutil and sysconfig to discover stdlib modules for Python 3.9. The fallback: - Collects built-in modules from sys.builtin_module_names - Discovers modules in stdlib path using pkgutil.iter_modules() - Discovers modules in platstdlib path for platform-specific modules This allows the stdlib-r2-upload workflow to successfully generate registries for all Python versions 3.9-3.14. Fixes the RuntimeError: "sys.stdlib_module_names not available (Python 3.10+ required)" that was blocking the R2 upload workflow.
…d support Enhanced the Python 3.9 fallback mechanism to be more comprehensive: 1. **lib-dynload / DLLs support**: Now discovers compiled extension modules from lib-dynload (Unix/Linux) and DLLs (Windows) directories. This fixes missing modules like sqlite3, ssl, _asyncio, etc. 2. **Error handling**: Added try-except blocks around pkgutil.iter_modules() calls to handle edge cases gracefully with warning messages. 3. **Documentation**: Added comprehensive docstring explaining the discovery strategy for both Python 3.10+ and 3.9 fallback. 4. **Cross-platform**: Works on Windows, Linux, and macOS by checking both lib-dynload and DLLs directories. Before: 261 modules discovered (missing compiled extensions) After: 306 modules discovered (includes all compiled extensions) This ensures feature parity between Python 3.9 and 3.10+ for stdlib registry generation.
…ibility Code quality and maintainability improvements: 1. **Moved imports to module level**: pkgutil, sysconfig, and os are now imported at the top instead of inside functions, following PEP 8. 2. **Added Python version check**: Script now validates it's running on Python 3.9+ at startup and provides a clear error message if not. 3. **Enhanced documentation**: Added "Minimum Python version: 3.9" note in module docstring for clarity. 4. **Verbose detection method logging**: When --verbose is used with --all, the script now logs whether it's using sys.stdlib_module_names (3.10+) or the pkgutil fallback (3.9). 5. **Expanded platform-specific module list**: Added comprehensive list of Unix-specific (grp, pwd, resource, syslog, termios, tty), macOS-specific (_scproxy), and optional modules (readline, dbm, gdbm, ossaudiodev, spwd) to prevent false positives in unexpected failure detection. These changes improve code maintainability and provide better debugging information when the script runs across different Python versions (3.9-3.14) and platforms (Windows, Linux, macOS). No functional changes to the core module discovery logic.
SafeDep Report Summary
No dependency changes detected. Nothing to scan. This report is generated by SafeDep Github App |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #464 +/- ##
=======================================
Coverage 81.64% 81.64%
=======================================
Files 94 94
Lines 10513 10513
=======================================
Hits 8583 8583
Misses 1615 1615
Partials 315 315 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The stdlib registry generation was failing for Python 3.9 because
sys.stdlib_module_names is only available in Python 3.10+.
This commit adds a fallback mechanism that uses pkgutil and sysconfig
to discover stdlib modules for Python 3.9. The fallback:
This allows the stdlib-r2-upload workflow to successfully generate
registries for all Python versions 3.9-3.14.
Fixes the RuntimeError: "sys.stdlib_module_names not available
(Python 3.10+ required)" that was blocking the R2 upload workflow.