feat(azure): add vault parallelization in keyvault service
#9876
Conversation
github-actions
bot
added
the
provider/azure
|
✅ All necessary |
|
✅ Conflict Markers Resolved All conflict markers have been successfully resolved in this pull request. |
…o PROWLER-737-clone-scans-running-for-more-than-24-hours-in-production-sdk-root-cause
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
...services/keyvault/keyvault_rbac_secret_expiration_set/keyvault_rbac_secret_expiration_set.py
Fixed
Show fixed
Hide fixed
| result = future.result() | ||
| if result is not None: | ||
| results.append(result) | ||
| except Exception: |
Check notice
Code scanning / CodeQL
Empty except Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 9 hours ago
In general, empty except blocks should be replaced with handling that at minimum logs the exception, and optionally re-raises it or aggregates it, depending on how critical the failure is. In this case, we want to preserve the current behavior of continuing to process other futures while improving observability.
The best minimal-impact fix is to log any exception raised by future.result() using the existing logger, similar to how __set_clients__ logs errors. We should not re-raise, because that would change the function’s behavior from “best effort, skip failures” to “fail entire call on first error”. Instead, we log the exception (optionally including the associated item and traceback line) and keep skipping that result, preserving the public behavior while making failures visible.
Concretely, in prowler/providers/azure/lib/service/service.py:
- Inside
AzureService.__threading_call__, replace theexcept Exception: passblock withexcept Exception as error:and alogger.error(...)call. We can mirror the formatting used in__set_clients__, and additionally include theitemassociated with the future (available viafutures[future]). - No new imports are needed;
loggeris already imported at the top of the file.
| @@ -38,8 +38,11 @@ | ||
| result = future.result() | ||
| if result is not None: | ||
| results.append(result) | ||
| except Exception: | ||
| pass | ||
| except Exception as error: | ||
| logger.error( | ||
| f"{error.__class__.__name__}[{error.__traceback__.tb_lineno}] " | ||
| f"while processing item {futures.get(future)!r}: {error}" | ||
| ) | ||
|
|
||
| return results | ||
|
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #9876 +/- ##
==========================================
- Coverage 86.60% 86.08% -0.52%
==========================================
Files 222 222
Lines 5645 5686 +41
==========================================
+ Hits 4889 4895 +6
- Misses 756 791 +35
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
🔒 Container Security ScanImage: 📊 Vulnerability Summary
3 package(s) affected
|
Context
Large environments were experiencing extremely slow scan times of approximately for Azure KeyVault checks. The root cause was sequential processing of vaults and their contents (keys, secrets, monitor settings), compounded by Azure Management API pagination returning only 2-3 items per page.
Description
This PR introduces vault-level parallelization to the
Azure KeyVault service, dramatically improving scan performance for large-scale environments.Changes:
prowler/providers/azure/lib/service/service.py__threading_call__method to the baseAzureServiceclassMAX_WORKERS=10thread pool for parallel processingprowler/providers/azure/services/keyvault/keyvault_service.py_get_key_vaults()to process vaults in parallel using threading_callThreadPoolExecutor(max_workers=3)_process_single_keyvault()for thread-safe vault processingprowler/providers/azure/services/keyvault/keyvault_rbac_secret_expiration_set/keyvault_rbac_secret_expiration_set.pyCheck_Report_Azureobject across secrets (causing incorrect status propagation) and lackingPASSfindings.Steps to review
service.py:30-60)__threading_call__handles exceptions gracefullyMAX_WORKERS=10) is reasonablekeyvault_service.py)_process_single_keyvault()is thread-safe (no shared mutable state)keyvault_rbac_secret_expiration_set.py)Check_Report_Azureinstanceresource=secret(notresource=keyvault) for proper attributionAzureenvironment with multiple vaults"Starting threads for...", "Completed ... in X.XXs"Checklist
Community Checklist
SDK/CLI
UI
API
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.