Skip to content

perf(api): replace JOINs with pre-check in threat score aggregation query#10394

Merged
josema-xyz merged 3 commits intomasterfrom
PROWLER-1237-optimize-threat-score-aggregation-query-to-eliminate-unnecessary-joi-ns
Mar 19, 2026
Merged

perf(api): replace JOINs with pre-check in threat score aggregation query#10394
josema-xyz merged 3 commits intomasterfrom
PROWLER-1237-optimize-threat-score-aggregation-query-to-eliminate-unnecessary-joi-ns

Conversation

@josema-xyz
Copy link
Contributor

Description

Eliminate 3 unnecessary JOINs (resource_finding_mappings, resources, providers) from the threat score aggregation query by moving the provider is_deleted check to a separate pre-check. Also removes COUNT(DISTINCT) since the M2M traversal that could produce duplicate rows is gone.

Steps to review

Run tests and run a local scan from the UI.

Checklist

API

  • All issue/task requirements work as expected on the API
  • Endpoint response output (if applicable)
  • EXPLAIN ANALYZE output for new/modified queries or indexes (if applicable)
  • Performance test results (if applicable)
  • Any other relevant evidence of the implementation (if applicable)
  • Verify if API specs need to be regenerated.
  • Check if version updates are required (e.g., specs, Poetry, etc.).
  • Ensure new entries are added to CHANGELOG.md, if applicable.

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@josema-xyz josema-xyz requested a review from a team as a code owner March 19, 2026 12:10
Copilot AI review requested due to automatic review settings March 19, 2026 12:10
@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2026

⚠️ Changes detected in the following folders without a corresponding update to the CHANGELOG.md:

  • prowler

Please add an entry to the corresponding CHANGELOG.md file to maintain a clear history of changes.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2026

Conflict Markers Resolved

All conflict markers have been successfully resolved in this pull request.

jfagoagas
jfagoagas previously approved these changes Mar 19, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to improve performance of the ThreatScore requirement-statistics aggregation by avoiding expensive JOINs in the main aggregation query and removing COUNT(DISTINCT) now that the query is intended to avoid row-multiplication sources.

Changes:

  • Adds a pre-check to short-circuit aggregation when the scan’s provider is deleted, allowing removal of provider/resource JOIN filters in the main query.
  • Simplifies the aggregation Count() calls by removing distinct=True.
  • Adds an API changelog entry documenting the query optimization.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
api/src/backend/tasks/jobs/threatscore_utils.py Refactors the ORM aggregation to remove resource/provider JOIN filters and adds a provider-deleted pre-check.
api/CHANGELOG.md Documents the ThreatScore aggregation query performance change in an UNRELEASED section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2026

🔒 Container Security Scan

Image: prowler-api:e243bdd
Last scan: 2026-03-19 16:15:38 UTC

📊 Vulnerability Summary

Severity Count
🔴 Critical 5
Total 5

4 package(s) affected

⚠️ Action Required

Critical severity vulnerabilities detected. These should be addressed before merging:

  • Review the detailed scan results
  • Update affected packages to patched versions
  • Consider using a different base image if updates are unavailable

📋 Resources:

@codecov
Copy link

codecov bot commented Mar 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.37%. Comparing base (0f2fdcf) to head (6ae0c7d).
⚠️ Report is 10 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #10394       +/-   ##
===========================================
+ Coverage   56.86%   93.37%   +36.50%     
===========================================
  Files          87      219      +132     
  Lines        2847    30415    +27568     
===========================================
+ Hits         1619    28399    +26780     
- Misses       1228     2016      +788     
Flag Coverage Δ
api 93.37% <100.00%> (?)
prowler-py3.10-oraclecloud ?
prowler-py3.11-oraclecloud ?
prowler-py3.12-oraclecloud ?
prowler-py3.9-oraclecloud ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
prowler ∅ <ø> (∅)
api 93.37% <100.00%> (∅)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jfagoagas
jfagoagas previously approved these changes Mar 19, 2026
@josema-xyz josema-xyz added the backport-to-v5.21 Backport PR to the v5.21 branch label Mar 19, 2026
…eted-provider pre-check - Modify api/CHANGELOG.md entry
@josema-xyz josema-xyz merged commit 872e6e2 into master Mar 19, 2026
38 checks passed
@josema-xyz josema-xyz deleted the PROWLER-1237-optimize-threat-score-aggregation-query-to-eliminate-unnecessary-joi-ns branch March 19, 2026 16:30
@prowler-bot prowler-bot added the was-backported The PR was successfully backported to the target branch label Mar 19, 2026
@prowler-bot
Copy link
Collaborator

💚 All backports created successfully

Status Branch Result
v5.21

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v5.21 Backport PR to the v5.21 branch component/api was-backported The PR was successfully backported to the target branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants