Skip to content

Optimized UserEvent.decay to resolve N+1 queries. #3506

Open
ayushgupta704 wants to merge 1 commit intointelowlproject:developfrom
ayushgupta704:performance/userevent-decay-optimization
Open

Optimized UserEvent.decay to resolve N+1 queries. #3506
ayushgupta704 wants to merge 1 commit intointelowlproject:developfrom
ayushgupta704:performance/userevent-decay-optimization

Conversation

@ayushgupta704
Copy link

@ayushgupta704 ayushgupta704 commented Mar 21, 2026

Description

This PR resolves the scalability bottleneck in UserEventQuerySet.decay(). Previously, the decay process followed a 3N+1 query pattern, triggering sequential database round-trips for every event in the loop (fetching the DataModel, updating its reliability, and saving the UserEvent).
For a batch of 100 events, this resulted in 301 database queries. With this optimization, the same workload now takes only 6 queries O(1) constant time).

Key Architectural Changes:

  1. Batch Fetching: Implemented conditional select_related/prefetch_related on the data_model
    relationship to collapse $N$ fetch queries into 1.
  2. Bulk Updates: Replaced row-level .save() calls with Django's bulk_update().
  3. Deduplication: Added logic to deduplicate DataModel updates, ensuring each record is updated only
    once per batch with its final reliability score.
  4. Transactional Integrity: Wrapped the entire operation in transaction.atomic() to prevent
    inconsistent states during partial failures.
  5. Regression Testing: Added a new test case test_decay_performance that asserts the query count remains $\leq 10$ regardless of the number of events.

Type of change

  • Bug fix (non-breaking change which fixes an issue).

Checklist

  • I have read and understood the rules about [how to Contribute
    (https://intelowlproject.github.io/docs/IntelOwl/contribute/) to this project
  • I have inserted the copyright banner at the start of the file.
  • Linters (Ruff) gave 0 errors.

@ayushgupta704 ayushgupta704 changed the title [Performance] Optimized UserEvent.decay to resolve N+1 queries. Close… Optimized UserEvent.decay to resolve N+1 queries. Mar 21, 2026
@ayushgupta704 ayushgupta704 force-pushed the performance/userevent-decay-optimization branch from 276828f to d0ddef5 Compare March 21, 2026 05:36
@ayushgupta704
Copy link
Author

ayushgupta704 commented Mar 21, 2026

Hey @mlodic, I've updated this PR to be a single clean commit on top of the latest develop.

  • In the Before screenshot, you can see how processing just 100 events was hitting the database 301 times it was a classic 3N+1 sequential loop that would eventually cause major lag as the number of observables grows.
Image
  • I’ve implemented the fix using set-based operations, and the benchmark results confirm a massive improvement. I ran a fresh test with 100 events, and the query count dropped from 301 queries to just 6.
Image

hopefully, this helps keep things running smoothly as the platform scales! Let me know if you have any feedback or if you'd like me to adjust the implementation Ready for review Thanks.
closes #3460

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant