fix: improve cardinality many relationship fetch #476
Conversation
|
""" WalkthroughThe Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant RelationshipManager
participant Client
User->>RelationshipManager: fetch()
RelationshipManager->>RelationshipManager: Validate each peer (id, typename)
RelationshipManager->>RelationshipManager: Group peer IDs by typename
loop for each typename
RelationshipManager->>Client: filters(kind, ids, populate_store, branch, parallel=True, order=Order())
end
RelationshipManager-->>User: return (after batch fetch)
Estimated code review effort2 (~15 minutes) 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
infrahub_sdk/node/relationship.py (3)
161-173: Excellent performance optimization with robust validation.The batched fetching approach effectively replaces individual peer fetch operations with grouped client.filters() calls, which should significantly improve performance for high-cardinality relationships as intended.
The implementation correctly:
- Groups peer IDs by typename to minimize API calls
- Validates required peer fields before processing
- Leverages parallel execution and disabled ordering for optimization
Consider enhancing the error message to include peer identification for easier debugging:
- raise Error("Unable to fetch the peer, id and/or typename are not defined") + raise Error(f"Unable to fetch peer at index {self.peers.index(peer)}: id and/or typename are not defined")
277-289: Consistent and well-implemented synchronous counterpart.The sync implementation mirrors the async version perfectly, maintaining consistency between both execution contexts. The batched fetching logic is identical, ensuring both async and sync users benefit equally from the performance improvements.
The same error message enhancement suggestion applies here for consistency:
- raise Error("Unable to fetch the peer, id and/or typename are not defined") + raise Error(f"Unable to fetch peer at index {self.peers.index(peer)}: id and/or typename are not defined")
146-173: Outstanding refactoring that delivers on performance objectives.This implementation successfully transforms individual peer fetch operations into efficient batch operations, directly addressing the PR's goal of improving high-cardinality relationship fetching performance.
Key architectural improvements:
- Batch optimization: Groups API calls by typename, reducing network overhead
- Parallel execution: Enables concurrent processing with
parallel=True- Query optimization: Disables unnecessary ordering with
Order(disable=True)- Consistent implementation: Identical logic across async/sync variants ensures uniform behavior
The refactoring maintains backward compatibility while delivering significant performance gains (17.5s → 9.4s as noted in PR objectives). The validation logic ensures data integrity, and the grouped approach scales much better with relationship cardinality.
Also applies to: 262-289
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
infrahub_sdk/node/relationship.py(3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: integration-tests-latest-infrahub
- GitHub Check: unit-tests (3.10)
- GitHub Check: unit-tests (3.12)
- GitHub Check: unit-tests (3.13)
- GitHub Check: unit-tests (3.11)
- GitHub Check: unit-tests (3.9)
- GitHub Check: Cloudflare Pages
🔇 Additional comments (1)
infrahub_sdk/node/relationship.py (1)
7-7: LGTM! Import additions support the refactored fetch logic.The new imports for
ErrorandOrderare correctly added and properly used in the updated fetch methods.Also applies to: 10-10
Codecov ReportAttention: Patch coverage is
@@ Coverage Diff @@
## stable #476 +/- ##
==========================================
+ Coverage 75.63% 75.68% +0.04%
==========================================
Files 100 100
Lines 8767 8789 +22
Branches 1714 1722 +8
==========================================
+ Hits 6631 6652 +21
Misses 1660 1660
- Partials 476 477 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
Signed-off-by: Fatih Acar <[email protected]>
fatih-acar
force-pushed
the
fac-fix-many-rels-fetch
branch
from
6c01aee to
519b99b
Compare
Deploying infrahub-sdk-python with
|
| Latest commit: |
95a7150
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://68d91c11.infrahub-sdk-python.pages.dev |
| Branch Preview URL: | https://fac-fix-many-rels-fetch.infrahub-sdk-python.pages.dev |
Signed-off-by: Fatih Acar <[email protected]>
| await peer.fetch() # type: ignore[misc] | ||
| if not peer.id or not peer.typename: | ||
| raise Error("Unable to fetch the peer, id and/or typename are not defined") | ||
| if peer.typename not in ids_per_kind_map: |
There was a problem hiding this comment.
with the defaultdict you can just do
ids_per_kind_map[peer.typename].append(peer.id)
There was a problem hiding this comment.
Thanks 😅 will try to include this in another PR
Fixes #475
Using plain
filters()call with a list of ids to fetch is better than sequentially fetching each id.Using
parallel=Falsewe get:With
parallel=Truewe get:Summary by CodeRabbit