feat: implement batching and optimization for deployment evaluation #1139
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add --indexer-min-stake-threshold CLI option to indexer-agent start command - Integrate CLI flag with existing INDEXER_MIN_STAKE_THRESHOLD env var - Update evaluateDeployments function to accept optional threshold parameter - Pass threshold value through agent configuration to deployment evaluation - Add comprehensive documentation in README including performance optimization section - Maintain backward compatibility with environment variable fallback This allows users to configure the minimum stake threshold for deployment filtering via CLI, improving the performance optimization feature usability without requiring environment variable changes.
… evaluation - Add batching and filtering to deployment evaluation for O(N×M) to O(N+M) complexity reduction - Implement --indexer-min-stake-threshold CLI flag for configurable stake filtering - Add async evaluateDeployments function with configurable batch sizes - Use Map-based rule lookups instead of linear scans for O(1) access - Filter deployments by minimum stake/signal thresholds before processing - Extract reusable evaluation logic into evaluateDeploymentByRules function - Add comprehensive performance logging and metrics - Update Agent class to support threshold configuration - Maintain backward compatibility with environment variables - Update Dockerfile.dev to ensure proper dependency resolution - Add comprehensive README documentation for performance features This addresses the performance bottleneck that caused indexer to hang when processing large numbers of deployments (15K+) and rules.
- Add comprehensive DEVELOPMENT.md with local development and testing guide - Move detailed development documentation from README.md to DEVELOPMENT.md - Add reference to DEVELOPMENT.md in main README.md - Include Dockerfile.dev usage, testing workflows, and troubleshooting - Maintain clean separation between user and developer documentation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
evaluateDeployments function with batching support
This addresses the O(N×M) performance bottleneck that caused the indexer
to hang when processing large numbers of deployments and rules.
Performance improvements scale from O(N×M) to O(N+M) complexity.
Fixes: Backend hanging at "Finished fetching subgraph deployments