feat: Replace tiers mechanism with Weighted Random Selection providers selection #2112
+6,074
−2,174
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Test Results 6 files - 1 83 suites - 2 29m 26s ⏱️ - 4m 1s For more details on these errors, see this check. Results for commit 0ea47c1. ± Comparison against base commit f0e3d53. This pull request removes 240 and adds 32 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
- Introduced a new `WeightedSelector` to replace the legacy tier-based selection with a probability-based approach, allowing for more nuanced provider selection based on composite QoS scores. - Added configuration options for weighted selection, including weights for availability, latency, sync, and stake. - Updated the `ProviderOptimizer` to support the new weighted selection mechanism, including methods for enabling/disabling and configuring the weighted selector. - Enhanced the `RPCConsumer` to apply weighted selection settings via CLI flags. - Added comprehensive tests for the new weighted selection logic and its components. This change aims to improve provider selection efficiency and adaptability in varying network conditions. Took 2 hours 23 minutes
…optimization - Eliminated the tier-based selection system in favor of a weighted selection approach, enhancing provider choice based on composite QoS scores. - Removed associated configuration options and methods related to tiers, including `OptimizerNumTiers` and `OptimizerMinTierEntries`. - Updated tests to reflect the new weighted selection logic, ensuring that providers are chosen based on their performance metrics rather than fixed tiers. - Adjusted the `ProviderOptimizer` and `RPCConsumer` to support the new selection mechanism, improving overall efficiency and adaptability. This refactor aims to simplify the provider selection process and improve performance in varying network conditions. Took 5 hours 38 minutes
- Enhanced tests in `provider_optimizer_test.go` to reflect the new weighted selection mechanism, ensuring that providers with better latency, availability, and sync are selected more frequently. - Adjusted assertions to validate that good providers collectively receive more selections than bad ones, accounting for minimum selection chances and QoS factors. - Improved clarity in test comments to better describe the expected behavior under the new selection strategy. These changes aim to ensure comprehensive coverage of the new provider selection logic and maintain test reliability.
- Updated tests in `provider_optimizer_test.go` to incorporate extreme latency differences, ensuring deterministic selection behavior in the weighted selection logic. - Adjusted latency values for providers to reflect a broader range, improving the clarity of expected outcomes in test assertions. - Enhanced comments throughout the tests to better describe the rationale behind the chosen latency values and their impact on provider selection. These changes aim to strengthen the test coverage for the provider optimizer's selection mechanism under varying network conditions.
- Updated `provider_optimizer_test.go` to increase the sync block difference to 50 blocks, ensuring more significant variations in provider selection behavior. - Adjusted the number of iterations in the test to 2000 to reduce variance in results, enhancing the reliability of the weighted selection assertions. - Improved comments to clarify the rationale behind the changes and the expected outcomes in the context of provider selection. These modifications aim to strengthen the test coverage for the provider optimizer's selection mechanism under extreme conditions.
- Updated `provider_optimizer_test.go` to increase the number of iterations to 10,000 for improved statistical confidence in provider selection behavior. - Added detailed comments explaining the statistical analysis and expected outcomes for good vs. bad providers based on sync performance. - Adjusted assertions to validate that the selection ratio of good providers remains within acceptable bounds, ensuring the weighted selection mechanism functions correctly. - Enhanced logging of selection ratios for better monitoring of test behavior. These modifications aim to strengthen the test coverage for the provider optimizer's selection mechanism, ensuring it operates reliably under varying conditions. Took 31 minutes
…guration - Refactored static provider entries to include names and updated node URLs to use port 2220 and 2221. - Added skip-verifications for enhanced validation of node URLs. - Improved organization of provider definitions for clarity and maintainability. These changes aim to streamline the configuration of static providers and ensure consistency across the setup.
…enhance weighted selection - Removed tier-based selection flags and logic from the ProviderOptimizer, streamlining the provider selection process. - Updated ChooseProvider method to eliminate tier return value, focusing solely on weighted selection. - Introduced ChooseBestProvider method for consistent provider selection in sticky sessions. - Refactored related tests to align with the new selection approach, ensuring deterministic behavior with fixed random seeds. - Deleted obsolete selection weight utility and associated tests to simplify the codebase. This refactor aims to improve the clarity and efficiency of the provider selection process by relying solely on weighted selection based on QoS scores.
- Updated the instantiation of ProviderOptimizer in multiple files to eliminate the unused 'autoAdjustment' parameter, simplifying the code. - Adjusted related tests to reflect the changes in the ProviderOptimizer constructor. - Ensured consistency across the consumer session manager and integration tests by aligning the optimizer initialization. This change aims to streamline the provider optimization process and improve code clarity.
- Introduced a Randomizer interface to allow for flexible random number generation, supporting both global probabilistic and deterministic RNG for testing. - Implemented a globalRandomizer struct to utilize the standard library's random functions. - Updated the NewWeightedSelector constructor to initialize the RNG with the global randomizer. - Added SetDeterministicSeed method to enable setting a specific seed for testing purposes, ensuring reproducible provider selection. These changes improve the testability of the WeightedSelector by allowing deterministic behavior during tests while maintaining the original probabilistic selection in production.
- Introduced SetDeterministicSeed method in ProviderOptimizer to allow setting a specific seed for reproducible provider selection during tests. - Updated multiple test files to utilize the new method, ensuring consistent behavior across tests. - Refactored consumer session manager initialization to use the deterministic seed, enhancing test reliability. These changes improve the testability of the provider optimization process by ensuring that random behaviors can be controlled during testing.
…ents - Added initialization of random seed in provider optimizer tests to ensure consistent test results. - Updated comments for clarity in the `runChooseManyTimesAndReturnResults` function. - Improved error handling in various test files to prevent recursive panic during log saving. - Introduced a manual build tag for the proxy test to facilitate integration testing. These changes aim to enhance the reliability and clarity of tests across the provider optimization and proxy components.
- Added logic to apply block availability penalties for historical queries in the provider optimizer. - Introduced `calculateBlockAvailability` function to determine the likelihood that a provider has synced to the requested block using a Poisson distribution model. - Updated tests to validate the new block availability logic and ensure correct provider selection based on sync states. - Enhanced existing tests for consistency and clarity, ensuring deterministic behavior in test results.
Compute block availability as P(X>=d)=1-P(X<=d-1) and stabilize tests.
Normalize latency/sync against score.WorstLatencyScore (30s) and score.WorstSyncScore (1200s), update tests accordingly.
Set default MinSelectionChance to 0.01 (1%) to reduce selection flattening while still preventing starvation.
Reject NaN/Inf/negative weights; fallback to default weights while preserving Strategy and MinSelectionChance; add unit tests.
Add unit tests to ensure NaN/Inf weights fall back to default weights while preserving Strategy and MinSelectionChance.
AnnaR-prog
force-pushed
the
feature/refactor-tiers
branch
from
2bc0e1e to
b377f27
Compare
… feature/refactor-tiers
… feature/refactor-tiers
- Added new metrics for tracking provider selection statistics, including availability, latency, sync, stake, and composite scores. - Updated ConsumerMetricsManager to include methods for setting and updating selection stats. - Integrated selection metrics into the provider selection process within ConsumerSessionManager. - Implemented periodic updates for selection stats from optimizer reports. - Enhanced the RPC consumer and smart router to initiate selection stats updates.
…ency Implements the recommended Phase 2 approach that combines: - EWMA on raw latency values (existing behavior, stable & interpretable) - Adaptive P10-P90 bounds from T-Digest (better distribution: 85-95% vs 70-85%) - Robust to both-sided outliers (excludes bottom 10% and top 10%) Changes: - Add GetAdaptiveBounds() to AdaptiveMaxCalculator returning (p10, p90) - Add GetAdaptiveBounds() to LatencyScoreStore exposing adaptive bounds - Update WeightedSelector.normalizeLatency() to use P10-P90 normalization: normalized = 1 - (clamp(latency, P10, P90) - P10) / (P90 - P10) - Update WeightedSelectorConfig.AdaptiveLatencyGetter signature to return (p10, p90) - Add 4 new comprehensive tests for P10-P90 functionality Key improvements over P95-only approach: - Better distribution: 85-95% range utilization (vs 70-85%) - Adaptive minimum: P10 adapts to network baseline (not fixed at 0) - More stable: EWMA on raw values preserves interpretability - More robust: Excludes outliers on both ends All 14 tests pass. Backward compatible with feature flag.
…ging
1. Fix P10 lower bound: 0.5s → 0.001s (1ms)
- Allows very fast providers (local, optimized) with <50ms latency
- Prevents artificial floor that compresses fast provider scores
2. Centralize all adaptive normalization constants in score_config.go
- AdaptiveP10MinBound = 0.001 (1ms)
- AdaptiveP10MaxBound = 10.0 (10s)
- DefaultTDigestCompression = 100.0
- DefaultLatencyAdaptiveMinMax/MaxMax = 1.0/30.0
- All constants now in one discoverable location
3. Add comprehensive logging for distribution visualization
- AdaptiveMaxCalculator.LogDistributionStats(): logs 10 percentiles (P01-P999),
clamped values, IQR, range, and config for Python plotting
- WeightedSelector.normalizeLatency(): logs every normalization with
raw/clamped/normalized values, P10-P90 bounds, clamping flags
- Enables data extraction from logs for distribution analysis
All 14 tests passing.
…eter
Implements the recommended Phase 2 approach for sync with same hybrid strategy as latency:
- EWMA on raw sync lag values (existing behavior, stable & interpretable)
- Adaptive P10-P90 bounds from T-Digest (better distribution: 85-95% vs 55-70%)
- Robust to both-sided outliers (excludes bottom 10% and top 10%)
Key Changes:
1. Add sync-specific constants (score_config.go):
- AdaptiveSyncP10MinBound = 0.1s (100ms per user request, not 1.0s)
- AdaptiveSyncP10MaxBound = 60.0s
- DefaultSyncAdaptiveMinMax = 30.0s
- DefaultSyncAdaptiveMaxMax = 1200.0s
2. Enhance AdaptiveMaxCalculator (adaptive_max_calculator.go):
- Add minP10/maxP10 fields for parameter-specific P10 bounds
- Update NewAdaptiveMaxCalculator() signature: (halfLife, minP10, maxP10, minMax, maxMax, compression)
- Update GetAdaptiveBounds(), GetStats(), LogDistributionStats() to use configured P10 bounds
- Allows different P10 bounds for latency (0.001s) vs sync (0.1s)
3. Enhance SyncScoreStore (score_store.go):
- Add adaptiveMax *AdaptiveMaxCalculator field
- Update Update() to feed samples to T-Digest
- Add GetAdaptiveBounds() returning (p10, p90)
- Add EnableAdaptiveMax() with sync-specific P10 bounds
- Add IsAdaptiveMaxEnabled() and GetAdaptiveMaxStats()
4. Update WeightedSelector (weighted_selector.go):
- Add useAdaptiveSyncMax flag and adaptiveSyncGetter func() (p10, p90)
- Update normalizeSync() to use P10-P90 normalization:
normalized = 1 - (clamp(syncLag, P10, P90) - P10) / (P90 - P10)
- Add comprehensive logging: raw_sync_lag, clamped_sync_lag, p10, p90,
range_p10_p90, normalized_score, was_clamped_low, was_clamped_high
- Update GetConfig() to include sync adaptive fields
5. Update all tests (adaptive_max_calculator_test.go):
- Add newTestAdaptiveMaxCalculator() helper for cleaner test code
- Update all 14 test cases to pass new P10 bounds
Improvements vs P95-only:
- Better distribution: 85-95% range utilization (vs 55-70%)
- Adaptive minimum: P10 adapts to network baseline (~25-30s typical)
- More stable: EWMA on raw values preserves interpretability
- More robust: Excludes outliers on both ends
- Consistent: Same approach as latency parameter
All 14 tests pass. Backward compatible with feature flag.
Replaces linear stake normalization with square root scaling to reduce whale dominance while maintaining staking incentives. Benefits: - Reduces whale-to-small-staker gap by 17% (0.140 → 0.116) - Small stakers (10% stake) get 3x boost: 0.020 → 0.063 (+215%) - Whale (80% stake) still rewarded but less dominant: 0.160 → 0.179 - Simpler formula than logarithmic: just sqrt(ratio) - Maintains monotonicity: more stake always better Mathematical comparison: - Linear: 80% stake = 0.80 → 0.160 contribution - Square Root: 80% stake = 0.894 → 0.179 contribution - Linear: 10% stake = 0.10 → 0.020 contribution - Square Root: 10% stake = 0.316 → 0.063 contribution Gap reduction: - Linear gap: 0.160 - 0.020 = 0.140 - Square Root gap: 0.179 - 0.063 = 0.116 (17% smaller) Changes: - Updated normalizeStake() to use math.Sqrt(stakeRatio) - Added comprehensive logging for distribution visualization - Updated documentation with correct analysis Note: Logarithmic scaling was found to increase whale dominance, not reduce it (0.170 > 0.160), so square root is the correct choice.
- Change MinAcceptableAvailability from 0.90 to 0.80 - More realistic threshold for production networks - Prevents unfairly penalizing providers with occasional issues - Still maintains strong quality bar (80% is industry acceptable) - Particularly important for archive providers - Tested with 50+ requests, shows proper distribution
…gest) Implements and enables Phase 2 adaptive normalization for latency and sync: **Architecture:** - Global T-Digest calculators aggregate samples from ALL providers - Network-wide P10 and P90 percentiles for consistent normalization - Exponential decay weighting aligned with EWMA (1-3 hour half-life) **Key Changes:** 1. Added global adaptive calculators to ProviderOptimizer: - globalLatencyCalculator: tracks network-wide latency distribution - globalSyncCalculator: tracks network-wide sync distribution 2. Feed samples to global T-Digests in updateDecayingWeightedAverage() - Every latency/sync sample from every provider is fed to global digest - Applies same CU weighting as EWMA for consistency 3. Wire adaptive getters in ConfigureWeightedSelector(): - getAdaptiveLatencyBounds() returns actual P10/P90 from global T-Digest - getAdaptiveSyncBounds() returns actual P10/P90 from global T-Digest - Previously returned fixed defaults (Phase 2 was effectively disabled) **Benefits:** - 85-95% range utilization (vs 70-85% with fixed max) - Robust to outliers (excludes bottom 10% and top 10%) - Adapts to network conditions automatically - Memory overhead: ~20 KB total (10 KB per parameter) **Enabled by Default:** - Latency: P10-P90 adaptive normalization (Phase 2) - Sync: P10-P90 adaptive normalization (Phase 2) - Availability: 80% threshold with simple rescaling (Phase 1) - Stake: Square root scaling (reduces whale dominance) Tested: Builds successfully, ready for integration testing
Add github.com/influxdata/tdigest v0.0.1 for T-Digest implementation used in AdaptiveMaxCalculator for P10-P90 percentile calculation. Required for Phase 2 adaptive normalization feature.
Switch from influxdata/tdigest v0.0.1 (2019) to caio/go-tdigest v5.0.0 (2025): **Why upgrade:** - influxdata/tdigest: Last release Nov 2019 (v0.0.1) - caio/go-tdigest: Latest release Nov 2025 (v5.0.0) - 6 years of improvements, optimizations, and bug fixes - Actively maintained (updated Jan 2026) - Better performance and reliability **API changes:** - NewWithCompression() → New(tdigest.Compression()) - Add(value, weight) → AddWeighted(value, weight uint64) - Centroids() → ForEachCentroid(callback) - Compression field → Compression() method - Count() returns uint64 instead of float64 **Implementation:** - Updated import to github.com/caio/go-tdigest/v5 - Adapted decay logic to use ForEachCentroid iterator - Fixed weight handling (uint64 with rounding) - Updated all tests to match new API **Testing:** - ✅ All AdaptiveMaxCalculator tests pass - ✅ Build successful - ✅ Backward compatible behavior (same percentile calculations)
- Introduced new metrics for provider selection, including availability, latency, sync, stake, and composite scores. - Updated ConsumerMetricsManager to manage and record these metrics during provider selection. - Implemented a conversion function to transform selection statistics into the new metrics format. - Adjusted the provider selection process in ConsumerSessionManager to utilize the new metrics. - Enhanced the ConsumerOptimizerQoSClient to track selection counts and average QoS scores for providers. This update improves the granularity of provider performance tracking and enables better decision-making based on historical selection data.
- Introduced sanitizeFloat function to ensure that NaN and Inf values are replaced with 0 in metrics. - Updated appendOptimizerQoSReport method to utilize sanitizeFloat for various score fields, enhancing the robustness of JSON output. - This change improves data integrity in metrics reporting by preventing invalid float values.
- Added warnings for missing selection statistics and provider scores in ConsumerSessionManager to improve debugging and tracking of provider selection metrics. - Updated ConsumerMetricsManager to log cases where the selected provider is not found in the scores list, ensuring better visibility into provider performance and selection integrity. - Enhanced validation checks for selected provider scores, including handling of NaN and Inf values, to maintain data integrity in metrics reporting.
- Add comprehensive logging to CalculateScore() showing: * Raw QoS values (availability, latency, sync, stake) * Normalized scores for each parameter * Weights applied to each parameter * Weighted contributions (normalized_score × weight) * Final composite score - Add detailed selection logging showing: * All candidate providers with their scores * Selection probabilities as percentages * Parameter breakdown for each candidate * RNG value and selected provider - Add weighted contributions to metrics: * availability_contribution * latency_contribution * sync_contribution * stake_contribution - Clean up metrics structure: * Remove redundant Raw* fields (duplicated legacy fields) * Add clear comments explaining legacy vs WRS fields * Document that legacy Score fields are raw EWMA values This enables detailed analysis of provider selection via logs or metrics endpoint without redundancy.
- Add init_test3_sync_impact.sh: Dedicated test script for sync parameter testing * Configures 3 providers with different sync lags (0, 1, 2 blocks behind) * Sets weights: Sync=0.5, Availability=0.5, Latency=0, Stake=0 * Includes optimizer-qos-listen flag for metrics endpoint * Runs 500 test requests with cache bypass * Provides distribution analysis - Add analyze_wrs_metrics.py: Python script to analyze metrics endpoint * Fetches and parses provider_optimizer_metrics * Verifies score calculations (composite = sum of contributions) * Calculates selection probabilities * Shows parameter contributions and impact analysis * Displays raw EWMA values vs normalized scores - Update init_lava_static_test_three_provider_with_archive.sh: * Add --optimizer-qos-listen flag to enable metrics endpoint * Add metrics URL to setup completion message - Update WRS_ENHANCED_LOGGING_AND_METRICS.md: * Clarify existing vs new fields in metrics * Update examples to reflect actual field names
- Added checks for NaN and Inf values in adaptive latency and sync bounds within ProviderOptimizer and WeightedSelector, ensuring robust metrics reporting. - Enhanced logging to warn when invalid bounds are detected, allowing fallback to default values. - Updated ConsumerMetricsManager to return an empty JSON array when the consumer optimizer client is not initialized, improving error handling in metrics responses. These changes enhance the reliability and clarity of metrics related to provider selection and performance.
- Renamed getProviderScore to getProviderSelectionWeight for clarity and updated its functionality to return the selection weight of a provider. - Introduced a new function, getProviderCompositeScore, to retrieve the composite score of a provider. - Updated logging in ChooseProviderWithStats and ChooseBestProviderWithStats to reflect the new metrics, enhancing the visibility of provider selection criteria. These changes improve the accuracy and clarity of provider selection metrics in the logging output.
- Added support for a third QuickNode endpoint (ETH_RPC_URL_3) to improve provider flexibility. - Updated validation checks to ensure all ETH RPC URLs and WebSocket URLs are set correctly, providing warnings for placeholders. - Clarified logging messages to reflect the new provider setup and load distribution strategy. - Improved error handling for WebSocket URL requirements, ensuring proper configuration for subscriptions. These changes enhance the robustness and clarity of the Ethereum provider setup process.
…nfig - Created provider1_test3_tendermintrpc_only.yml with only tendermintrpc endpoint - Reduces Provider 1 startup time from ~45s to ~12s (rest endpoint takes 40s to initialize) - Updated init_test3_sync_impact.sh to use new config - Ensures faster and more reliable latestSync establishment in Phase 1
… feature/refactor-tiers
- Added optimizer-qos-listen flag to expose metrics endpoint - Configured explicit optimizer weights (availability, latency, sync, stake) - Added informative messages about active QoS improvements - Updated Provider 3 test config: availability 0.95→1.0, head_on_first_request false→true - Fixed whitespace in adaptive_max_calculator.go
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 185 files with indirect coverage changes 🚀 New features to boost your workflow:
|
…ReputationMetric) with Prometheus The lava_consumer_latest_provider_block metric and QoS metrics were not working because they were never registered with Prometheus registry. Added missing registerMetric() calls for: - latestBlockMetric (lava_consumer_latest_provider_block) - qosMetric (lava_consumer_qos_metrics) - providerReputationMetric (lava_consumer_provider_reputation_metrics) This was a pre-existing bug in main branch that prevented these metrics from being exposed in the Prometheus metrics endpoint.
Root cause: Two issues prevented the metric from working: 1. Missing Prometheus registration (pre-existing bug in main): - latestBlockMetric was defined and used but never registered - Also registered qosMetric and providerReputationMetric 2. Missing reply.LatestBlock population (merge conflict resolution issue): - Commit c6e17d5 in main added 'reply.LatestBlock = latestBlock' - This line was lost during merge conflict resolution - Without it, reply.LatestBlock stays 0 (node only populates reply.Data) - Consumers depend on this for consistency tracking and provider selection Also removed obsolete testModeLatestBlockOverride code since DR was removed in main. Fixes: lava_consumer_latest_provider_block metric now correctly reports provider blocks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR refactors the provider optimization logic by removing the legacy tier-based selection mechanism and implementing a modern weighted selection system. This change simplifies the codebase while improving provider selection flexibility and testability.
Key Changes