Probe Jitter #167
Description
Description
Probe jitter is a scheduling enhancement that prevents the "thundering herd" problem by introducing controlled randomization in probe execution timing. Instead of all probes firing simultaneously at their configured intervals, jitter distributes probe execution across time windows to smooth network load and system resource utilization.
Currently, ThingConnect Pulse executes probes at fixed intervals (default 10 seconds), which can cause network congestion and resource spikes when monitoring hundreds of endpoints simultaneously. Probe jitter resolves this by adding small, randomized delays to spread probe execution more evenly.
Scope of Work
Core Implementation:
- Implement automatic jitter calculation (10% of probe interval)
- Apply jitter uniformly across all endpoints during service startup
- Use endpoint ID-based seeding for consistent jitter patterns across restarts
Algorithm:
textjitteredDelay = baseInterval + (hash(endpointId) % (baseInterval * 0.2))
Acceptance Criteria
- Zero Configuration: No YAML changes required, works automatically
- Transparent Operation: Users see smooth system performance without understanding jitter
- Consistent Behavior: Same jitter pattern after service restarts
- Performance: Eliminates thundering herd effects during startup
- Backwards Compatibility: Existing configurations work unchanged
Developer Implementation Notes
Implementation Details:
- Calculate jitter in
MonitoringBackgroundService.ProbeEndpointAsync() - Use
endpointId.GetHashCode()for deterministic randomization - Apply 10% jitter automatically (no configuration option)
- Jitter affects timer scheduling only, not timestamp recording