Add adaptive circuit breaker

[AbdulRahmanAlHamali]

Add a new adaptive circuit breaker that:

1. Works without need for configuration
2. Has the ability to open partially/fully depending on the severity of the incident

The circuit breaker has two main components:

Ideal Error Rate Estimator

This estimator tries to find out the expected error rate from a healthy dependency. It does that through a method of simple exponential smoothing, which basically relies on calculating a weighted average, with a few extra domain-specific hints:

1. Ignore any observations that are obviously too high
2. Converge slowly towards higher values, and quickly towards lower values
3. Be more receptive to signals for the first 30 minutes after boot (because we start with a random guess for the value), and less receptive afterwards

PID Controller (Process, Integral, Derivative) Controller

See wikipedia link

This controller increases/decreases the the rejection rate of the circuit breaker based on the value of:

kI * Integral + kP * P + kD * derivative

Where P is:

ErrorDelta - (1 - ErrorDelta) * RejectionRate

The intuition of the value of P:

• Increases when error rate increases
• Decreases when rejection rate increases
• The rejection rate has a (1 - ErrorDelta) multiplier, that we call the "defensiveness" multiplier. This allows rejection to increase more aggressively if the error rate is high

Note: The derivate component is currently set to 0. The Integral component mostly contributes as a history to prevent the circuit breaker from fluctuating too much.

[kris-gaudel]

Regarding the calculation of online mean, Welford's algorithm was discussed as a solution to use constant space

[kris-gaudel]

Also, do we want to allow developers to configure the interval of integration? (history_duration)?

[kris-gaudel]

Hey guys, hope you had a good start to the new year! Are there any issue I could work on in an open-source capacity on the ACB?

[samuel-williams-shopify]

Suggested change

	Kernel.sleep(@sliding_interval)
	Kernel.sleep(@sliding_interval * rand(0.9...1.1)

[samuel-williams-shopify]

Here are the notes from our tuple:

(A) 1 PIDController
(B) 1 Faraday request

(A) -> sleep(sliding_window_timeout)
(B) -> timeout(t, IO#write -> IO#gets)
(A) -> wakes up, update
(B) -> timeout(t, IO#write -> IO#gets -> response line) -> good

(An) -> sleep(sliding_window_timeout + random_jitter)
(A0) -> wakes up, update -> some CPU time, -> A(N+1) -> time has elapsed
(B) -> timeout(t, IO#write -> IO#gets)
(AN+1) -> wakes up, update # process all of these O(An)
(B) -> timeout(t, IO#write -> IO#gets -> response line) -> times out


Assume sliding window of 1 second
(A0) t+0.1
(A1) t+0.2
(A2) t+0.3
(A0) -> wakes up, update -> some CPU time, -> A(N+1) -> time has elapsed
(A0) t+0.1 -> ready at t+1.0 -> sleep(sliding_window_timeout)
(A1) t+0.2 -> ready at t+1.0 -> sleep(sliding_window_timeout)
(A2) t+0.3 -> ready at t+1.0 -> sleep(sliding_window_timeout)

(1) +random jitter -> always late
(2) +/- random jitter -> gaussian distribution of lateness
(3) + robust loop which computes offset, but you may need to drop sample windows if running late
	(real time stuff)

duration = 1.0
def now
	Process.clock_gettime(Process::CLOCK_MONOTONIC)
end


while true
	current = now
	yield

	next_time = current + duration
	duration = next_time - now
	if duration > 0
		sleep(duration)
	else
		# droppped window
	end
end

while true
	# uniform distribution:
	duration = duration * rand(0.9..1.1)
	sleep(duration * rand(0.9..1.1))

	yield(duration) -> pid_controller.update(duration)
end