Add adaptive circuit breaker

.github/workflows/automated-experiment-result-checker.yml

@@ -0,0 +1,70 @@
+---
+name: Automated Experiment Result Checker
+
+# yamllint disable-line rule:truthy
+on:
+  pull_request:
+    types: [opened, reopened, synchronize]
+
+concurrency:
+  group: ${{ github.ref }}-automated-experiment-result-checker
+  cancel-in-progress: true
+
+permissions:
+  contents: write
+  pull-requests: write
+jobs:
+  automated-experiment-result-checker:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8  # v5.0.0
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+          fetch-depth: 0
+
+      - name: Check for updated experiment result graphs and tables
+        run: |
+          set -e
+          cd "$(git rev-parse --show-toplevel)"
+
+          # TODO: Include lower bound windup experiment once we have a way to make it run in a reasonable time.
+          # Find all PNGs and CSV files, excluding those with "windup" in their filename
+          mapfile -t all_pngs < <(find experiments/results/main_graphs -type f -name '*.png' ! -name '*windup*.png' | sort)
+          mapfile -t all_csvs < <(find experiments/results/csv -type f -name '*.csv' ! -name '*windup*.csv' 2>/dev/null | sort)
+
+          # Combine all result files
+          all_files=("${all_pngs[@]}" "${all_csvs[@]}")
+
+          # Find all changed PNGs and CSVs in the latest commit
+          mapfile -t changed_files < <(git diff --name-only --diff-filter=AM HEAD~1..HEAD | grep -E '^experiments/results/main_graphs/.*\.png$|^experiments/results/csv/.*\.csv$' | grep -v windup | sort)
+
+          # Report any files that are not updated in the latest commit
+          declare -a not_updated=()
+          for file in "${all_files[@]}"; do
+            if ! printf "%s\n" "${changed_files[@]}" | grep -qx "$file"; then
+              not_updated+=("$file")
+            fi
+          done
+
+          if [ ${#not_updated[@]} -gt 0 ]; then
+            echo "❌ The following result files have NOT been updated in the latest commit:"
+            for f in "${not_updated[@]}"; do
+              echo "  - $f"
+            done
+            echo ""
+            echo "Every commit must update all non-windup experiment result graphs and CSV files. You may be missing updates."
+            echo "Run:"
+            echo ""
+            echo "    cd experiments"
+            echo "    bundle install"
+            echo "    bundle exec ruby run_all_experiments.rb"
+            echo ""
+            echo "Commit the updated graphs and CSV files to resolve this check."
+            exit 1
+          fi
+
+          echo "✅ All non-windup experiment result graphs and CSV files are up to date for this commit!"
+
+
+

.github/workflows/test.yml

@@ -6,7 +6,7 @@ on:
   push:
     branches: [ main ]
   pull_request:
-    branches: [ main ]
+    branches: [ "**" ]
   workflow_call:
 
 concurrency:

Gemfile

@@ -15,6 +15,7 @@ group :test do
   gem "pry-byebug", require: false
   gem "toxiproxy"
   gem "webrick"
+  gem "rubystats"
 
   gem "bigdecimal"
   gem "mutex_m"

README.md

@@ -576,6 +576,52 @@ It is possible to disable Circuit Breaker with environment variable
 For more information about configuring these parameters, please read
 [this post](https://shopify.engineering/circuit-breaker-misconfigured).
 
+#### Adaptive Circuit Breaker (Experimental)
+
+Semian also includes an experimental adaptive circuit breaker that uses a [PID controller](https://en.wikipedia.org/wiki/Proportional%E2%80%93integral%E2%80%93derivative_controller)
+to dynamically adjust the rejection rate based on real-time error rates. Unlike the
+traditional circuit breaker with fixed thresholds, the adaptive circuit breaker continuously
+monitors error rates and adjusts its behavior accordingly.
+
+##### How It Works
+
+The adaptive circuit breaker uses the error function:
+```
+P = (error_rate - ideal_error_rate) - (1 - (error_rate - ideal_error_rate)) * rejection_rate
+```
+
+This formula ensures that:
+- Rejection rate increases when the service is unhealthy
+- Rejection rate decreases when the service recovers
+- The system finds an equilibrium that protects against cascading failures while allowing recovery
+
+##### Adaptive Circuit Breaker Configuration
+
+To enable the adaptive circuit breaker, simply set:
+
+- **adaptive_circuit_breaker**. Enable adaptive circuit breaker instead of traditional. Defaults to `false`.
+
+Example configuration:
+```ruby
+Semian.register(
+  :my_service,
+  adaptive_circuit_breaker: true,  # Use adaptive instead of traditional
+  bulkhead: false                   # Can be combined with bulkhead
+)
+```
+
+The adaptive circuit breaker uses carefully tuned internal parameters based on extensive testing:
+- PID controller gains optimized for stability and responsiveness
+- 10-second window for rate calculations
+- 1-hour history for ideal error rate calculation (p90)
+- 1-second interval for background health checks
+
+The adaptive circuit breaker can be disabled with the environment variable
+`SEMIAN_ADAPTIVE_CIRCUIT_BREAKER_DISABLED=1`.
+
+**Note**: When `adaptive_circuit_breaker: true` is set, traditional circuit breaker
+parameters (`error_threshold`, `error_timeout`, etc.) are ignored.
+
 ### Bulkheading
 
 For some applications, circuit breakers are not enough. This is best illustrated

examples/dual_circuit_breaker_demo.rb

@@ -0,0 +1,175 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "semian"
+
+# Example: Dual Circuit Breaker Demo
+# This demonstrates how to use both legacy and adaptive circuit breakers
+# simultaneously, switching between them at runtime based on a callable.
+
+# Simulate a feature flag that can be toggled
+class ExperimentFlags
+  @enabled = false
+
+  def enable_adaptive!
+    @enabled = true
+  end
+
+  def disable_adaptive!
+    @enabled = false
+  end
+
+  def use_adaptive_circuit_breaker?
+    @enabled
+  end
+end
+
+# Helper function to print state of all Semian objects between each phase
+def print_semian_state
+  puts "\n=== Semian Resources State ===\n"
+  Semian.resources.values.each do |resource|
+    puts "Resource: #{resource.name}"
+
+    # Bulkhead info
+    if resource.bulkhead
+      puts "  Bulkhead: tickets=#{resource.tickets}, count=#{resource.count}"
+    else
+      puts "  Bulkhead: disabled"
+    end
+
+    # Circuit breaker info
+    cb = resource.circuit_breaker
+    if cb.nil?
+      puts "  Circuit Breaker: disabled"
+    elsif cb.is_a?(Semian::DualCircuitBreaker)
+      puts "  Circuit Breaker: DualCircuitBreaker"
+      metrics = cb.metrics
+      puts "    Active: #{metrics[:active]}"
+      puts "    Classic: state=#{metrics[:classic][:state]}, open=#{metrics[:classic][:open]}, half_open=#{metrics[:classic][:half_open]}"
+      puts "    Adaptive: rejection_rate=#{metrics[:adaptive][:rejection_rate]}, error_rate=#{metrics[:adaptive][:error_rate]}"
+    elsif cb.is_a?(Semian::AdaptiveCircuitBreaker)
+      puts "  Circuit Breaker: AdaptiveCircuitBreaker"
+      puts "    open=#{cb.open?}, closed=#{cb.closed?}, half_open=#{cb.half_open?}"
+    else
+      puts "  Circuit Breaker: Legacy"
+      puts "    state=#{cb.state&.value}, open=#{cb.open?}, closed=#{cb.closed?}, half_open=#{cb.half_open?}"
+      puts "    last_error=#{cb.last_error&.class}"
+    end
+    puts ""
+  end
+  puts "=== END STATE OUTPUT ===\n\n"
+end
+
+# Register a resource with dual circuit breaker mode
+resource = Semian.register(
+  :my_service,
+  # Enable dual circuit breaker mode
+  dual_circuit_breaker: true,
+
+  # Legacy circuit breaker parameters (required)
+  success_threshold: 2,
+  error_threshold: 3,
+  error_timeout: 10,
+
+  # Adaptive circuit breaker parameters (optional, has defaults)
+  seed_error_rate: 0.01,
+
+  # Common parameters
+  tickets: 5,
+  timeout: 0.5,
+  exceptions: [RuntimeError],
+)
+
+experiment_flags = ExperimentFlags.new
+Semian::DualCircuitBreaker.adaptive_circuit_breaker_selector(->(_resource) { experiment_flags.use_adaptive_circuit_breaker? })
+
+puts "=== Dual Circuit Breaker Demo ===\n\n"
+
+# Helper to simulate service calls
+def simulate_call(success: true)
+  if success
+    "Success!"
+  else
+    raise "Service error"
+  end
+end
+
+# Test with legacy circuit breaker (use_adaptive returns false)
+puts "Phase 1: Using LEGACY circuit breaker (use_adaptive=false)"
+puts "The first 3 requests will succeed, the rest will fail."
+puts "-" * 50
+
+experiment_flags.disable_adaptive!
+
+10.times do |i|
+  result = Semian[:my_service].acquire do
+    simulate_call(success: i < 3) # First 3 succeed, rest fail
+  end
+  puts "  Request #{i + 1}: #{result}"
+rescue => e
+  puts "  Request #{i + 1}: Failed - #{e.class.name}: #{e.message}"
+end
+
+print_semian_state
+
+# Reset both circuit breakers
+puts "\n" + "=" * 50
+puts "Resetting circuit breakers..."
+resource.circuit_breaker.reset
+
+# Test with adaptive circuit breaker (use_adaptive returns true)
+puts "\nPhase 2: Using ADAPTIVE circuit breaker (use_adaptive=true)"
+puts "The first 3 requests will succeed, then the rest will be failures."
+puts "The adaptive circuit breaker is not expected to open yet."
+puts "-" * 50
+
+experiment_flags.enable_adaptive!
+
+10.times do |i|
+  begin
+    result = Semian[:my_service].acquire do
+      simulate_call(success: i < 3) # First 3 succeed, rest fail
+    end
+    puts "  Request #{i + 1}: #{result}"
+  rescue => e
+    puts "  Request #{i + 1}: Failed - #{e.class.name}: #{e.message}"
+  end
+  sleep 0.05 # Small delay to see adaptive behavior
+end
+
+print_semian_state
+
+# Demonstrate dynamic switching
+puts "\n" + "=" * 50
+puts "Phase 3: Dynamic switching between circuit breakers"
+puts "-" * 50
+
+5.times do |i|
+  # Toggle every 2 requests
+  if i.even?
+    experiment_flags.disable_adaptive!
+    puts "  Switched to LEGACY"
+  else
+    experiment_flags.enable_adaptive!
+    puts "  Switched to ADAPTIVE"
+  end
+
+  begin
+    result = Semian[:my_service].acquire do
+      simulate_call(success: true)
+    end
+    puts "    Request #{i + 1}: #{result}"
+  rescue => e
+    puts "    Request #{i + 1}: Failed - #{e.class.name}"
+  end
+end
+
+puts "\n=== Demo Complete ===\n"
+puts "Both circuit breakers tracked all requests, but only the active one"
+puts "was used for decision-making based on the adaptive_circuit_breaker_selector callable."
+
+print_semian_state
+
+# Cleanup
+Semian.destroy(:my_service)

experiments/Gemfile

@@ -2,8 +2,9 @@
 
 source "https://rubygems.org"
 
+gemspec path: ".."
+
 # Graphing library for visualization
 gem "gruff", "~> 0.23"
-
-# Alternative: Use rubyplot for pure Ruby plotting (no ImageMagick dependency)
-# gem 'rubyplot', '~> 0.1'
+gem "logger"
+gem "csv"