Add functional test scenario with observability stack

Revamp the HTTP functional test to use the new circuit breaker API:
- Server with Run[T], BreakerBox, MetricsCollector, /metrics endpoint
- Client with Run[T], IsExcluded, cascading breaker protection
- Docker Compose with Prometheus and Grafana
- Pre-provisioned Grafana dashboard (state, rates, rejections, totals)
- Thorough README with architecture, what to observe, PromQL queries

Run with: cd _functional_tests/scenarios/http && docker compose up --build

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: schigh

_functional_tests/scenarios/http/README.md

@@ -0,0 +1,150 @@
+# HTTP Functional Test — Circuit Breaker in Action
+
+This scenario demonstrates the circuit breaker library in a realistic microservice environment: an unreliable server, a client generating load, and a full observability stack to visualize circuit breaker behavior in real time.
+
+## Architecture
+
+```
+┌────────┐         HTTP         ┌────────┐
+│ Client │ ───────────────────► │ Server │
+│        │  7 named breakers    │        │
+│        │  100ms intervals     │        │──► /metrics (Prometheus format)
+└────────┘                      └────────┘
+                                     ▲
+                                     │ scrape every 5s
+                                ┌────┴─────┐
+                                │Prometheus │ :9090
+                                └────┬─────┘
+                                     │
+                                ┌────┴─────┐
+                                │ Grafana  │ :3000
+                                └──────────┘
+```
+
+**Server** — An HTTP service that wraps a simulated unreliable dependency with circuit breakers. The dependency fails ~90% of the time. Each unique breaker name (`foo`, `bar`, `baz`, etc.) gets its own independent circuit breaker via `BreakerBox`. The server exposes a `/metrics` endpoint with Prometheus-format counters.
+
+**Client** — A load generator that sends requests to the server every 100ms, randomly selecting from 7 breaker names. The client also wraps outbound calls with its own circuit breakers, demonstrating **cascading circuit breaker protection** — breakers on both sides of the network boundary.
+
+**Prometheus** — Scrapes the server's `/metrics` endpoint every 5 seconds.
+
+**Grafana** — Pre-provisioned with a dashboard showing all circuit breaker metrics. No setup required.
+
+## Running
+
+```bash
+cd _functional_tests/scenarios/http
+docker compose up --build
+```
+
+Wait for the build to complete and services to start. You'll see logs from both the server and client in your terminal.
+
+## What to Observe
+
+### Terminal Logs
+
+Within seconds, you'll see a flurry of activity:
+
+**Server logs** show state transitions as JSON:
+```
+server - state change: {"name":"foo","state":"open","opened":"2024-...","lockout_ends":"2024-..."}
+server - state change: {"name":"foo","state":"throttled","throttled":"2024-...","backoff_ends":"2024-..."}
+server - state change: {"name":"bar","state":"open","opened":"2024-...","lockout_ends":"2024-..."}
+```
+
+**Client logs** show the client-side breaker reacting to server failures:
+```
+client - [foo] error: server error 502: error inside circuit breaker 'foo': dependency failure
+client - [foo] client breaker OPEN — request rejected
+client - [foo] client breaker OPEN — request rejected
+client - client breaker 'foo' → throttled
+client - [bar] server throttled (excluded from client tracking)
+```
+
+### What's Happening
+
+1. **Errors accumulate** — The server's dependency fails 90% of the time. After 5 failures within the 1-minute window, the server-side breaker opens.
+
+2. **Server rejects requests** — While open, the server returns `503 Service Unavailable`. After a 5-second lockout, it enters the throttled state, gradually allowing more requests through using exponential backoff.
+
+3. **Client detects failures** — The client sees 502/503 responses and its own breakers open after 3 failures within 10 seconds.
+
+4. **Client sheds load** — While the client breaker is open, requests are rejected locally without hitting the network. This protects the server from thundering herd during recovery.
+
+5. **Recovery** — After backoff periods expire and error counts drop, breakers transition back to closed. The cycle repeats because the dependency remains 90% unreliable.
+
+6. **Excluded errors** — The client's `IsExcluded` callback treats server-side throttling (`429`) as excluded. These responses don't count against the client's error threshold, preventing the client from opening its own breaker just because the server is recovering.
+
+### Grafana Dashboard
+
+Open [http://localhost:3000](http://localhost:3000) (no login required — anonymous access enabled).
+
+Navigate to **Dashboards → Circuit Breaker Dashboard**, or go directly to:
+```
+http://localhost:3000/d/circuit-breaker-demo
+```
+
+The dashboard has 6 panels:
+
+| Panel | What to Look For |
+|-------|-----------------|
+| **Circuit Breaker State** | Line chart showing each breaker cycling between 0 (Closed), 1 (Throttled), and 2 (Open). You should see breakers flipping open and recovering independently. |
+| **Successes (rate/s)** | Drops to zero when breakers open, gradually recovers during throttle, returns to normal when closed. |
+| **Errors (rate/s)** | Spikes that trigger breaker opens. Watch for the rate dropping during open state (errors aren't recorded when the breaker rejects requests). |
+| **Rejected Requests (rate/s)** | Stacked bars showing load being shed. High rejection during open state, tapering off during throttle. This is the breaker doing its job. |
+| **Timeouts (rate/s)** | Should be near zero in this scenario (the dependency fails fast, not slow). |
+| **Total Counters** | Running totals for each breaker — useful for comparing relative activity across breaker names. |
+
+### Prometheus
+
+Open [http://localhost:9090](http://localhost:9090) and try these queries:
+
+```promql
+# Current state of all breakers (0=closed, 1=throttled, 2=open)
+circuit_breaker_state
+
+# Error rate per breaker over the last 30 seconds
+rate(circuit_breaker_errors[30s])
+
+# Ratio of rejected to total requests
+rate(circuit_breaker_rejected[30s]) / (rate(circuit_breaker_successes[30s]) + rate(circuit_breaker_errors[30s]) + rate(circuit_breaker_rejected[30s]))
+
+# Which breakers are currently open?
+circuit_breaker_state == 2
+```
+
+### Key Behaviors to Verify
+
+- [ ] **Independent breakers** — Each of the 7 named breakers opens and recovers on its own schedule. One breaker opening doesn't affect the others.
+- [ ] **Lockout works** — After a breaker opens, it stays open for 5 seconds (server-side) before transitioning to throttled, even if errors drop below the threshold.
+- [ ] **Gradual recovery** — During the throttled state, the rejection rate decreases over the 10-second backoff period, not all at once.
+- [ ] **Error exclusion** — Client-side breakers stay closed longer than server-side ones because server throttling responses are excluded from the client's error count.
+- [ ] **No goroutine leaks** — The server's pprof endpoint at [http://localhost:6060/debug/pprof/goroutine?debug=1](http://localhost:6060/debug/pprof/goroutine?debug=1) should show a stable goroutine count (circuit breakers create zero background goroutines).
+
+## Tuning
+
+To experiment with different behaviors, edit the circuit breaker options in the server and client `main.go` files:
+
+| Parameter | Server | Client | Effect of Increasing |
+|-----------|--------|--------|---------------------|
+| `Threshold` | 5 | 3 | More errors tolerated before opening |
+| `Window` | 1m | 10s | Longer error memory |
+| `LockOut` | 5s | 5s | Longer forced-open period |
+| `BackOff` | 10s | 10s | Slower recovery from throttle |
+| `errchance` | 9000 | — | Lower = more server-side errors (inverted: `> errchance` fails) |
+
+To change the error rate without rebuilding, adjust the `errchance` query parameter the client sends (in `client/main.go`, line with `errchance=%d`). Value of `9000` means the server fails when `rand(10000) > 9000`, so ~90% failure rate.
+
+## Stopping
+
+```bash
+docker compose down
+```
+
+## Ports
+
+| Port | Service |
+|------|---------|
+| 3000 | Grafana (dashboard) |
+| 8080 | Server (API + metrics) |
+| 6060 | Server (pprof) |
+| 9090 | Prometheus |

_functional_tests/scenarios/http/client/Dockerfile

@@ -0,0 +1,10 @@
+FROM golang:1.22-alpine AS build
+WORKDIR /src
+COPY go.mod ./
+RUN go mod download
+COPY . .
+RUN CGO_ENABLED=0 go build -o /client ./_functional_tests/scenarios/http/client
+
+FROM alpine:3.19
+COPY --from=build /client /client
+CMD ["/client"]

_functional_tests/scenarios/http/client/main.go

@@ -4,9 +4,9 @@ import (
 	"context"
 	"errors"
 	"fmt"
-	"io/ioutil"
+	"io"
 	"log"
-	"math/rand"
+	"math/rand/v2"
 	"net/http"
 	"os"
 	"os/signal"
@@ -21,76 +21,99 @@ var theBox *circuit.BreakerBox
 func main() {
 	stopChan := make(chan os.Signal, 1)
 	signal.Notify(stopChan, syscall.SIGINT, syscall.SIGTERM)
-	rand.Seed(time.Now().UnixNano())
 	log.SetPrefix("client - ")
 
+	serverAddr := os.Getenv("SERVER_ADDR")
+	if serverAddr == "" {
+		serverAddr = "http://localhost:8080"
+	}
+
 	theBox = circuit.NewBreakerBox()
 
+	// Log state changes from all client-side breakers
+	go func() {
+		for state := range theBox.StateChange() {
+			log.Printf("client breaker '%s' → %s", state.Name, state.State)
+		}
+	}()
+
+	httpClient := &http.Client{Timeout: 5 * time.Second}
+	breakerNames := []string{"foo", "bar", "baz", "fizz", "buzz", "herp", "derp"}
+
+	ticker := time.NewTicker(100 * time.Millisecond)
+	defer ticker.Stop()
+
+	log.Printf("starting load against %s", serverAddr)
+
 	for {
 		select {
 		case <-stopChan:
-			goto END
+			log.Println("shutting down...")
+			return
+		case <-ticker.C:
+			name := breakerNames[rand.IntN(len(breakerNames))]
+			go makeRequest(httpClient, serverAddr, name)
+		}
+	}
+}
+
+func makeRequest(httpClient *http.Client, serverAddr, name string) {
+	breaker, err := theBox.LoadOrCreate(name,
+		circuit.WithThreshold(3),
+		circuit.WithTimeout(2*time.Second),
+		circuit.WithBackOff(10*time.Second),
+		circuit.WithWindow(10*time.Second),
+		circuit.WithLockOut(5*time.Second),
+		circuit.WithEstimationFunc(circuit.Exponential),
+		circuit.WithIsExcluded(func(err error) bool {
+			// Don't count server-side throttling against the client breaker
+			return errors.Is(err, errServerThrottled)
+		}),
+	)
+	if err != nil {
+		log.Printf("failed to get breaker '%s': %v", name, err)
+		return
+	}
+
+	body, err := circuit.Run(breaker, context.Background(), func(ctx context.Context) (string, error) {
+		uri := fmt.Sprintf("%s?cb=%s&errchance=%d", serverAddr, name, 9000)
+		req, _ := http.NewRequestWithContext(ctx, http.MethodGet, uri, nil)
+		resp, err := httpClient.Do(req)
+		if err != nil {
+			return "", fmt.Errorf("request failed: %w", err)
+		}
+		defer resp.Body.Close()
+		data, _ := io.ReadAll(resp.Body)
+
+		switch resp.StatusCode {
+		case http.StatusOK:
+			return string(data), nil
+		case http.StatusTooManyRequests:
+			return string(data), errServerThrottled
+		case http.StatusServiceUnavailable:
+			return string(data), fmt.Errorf("server circuit open: %s", data)
+		case http.StatusGatewayTimeout:
+			return string(data), fmt.Errorf("server timeout: %s", data)
 		default:
-			<-time.After(100 * time.Millisecond)
-			breakernames := []string{
-				"foo",
-				"bar",
-				"baz",
-				"fizz",
-				"buzz",
-				"herp",
-				"derp",
-			}
-			breakerName := breakernames[rand.Intn(len(breakernames))]
-			breaker, _ := theBox.LoadOrCreate(circuit.BreakerOptions{
-				OpeningWillResetErrors: false,
-				Threshold:              3,
-				Timeout:                10 * time.Millisecond,
-				BaudRate:               0,
-				BackOff:                10 * time.Second,
-				Window:                 10 * time.Second,
-				LockOut:                5 * time.Second,
-				Name:                   breakerName,
-				EstimationFunc:         circuit.Exponential,
-			})
-
-			go func(breaker *circuit.Breaker, name string) {
-				_, err := breaker.Run(context.Background(), func(ctx context.Context) (interface{}, error) {
-					uri := fmt.Sprintf("http://localhost:8080?cb=%s&errchance=%d", name, 9000)
-					resp, err := http.DefaultClient.Get(uri)
-					if err != nil {
-						return []byte(nil), err
-					}
-					defer resp.Body.Close()
-					data, _ := ioutil.ReadAll(resp.Body)
-					switch resp.StatusCode {
-					case http.StatusOK, http.StatusPartialContent:
-						return data, nil
-					case http.StatusForbidden:
-						log.Printf("throttled on server: '%s'", name)
-						return data, errors.New("throttled")
-					case http.StatusInternalServerError:
-						log.Printf("open on server: '%s'", name)
-						return data, errors.New("open")
-					}
-					log.Println("wat")
-					return data, nil
-				})
-
-				if err != nil {
-					if errors.Is(err, circuit.StateOpenError) {
-						log.Printf("open on client: '%s'", name)
-					} else if errors.Is(err, circuit.StateThrottledError) {
-						log.Printf("throttled on client: '%s'", name)
-					} else {
-						log.Printf("error from inside breaker '%s': %v", name, err)
-					}
-				}
-
-			}(breaker, breakerName)
+			return string(data), fmt.Errorf("server error %d: %s", resp.StatusCode, data)
 		}
+	})
+
+	if err != nil {
+		switch {
+		case errors.Is(err, circuit.ErrStateOpen):
+			log.Printf("[%s] client breaker OPEN — request rejected", name)
+		case errors.Is(err, circuit.ErrStateThrottled):
+			log.Printf("[%s] client breaker THROTTLED — request shed", name)
+		case errors.Is(err, errServerThrottled):
+			log.Printf("[%s] server throttled (excluded from client tracking)", name)
+		default:
+			log.Printf("[%s] error: %v", name, err)
+		}
+		return
 	}
 
-END:
-	println("bye")
+	_ = body // success — quiet in logs to reduce noise
 }
+
+var errServerThrottled = errors.New("server throttled")

_functional_tests/scenarios/http/docker-compose.yml

@@ -0,0 +1,48 @@
+services:
+  server:
+    build:
+      context: ../../..
+      dockerfile: _functional_tests/scenarios/http/server/Dockerfile
+    ports:
+      - "8080:8080"
+      - "6060:6060"
+    healthcheck:
+      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/health"]
+      interval: 2s
+      timeout: 2s
+      retries: 10
+
+  client:
+    build:
+      context: ../../..
+      dockerfile: _functional_tests/scenarios/http/client/Dockerfile
+    environment:
+      - SERVER_ADDR=http://server:8080
+    depends_on:
+      server:
+        condition: service_healthy
+
+  prometheus:
+    image: prom/prometheus:v2.51.0
+    volumes:
+      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+    ports:
+      - "9090:9090"
+    depends_on:
+      server:
+        condition: service_healthy
+
+  grafana:
+    image: grafana/grafana:10.4.0
+    environment:
+      - GF_SECURITY_ADMIN_USER=admin
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
+    ports:
+      - "3000:3000"
+    depends_on:
+      - prometheus