docs(madr): add 094 modern error handling strategy

Addresses inconsistency between MADR 073 (stdlib) and codebase (pkg/errors).
Recommends migrating to cockroachdb/errors for stack traces + performance.

Signed-off-by: Marcin Skalski <skalskimarcin33@gmail.com>

Author: Automaat

docs/madr/decisions/094-modern-error-handling.md

@@ -0,0 +1,293 @@
+# Modern Go Error Handling Strategy
+
+* Status: accepted
+
+## Context and Problem Statement
+
+Kuma currently has **inconsistent error handling** across the codebase:
+
+1. **MADR 073** (accepted 2023) recommends sentinel errors + `fmt.Errorf` with `%w`
+2. **Codebase reality**: Extensive use of `github.com/pkg/errors` (v0.9.1)
+   - 30+ usages of `errors.Wrap/Wrapf/Errorf` in `pkg/`
+   - Dependency in `go.mod`
+3. **New code** (e.g., `pkg/core/resources/store/error.go`) follows MADR 073 (stdlib)
+
+**Key developments since MADR 073:**
+- `github.com/pkg/errors` **archived in 2024** - no longer maintained
+- Security/maintenance risk for frozen dependency
+- Modern alternatives emerged with better performance
+- Stack traces proven valuable for Kuma debugging
+
+**Requirements:**
+- Stack traces for production debugging
+- Best performance (high-throughput control plane, many data planes)
+- Type-safe error checking (`errors.Is/As`)
+- Error wrapping with context
+- No migration timeline pressure
+
+## Design
+
+### Option 1: Standard Library Only (`fmt.Errorf` + `%w`)
+
+**Current MADR 073 recommendation**
+
+```go
+var ErrNotFound = errors.New("not found")
+
+func ErrorResourceNotFound(rt, name, mesh string) error {
+    return fmt.Errorf("resource %w: type=%q name=%q mesh=%q", ErrNotFound, rt, name, mesh)
+}
+
+func IsNotFound(err error) bool {
+    return errors.Is(err, ErrNotFound)
+}
+```
+
+#### Pros
+
+- Zero external dependencies
+- Actively maintained by Go team
+- Consistent with Go 1.13+ idioms
+- Works with `errors.Is/As`
+
+#### Cons
+
+- No stack traces - dealbreaker for debugging requirements
+- Limited error context extraction
+- Performance overhead with `errors.Is()` (up to 5x slower per benchmarks)
+
+### Option 2: Keep `github.com/pkg/errors`
+
+**Status quo**
+
+```go
+return errors.Wrapf(err, "failed to sync: zone=%s", zone)
+```
+
+#### Pros
+
+- Already in use - no migration
+- Stack traces included
+- Familiar to team
+
+#### Cons
+
+- Archived dependency - security/maintenance risk
+- Performance: 500% slowdown with `errors.Is()` benchmarks
+- Inconsistent with MADR 073
+- Dave Cheney (author): "I no longer use this package"
+
+### Option 3: Migrate to `github.com/cockroachdb/errors`
+
+**Modern, actively maintained alternative**
+
+```go
+import "github.com/cockroachdb/errors"
+
+// Drop-in replacement
+return errors.Wrapf(err, "failed to sync: zone=%s", zone)
+
+// Stack traces automatic
+errors.WithStack(ErrNotFound)
+
+// Works with stdlib
+errors.Is(err, ErrNotFound)  // ✓
+errors.As(err, &targetErr)   // ✓
+```
+
+#### Pros
+
+- Stack traces with full call context
+- Better performance than pkg/errors
+- Drop-in replacement - minimal code changes
+- Actively maintained (CockroachDB production use)
+- Stdlib compatible (`errors.Is/As` work correctly)
+- Rich features: error formatting, PII redaction, error barriers
+- Structured error metadata
+- Production-proven at scale
+
+#### Cons
+
+- External dependency (but from reputable source)
+- Migration effort for existing `fmt.Errorf` usage
+- Slightly larger binary size
+
+### Option 4: Hybrid Approach (Custom Types + Sentinels)
+
+**Enhanced structured errors**
+
+```go
+type ResourceError struct {
+    sentinel error
+    Type     model.ResourceType
+    Name     string
+    Mesh     string
+    stack    []uintptr  // manual stack capture
+}
+
+func (e *ResourceError) Error() string {
+    return fmt.Sprintf("resource %s: type=%q name=%q mesh=%q",
+        e.sentinel, e.Type, e.Name, e.Mesh)
+}
+```
+
+#### Pros
+
+- Full control over error structure
+- Extractable metadata
+
+#### Cons
+
+- Significant implementation effort (stack capture, formatting)
+- Reinventing the wheel
+- Maintenance burden
+- Won't match cockroachdb/errors features
+
+## Security implications and review
+
+**Archived dependency risk:**
+- `pkg/errors` receives no security updates
+- Future CVEs won't be patched
+- Recommendation: Migrate away
+
+**cockroachdb/errors:**
+- Active security maintenance
+- Used in production by CockroachDB (security-conscious)
+- Regular updates and CVE monitoring
+
+## Reliability implications
+
+**Performance (Critical for Kuma CP):**
+- Control plane handles many data planes
+- Error handling in hot paths (xDS updates)
+- Benchmarks show `pkg/errors` + `errors.Is()` = 5x slowdown
+- `cockroachdb/errors` optimized for performance
+
+**Debugging:**
+- Stack traces essential for multi-zone issues
+- Error context helps diagnose cross-component failures
+- Structured errors improve observability/telemetry
+
+**Error propagation:**
+- KDS (multi-zone sync) propagates errors between CPs
+- Stack traces help debug distributed failures
+- Consistent error wrapping improves troubleshooting
+
+## Implications for Kong Mesh
+
+Kong Mesh inherits Kuma's error handling:
+- Stack traces valuable for enterprise debugging
+- Performance improvements benefit enterprise deployments
+- Migration can be synchronized with Kuma changes
+- Error telemetry integration potential
+
+## Decision
+
+**Adopt Option 3: Migrate to `github.com/cockroachdb/errors`**
+
+### Rationale
+
+Meets all requirements:
+1. Stack traces (required)
+2. Better performance than pkg/errors (required)
+3. Drop-in replacement for existing code (low migration cost)
+4. Actively maintained (eliminates security risk)
+5. Stdlib compatible (future-proof)
+
+### Migration Strategy
+
+**Phase 1: Add dependency**
+```bash
+go get github.com/cockroachdb/errors
+```
+
+**Phase 2: Update imports (gradual)**
+```go
+// Old
+import "github.com/pkg/errors"
+
+// New
+import "github.com/cockroachdb/errors"
+```
+
+**Phase 3: Update MADR 073**
+- Document cockroachdb/errors as standard
+- Update examples
+- Keep sentinel error pattern
+
+**Phase 4: Convert fmt.Errorf usage**
+```go
+// Old (no stack trace)
+return fmt.Errorf("sync failed: %w", err)
+
+// New (with stack trace)
+return errors.Wrap(err, "sync failed")
+```
+
+**No timeline pressure** - can be done incrementally across releases.
+
+### Usage Convention
+
+```go
+// Sentinel errors (preserved from MADR 073)
+var ErrNotFound = errors.New("not found")
+
+// Error construction with context
+func ErrorResourceNotFound(rt model.ResourceType, name, mesh string) error {
+    return errors.Wrapf(ErrNotFound,
+        "resource: type=%q name=%q mesh=%q", rt, name, mesh)
+}
+
+// Error checking (unchanged)
+if errors.Is(err, ErrNotFound) {
+    // handle
+}
+
+// Extract structured errors
+var resourceErr *ResourceError
+if errors.As(err, &resourceErr) {
+    log.Error("failed", "type", resourceErr.Type)
+}
+```
+
+## Notes
+
+**Alternative considered:**
+- `gitlab.com/tozd/go/errors` - similar features, less adoption
+- CockroachDB chosen for production track record
+
+**Stack trace overhead:**
+- Only captured on error creation (not propagation)
+- Negligible for error-case performance
+- Can be disabled in production if needed (env var)
+
+## References
+
+### Go Error Handling Best Practices
+- [Mastering Go Error Handling: A Practical Guide - DEV Community](https://dev.to/leapcell/mastering-go-error-handling-a-practical-guide-3411)
+- [Best Practices for Error Handling in Go - JetBrains Guide](https://www.jetbrains.com/guide/go/tutorials/handle_errors_in_go/best_practices/)
+- [Mastering Error Handling in Go: Best Practices and Patterns - SkyCode](https://vickychhetri.com/2024/11/19/mastering-error-handling-in-go-best-practices-and-patterns/)
+- [A practical guide to error handling in Go - Datadog](https://www.datadoghq.com/blog/go-error-handling/)
+
+### Sentinel Errors vs Custom Types
+- [Error Values vs Sentinel Errors in Go - Medium](https://medium.com/@AlexanderObregon/error-values-vs-sentinel-errors-in-go-a377c3d831d1)
+- [Go Error Handling Techniques: Exploring Sentinel Errors, Custom Types, and Client-Facing Errors](https://arashtaher.wordpress.com/2024/09/05/go-error-handling-techniques-exploring-sentinel-errors-custom-types-and-client-facing-errors/)
+- [Go Error Handling: Sentinel vs Custom Types - alesr](https://alesr.github.io/posts/go-errors/)
+
+### Error Wrapping
+- [Working with Errors in Go 1.13 - Official Go Blog](https://go.dev/blog/go1.13-errors)
+- [Error Wrapping with Go's Standard Library - Medium](https://medium.com/@AlexanderObregon/error-wrapping-with-gos-standard-library-0a345eeea019)
+- [Error wrapping in Go - Bitfield Consulting](https://bitfieldconsulting.com/posts/wrapping-errors)
+- [Wrapping errors in Go - barney's tech blog](https://southcla.ws/wrapping-errors-in-go-a-new-approach)
+
+### fmt.Errorf vs pkg/errors
+- [What's the difference between errors.Wrapf(), errors.Errorf(), and fmt.Errorf()? - Stack Overflow](https://stackoverflow.com/questions/61933650/whats-the-difference-between-errors-wrapf-errors-errorf-and-fmt-errorf)
+- [Can new Go errors wrapper replace pkg/errors? - Dmitry Harnitski's Blog](https://blog.dharnitski.com/2019/09/09/go-errors-are-not-pkg-errors/)
+- [decide on the future use of errors, pkg/errors - moby/moby Discussion](https://github.com/moby/moby/discussions/46358)
+
+### Performance
+- [Sentinel errors and errors.Is() slow your code down by 500% - DoltHub Blog](https://www.dolthub.com/blog/2024-05-31-benchmarking-go-error-handling/)
+
+### Modern Alternatives
+- [Beyond fmt.Errorf() - CockroachDB errors library](https://dr-knz.net/cockroachdb-errors-everyday.html)
+- [cockroachdb/errors - Go Packages](https://pkg.go.dev/github.com/cockroachdb/errors)