Log redactor upgrade w shannon enthropy

Author: belisarh

README.md

@@ -108,7 +108,7 @@ class StripePaymentService
 
 ### Controlled Execution Blocks
 
-**What it does:** Monitors critical operations with automatic start/end logging, exception handling, circuit breakers, and failure callbacks.
+**What it does:** Monitors critical operations with automatic start/end logging, exception handling, DB transactions, circuit breakers, and failure callbacks.
 
 ```php
 use Kirschbaum\Monitor\Facades\Monitor;
@@ -118,14 +118,9 @@ class PaymentService
     public function processPayment($amount, $userId)
     {
         return Monitor::controlled('payment_processing')
-            ->context(['amount' => $amount, 'user_id' => $userId])
-            ->failing(function ($exception, $context) {
-                // Alert ops team immediately
-                NotificationService::alertOps('Payment failure', $context);
-                
-                // Open circuit breaker
-                CircuitBreaker::open('payment_service', '5 minutes');
-            })
+            ->with(['amount' => $amount, 'user_id' => $userId]) // Additional log context
+            ->transactioned(3)
+            ->escalated(fn (ControlledFailureMeta $meta) => Slack::notify($meta->toArray()))
             ->run(function () use ($amount) {
                 return $this->chargeCard($amount);
             });
@@ -145,9 +140,9 @@ class PaymentService
 ```
 
 **Advanced Features:**
-- **Circuit Breakers:** `->breaker('service_name', threshold, decaySeconds)`
-- **Database Transactions:** `->transactioned(retries, onlyExceptions, excludeExceptions)`
-- **Failure Escalation:** `->escalated($callback)` for critical business processes
+- **Circuit Breakers:** `->breaker('service_name', threshold, decaySeconds)` - Automatically opens/closes breaker based on failures
+- **Database Transactions:** `->transactioned(retries, onlyExceptions, excludeExceptions)`. This allows to only retry transaction on specific exceptions, or otherwise ignore specific exception.
+- **Failure Escalation Path:** `->escalated($callback)` for critical business processes
 
 ### Distributed Tracing
 
@@ -250,39 +245,90 @@ class DataProcessor
 
 ### Log Redaction
 
-**What it does:** Automatically scrubs sensitive data from log context to ensure compliance and security.
+**What it does:** Automatically scrubs sensitive data from log context using a priority-based system to ensure compliance and security while preserving important data.
+
+**Priority System:**
+1. **Safe Keys** (highest) - Never redacted, always shown
+2. **Blocked Keys** - Always redacted, regardless of content  
+3. **Regex Patterns** - Redacts values matching specific patterns
+4. **Shannon Entropy** (lowest) - Detects high-entropy secrets like API keys
 
 **Configuration:** Redaction options in `config/monitor.php`:
 
 ```php
 'log_redactor' => [
     'enabled' => true,
-    'redact_keys' => [
-        'password', 'token', 'api_key', 'authorization', 
-        'ssn', 'credit_card', 'private_key'
+    
+    // Priority 1: Keys that should NEVER be redacted
+    'safe_keys' => [
+        'id', 'uuid', 'created_at', 'updated_at', 'timestamp',
+        'user_id', 'order_id', 'status', 'type', 'name'
+    ],
+    
+    // Priority 2: Keys that should ALWAYS be redacted
+    'blocked_keys' => [
+        'password', 'token', 'api_key', 'authorization', 'secret',
+        'ssn', 'ein', 'credit_card', 'private_key', 'email'
     ],
+    
+    // Priority 3: Regex patterns for value-based detection
     'patterns' => [
         'email' => '/[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+/',
         'credit_card' => '/\b(?:\d[ -]*?){13,16}\b/',
-        'bearer_token' => '/Bearer\s+[A-Za-z0-9\-._~+\/]+=*/',
+        'ssn' => '/\b\d{3}-?\d{2}-?\d{4}\b/',
+        'phone' => '/\b\d{3}[.-]?\d{3}[.-]?\d{4}\b/',
+    ],
+    
+    // Priority 4: Shannon entropy detection for unknown secrets
+    'shannon_entropy' => [
+        'enabled' => true,
+        'threshold' => 4.5,    // Entropy threshold (0-8 scale)
+        'min_length' => 20,    // Minimum string length to analyze
     ],
+    
     'replacement' => '[REDACTED]',
+    'mark_redacted' => true,         // Add "_redacted": true marker
     'max_value_length' => 10000,     // Truncate large values
     'redact_large_objects' => true,  // Limit large arrays/objects
     'max_object_size' => 50,
 ],
 ```
 
-**What happens:**
+**How it works:**
 ```php
 Monitor::from($this)->info('User data', [
-    'email' => '[email protected]',    // → '[REDACTED]'
-    'password' => 'secret123',        // → '[REDACTED]'  
-    'token' => 'Bearer abc123',       // → '[REDACTED]'
-    'name' => 'John Doe'              // → 'John Doe' (unchanged)
+    // Safe keys - never redacted (Priority 1)
+    'id' => 123,                      // → 123 (safe key)
+    'user_id' => 456,                 // → 456 (safe key)
+    'created_at' => '2024-01-15',     // → '2024-01-15' (safe key)
+    
+    // Blocked keys - always redacted (Priority 2)  
+    'password' => 'secret123',        // → '[REDACTED]' (blocked key)
+    'email' => '[email protected]',    // → '[REDACTED]' (blocked key wins over pattern)
+    
+    // Pattern matching - value-based (Priority 3)
+    'contact' => '[email protected]',  // → '[REDACTED]' (email pattern)
+    'card' => '4111-1111-1111-1111',  // → '[REDACTED]' (credit card pattern)
+    
+    // Shannon entropy - high entropy secrets (Priority 4)
+    'api_token' => 'sk-1234567890abcdef...', // → '[REDACTED]' (high entropy)
+    'jwt' => 'eyJ0eXAiOiJKV1QiLCJhbGc...', // → '[REDACTED]' (high entropy)
+    
+    // Normal data - unchanged
+    'name' => 'John Doe',             // → 'John Doe' (low entropy, not blocked)
+    'description' => 'A simple task', // → 'A simple task' (normal text)
 ]);
+
+// Result includes redaction marker when data was modified
+// { ..., "_redacted": true }
 ```
 
+**Shannon Entropy Detection:**
+- Automatically detects API keys, JWT tokens, and other high-entropy secrets
+- Ignores common patterns like URLs, UUIDs, dates, and file paths
+- Configurable threshold and minimum length requirements
+- Prevents false positives on normal text and structured data
+
 ## Configuration
 
 **Environment Variables:**
@@ -304,6 +350,15 @@ MONITOR_TRACE_HEADER=X-Trace-Id
 # Log redaction
 MONITOR_LOG_REDACTOR_ENABLED=true
 MONITOR_LOG_REDACTOR_REPLACEMENT='[REDACTED]'
+MONITOR_LOG_REDACTOR_MARK_REDACTED=true
+MONITOR_LOG_REDACTOR_MAX_VALUE_LENGTH=10000
+MONITOR_LOG_REDACTOR_LARGE_OBJECTS=true
+MONITOR_LOG_REDACTOR_MAX_OBJECT_SIZE=50
+
+# Shannon entropy detection
+MONITOR_LOG_REDACTOR_SHANNON_ENABLED=true
+MONITOR_LOG_REDACTOR_SHANNON_THRESHOLD=4.5
+MONITOR_LOG_REDACTOR_SHANNON_MIN_LENGTH=20
 ```
 
 **Logging Channel:** Configure a dedicated Monitor logging channel:

composer.json

@@ -1,5 +1,5 @@
 {
-    "name": "kirschbaum-development/monitor",
+    "name": "kirschbaum/monitor",
     "description": "Laravel observability toolkit with critical control points, structured logging, performance timing, and trace context.",
     "type": "library",
     "license": "MIT",

config/monitor.php

@@ -279,30 +279,97 @@
 
         /*
         |----------------------------------------------------------------------
-        | Redact Keys
+        | Safe Keys
         |----------------------------------------------------------------------
         |
-        | Keys that always get redacted (case-insensitive match). If any
-        | context key matches these names, the entire value will be replaced
-        | with the configured replacement string.
+        | Keys that should NEVER be redacted (case-insensitive match).
+        | These keys will always show their values unredacted, regardless
+        | of other redaction rules. Useful for identifiers and timestamps.
         |
         */
 
-        'redact_keys' => [
+        'safe_keys' => [
+            // Core identifiers (high frequency)
+            'id',
+            'uuid',
+            'user_id',
+            'order_id',
+            'session_id',
+            'request_id',
+
+            // Timestamps & metadata (high frequency)
+            'created_at',
+            'updated_at',
+            'timestamp',
+
+            // Monitor framework keys (highest frequency)
+            'level',
+            'event',
+            'message',
+            'trace_id',
+            'channel',
+            'duration_ms',
+            'memory_mb',
+
+            // Controlled block keys (frequent in enterprise usage)
+            'controlled_block',
+            'controlled_block_id',
+            'attempt',
+            'status',
+            'breaker_tripped',
+            'escalated',
+
+            // Common business identifiers
+            'name',
+            'title',
+            'type',
+            'method',
+            'path',
+            'url',
+            'ip',
+            'user_agent',
+            'operation',
+            'action',
+            'source',
+            'target',
+            'version',
+            'platform',
+            'environment',
+        ],
+
+        /*
+        |----------------------------------------------------------------------
+        | Blocked Keys
+        |----------------------------------------------------------------------
+        |
+        | Keys that should ALWAYS be redacted (case-insensitive match).
+        | These keys will always be redacted, even if they would normally
+        | be considered safe. Takes priority over safe_keys.
+        |
+        */
+
+        'blocked_keys' => [
             'password',
-            'token',
             'secret',
+            'token',
             'api_key',
             'authorization',
-            'ssn',
-            'credit_card',
             'auth_token',
             'bearer_token',
             'access_token',
             'refresh_token',
             'session_id',
             'private_key',
             'client_secret',
+            'email',
+            'ssn',
+            'ein',
+            'social_security_number',
+            'tax_id',
+            'credit_card',
+            'card_number',
+            'cvv',
+            'pin',
         ],
 
         /*
@@ -317,15 +384,16 @@
         */
 
         'patterns' => [
+            // Ordered by frequency and performance (most common/fastest first)
             'email' => '/[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+/',
+            'phone_simple' => '/\b\d{3}[.-]?\d{3}[.-]?\d{4}\b/',
+            'ssn' => '/\b\d{3}-?\d{2}-?\d{4}\b/',
             'credit_card' => '/\b(?:\d[ -]*?){13,16}\b/',
-            'ssn' => '/\b\d{3}-\d{2}-\d{4}\b/',
-            'phone' => '/\+?\d[\d -]{8,14}\d/',
-            'bearer_token' => '/Bearer\s+[A-Za-z0-9\-._~+\/]+=*/',
-            'api_key' => '/(api|apikey|api_key)\s*[:=]\s*[A-Za-z0-9\-_]{20,}/i',
-            'jwt_token' => '/eyJ[A-Za-z0-9\-_]+\.eyJ[A-Za-z0-9\-_]+\.[A-Za-z0-9\-_.+\/=]*/',
-            'ipv4' => '/\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b/',
             'url_with_auth' => '/https?:\/\/[^:\/\s]+:[^@\/\s]+@[^\s]+/',
+
+            // Note: IPv4 removed (common pattern bypass handles this)
+            // Note: Complex phone patterns removed (Shannon entropy will catch international)
+            // Note: API key patterns removed (Shannon entropy is better for these)
         ],
 
         /*
@@ -360,10 +428,13 @@
         | Maximum length for string values before they are considered "large
         | blobs" and get redacted. Set to null to disable length-based redaction.
         | This helps prevent large payloads from cluttering logs.
+        | 20000: Performance optimized (allows larger strings, less truncation overhead)
+        | 10000: Balanced approach
+        | 5000: More aggressive truncation (better for storage-constrained environments)
         |
         */
 
-        'max_value_length' => env('MONITOR_LOG_REDACTOR_MAX_VALUE_LENGTH', 10000),
+        'max_value_length' => env('MONITOR_LOG_REDACTOR_MAX_VALUE_LENGTH', 20000),
 
         /*
         |----------------------------------------------------------------------
@@ -385,9 +456,47 @@
         |
         | Maximum number of items in an array or object before it gets redacted.
         | Only applies when redact_large_objects is enabled.
+        | 100: Performance optimized (allows larger objects, less redaction overhead)
+        | 50: Balanced approach
+        | 25: More aggressive redaction (better for memory-constrained environments)
         |
         */
 
-        'max_object_size' => env('MONITOR_LOG_REDACTOR_MAX_OBJECT_SIZE', 50),
+        'max_object_size' => env('MONITOR_LOG_REDACTOR_MAX_OBJECT_SIZE', 100),
+
+        /*
+        |----------------------------------------------------------------------
+        | Shannon Entropy Configuration
+        |----------------------------------------------------------------------
+        |
+        | Shannon entropy analysis for detecting high-entropy strings like
+        | API keys, tokens, and secrets that might not match specific patterns.
+        | This is used as a last resort after safe_keys, blocked_keys, and
+        | regex patterns have been checked.
+        |
+        */
+
+        'shannon_entropy' => [
+            /*
+            | Enable Shannon entropy analysis for detecting potential secrets
+            */
+            'enabled' => env('MONITOR_LOG_REDACTOR_SHANNON_ENABLED', true),
+
+            /*
+            | Entropy threshold (0.0 - ~8.0). Higher values = more selective.
+            | 4.8: Balanced performance/security (recommended for production)
+            | 4.5: More sensitive detection (better security, more CPU)
+            | 5.0: Higher performance (fewer false positives, less CPU)
+            */
+            'threshold' => env('MONITOR_LOG_REDACTOR_SHANNON_THRESHOLD', 4.8),
+
+            /*
+            | Minimum string length to analyze. Shorter strings are ignored.
+            | 25: Performance optimized (recommended for high-volume logging)
+            | 20: More sensitive detection
+            | 30: Higher performance, may miss some short tokens
+            */
+            'min_length' => env('MONITOR_LOG_REDACTOR_SHANNON_MIN_LENGTH', 25),
+        ],
     ],
 ];