Release v1.0.1: Add max file size flag, enhance pattern accuracy, and introduce new detection patterns for improved secret detection

Author: rafabd1

.github/workflows/release.yml

@@ -56,23 +56,19 @@ jobs:
                 # SecretHound v${{ env.VERSION }}
 
                 ## New Features
-                - **Expanded Pattern Library**: Introduced new pattern categories including PII (Personally Identifiable Information) and Web3 (e.g., Ethereum/Bitcoin addresses, private keys), increasing total patterns to over 60.
-                - **URL/Domain Extraction Mode**: Added `--scan-urls` flag to exclusively scan for URL and domain patterns, overriding other category filters.
-                - **Grouped Output Format**: Introduced `--group-by-source` flag to group found secrets by their source URL/file in TXT and JSON output formats, improving readability for large scans.
-                - **Pattern Category Control**: Implemented `--include-categories` and `--exclude-categories` flags to allow users to specify which pattern categories to use or ignore during scans.
+                - **Max File Size Flag**: Added `--max-file-size` flag to set the maximum file size for local file scanning.
+                - **Netlify Access Token Pattern**: Added new detection pattern for Netlify Access Tokens.
 
                 ## Improvements
-                - **Enhanced Pattern Accuracy**: Iteratively refined numerous existing patterns (IPv4, Bitcoin Address, Email Address, MAC Address, Generic Domain, Session Token) to significantly reduce false positives and improve detection of legitimate secrets based on extensive real-world test cases.
-                - **Log Custom Headers**: Initial configuration log now indicates if custom HTTP headers (`-H`) are being used.
-                - **Queue Logic & Rate Limiting**: Improved URL processing queue logic and refined the auto mode for rate limiting for more efficient and considerate scanning.
-                - **CLI Options Refinement**: Corrected and improved behavior of `--silent` and `--no-progress` flags.
-                - **Regex Engine Compatibility**: Added internal logging for regex compilation errors and refactored incompatible regex syntax (e.g., unsupported lookaheads) to ensure all patterns load correctly with Go's standard regex engine.
+                - **Enhanced Pattern Accuracy**: Refined multiple regex patterns (MAC Address, IPv4, IPv6, PayPal, Private Key, Phone Number) to significantly reduce false positives based on real-world testing.
 
                 ## Bug Fixes
-                - **JSON Output Formatting**: Addressed issues to ensure valid JSON output, especially when no secrets are found or in raw mode.
-                - **Progress Bar Rendering**: Fixed a bug where the progress bar would sometimes only update when new logs were printed, ensuring it now refreshes independently and consistently.
-                - **Execution Deadlocks**: Resolved potential deadlocks and improved goroutine management for more stable execution during long scans.
-                - **Pattern Loading**: Fixed an issue where the incorrect number of loaded patterns was reported when using category filters, ensuring accurate reflection of active patterns.
+                - Fixed false positives where SVG path data was detected as MAC addresses.
+                - Fixed false positives where OIDs were detected as IPv4 addresses.
+                - Fixed false positives where SHA-256 fingerprints were detected as IPv6 addresses.
+                - Fixed false positives where CSS class names were detected as PayPal credentials.
+                - Fixed false positives where event tracking strings were detected as private keys.
+                - Fixed Private Key Content pattern to require actual key data after BEGIN header.
 
                 ## Installation
                 

CHANGELOG.md

@@ -1,5 +1,28 @@
 # SecretHound Changelog
 
+## v1.0.1 (2025-12-05)
+
+### New Features
+- **Max File Size Flag**: Added `--max-file-size` flag to set the maximum file size for local file scanning, allowing users to skip large files that may slow down scans.
+- **Netlify Access Token Pattern**: Added new detection pattern for Netlify Access Tokens.
+
+### Improvements
+- **Enhanced Pattern Accuracy**: Refined multiple regex patterns to significantly reduce false positives:
+  - **MAC Address**: Now requires explicit keywords (`mac_address`, `ethernet_addr`, `hw_addr`) and only matches colon-separated format to avoid false positives from SVG paths.
+  - **IPv4 Address**: More restrictive pattern requiring explicit keywords (`ip_addr`, `host_addr`, `server_ip`). Added exclusions for OIDs (`1.3.6.1`, `2.16.840`).
+  - **IPv6 Address**: Simplified regex requiring `ipv6` or `ip6` keywords. Added exclusions for SHA-256 fingerprints.
+  - **PayPal/Braintree**: Now requires specific keywords (`paypal_client_id`, `braintree_secret`) instead of loose matching. Added exclusions for CSS class names.
+  - **Private Key Variable**: Added exclusions for tracking/event patterns (`click_`, `export_`, `track_`).
+  - **Phone Number**: More restrictive US format pattern, now keyword-dependent.
+
+### Bug Fixes
+- Fixed false positives where SVG path data was being detected as MAC addresses.
+- Fixed false positives where OIDs (Object Identifiers) were being detected as IPv4 addresses.
+- Fixed false positives where SHA-256 fingerprints were being detected as IPv6 addresses.
+- Fixed false positives where CSS class names containing "paypal" were being detected as PayPal credentials.
+- Fixed false positives where event tracking strings were being detected as private keys.
+- Fixed Private Key Content pattern to require actual key data after the BEGIN header, preventing false positives from standalone headers.
+
 ## v1.0.0 (2025-05-06)
 
 ### New Features

README.md

@@ -117,6 +117,7 @@ SecretHound supports the following options:
 | `--include-categories` | Comma-separated list of pattern categories to include (e.g., aws,gcp). | all enabled |
 | `--exclude-categories` | Comma-separated list of pattern categories to exclude (e.g., pii,url). | none |
 | `--scan-urls` | URL Extraction Mode: Scan ONLY for URL/Endpoint patterns (overrides category filters). | false |
+| `--max-file-size` | Maximum file size to scan in MB (0 for no limit). | 10 |
 | `--list-patterns` | List available pattern categories and patterns, then exit. | false |
 | `-v, --verbose` | Enable verbose logging output. | false |
 | `-n, --no-progress` | Disable the progress bar display. | false |

cmd/version.go

@@ -10,7 +10,7 @@ import (
 
 // Build information
 var (
-	Version = "1.0.0"
+	Version = "1.0.1"
 )
 
 // versionCmd represents the version command

core/patterns/patterns.go

@@ -176,11 +176,11 @@ var DefaultPatterns = &PatternDefinitions{
 			MinLength:   70,
 		},
 		"private_key_content": {
-			Regex:       `-----BEGIN (?:RSA|OPENSSH|DSA|EC|PGP) PRIVATE KEY( BLOCK)?-----`,
-			Description: "Private Key Content (BEGIN Block)",
+			Regex:       `-----BEGIN (?:RSA |OPENSSH |DSA |EC |PGP |ENCRYPTED )?PRIVATE KEY(?:\sBLOCK)?-----[\s]*[A-Za-z0-9+/=]{20,}`,
+			Description: "Private Key Content (with actual key data)",
 			Enabled:     true,
 			Category:    "crypto",
-			MinLength:   30,
+			MinLength:   60,
 		},
 		"square_access_token": {
 			Regex:       `sq0atp-[0-9A-Za-z\-_]{22}`,

docs/SUPPORTED_SECRETS.md

@@ -41,24 +41,25 @@ secrethound --list-patterns
 - PGP Private Keys
 - SSH Private Keys
 
-### Personally Identifiable Information (PII) - *New Category*
-- Email Addresses
-- Phone Numbers (various international formats)
-- Credit Card Numbers (heuristic based, common prefixes)
-- Social Security Numbers (SSN - US format, heuristic)
-
-### Web3 & Cryptocurrency - *New Category*
+### Personally Identifiable Information (PII)
+- Email Addresses (keyword-dependent)
+- Phone Numbers (US format, keyword-dependent)
+- IP Addresses (IPv4/IPv6, keyword-dependent with OID exclusions)
+- MAC Addresses (keyword-dependent, colon format only)
+- US ZIP Codes (keyword-dependent)
+- Serial Numbers (keyword-dependent)
+
+### Web3 & Cryptocurrency
 - Ethereum Addresses
 - Ethereum Private Keys
 - Bitcoin Addresses (P2PKH, P2SH, Bech32)
 - Bitcoin Private Keys (WIF format)
 - Generic Cryptocurrency Private Keys (common hex patterns)
 
 ### Network & Infrastructure
-- IP Addresses (IPv4, IPv6)
-- MAC Addresses
 - Generic Domain Names / Hostnames
 - URLs with potentially sensitive parameters or paths
+- Netlify Access Tokens
 
 ### Generic & Miscellaneous
 - Generic High Entropy Strings (potential secrets)
@@ -68,6 +69,8 @@ secrethound --list-patterns
 
 This list is continuously updated. Always refer to `secrethound --list-patterns` for the most current set of patterns and their categories.
 
+> **Note**: Many patterns are now "keyword-dependent", meaning they only match when specific keywords (like `ip_addr`, `mac_address`, `phone`, etc.) are found near the value. This significantly reduces false positives.
+
 ## API Keys and Tokens
 
 | Secret Type | Description | Example Pattern |
@@ -90,7 +93,7 @@ This list is continuously updated. Always refer to `secrethound --list-patterns`
 | Stripe Test Publishable Key | Stripe test publishable key | `pk_test_[0-9a-zA-Z]{24,34}` |
 | Square Access Token | Square OAuth token | `sq0atp-[0-9A-Za-z\-_]{22}` |
 | Square OAuth Secret | Square OAuth secret | `sq0csp-[0-9A-Za-z\-_]{43}` |
-| PayPal/Braintree | PayPal/Braintree credentials | `(?i)(?:paypal\|braintree).{0,20}['\"][A-Za-z0-9_-]{20,64}['\"]` |
+| PayPal/Braintree Client ID | PayPal/Braintree Client ID (keyword-dependent) | `(?i)(?:paypal\|braintree)[_-]?(?:client[_-]?)?(?:id\|key\|secret)\s*[:=]\s*['"](...)['"` |
 | Braintree Token | Braintree access token | `access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}` |
 
 ## Email and Communication Services
@@ -138,7 +141,7 @@ This list is continuously updated. Always refer to `secrethound --list-patterns`
 | OAuth 2.0 Access Token | OAuth 2.0 access token | `ya29\.[0-9A-Za-z\-_]+` |
 | Generic Password | Password in configuration | `(?i)(?:password\|passwd\|pwd\|secret)[\s]*[=:]+[\s]*["']([^'"]{8,30})["']` |
 | Authentication Token | Authentication token with comment | `['"]?([a-zA-Z0-9_\-\.]{32,64})['"]?\s*[,;]?\s*\/\/\s*[Aa]uth(?:entication)?\s+[Tt]oken` |
-| Private Key Variable | Private key variable | `['"]?(?:private_?key\|secret_?key)['"]?\s*[:=]\s*['"]([^'"]{20,})['"]` |
+| Private Key Variable | Private key variable (excludes tracking events) | `['"?(?:private_?key\|secret_?key)['"?\s*[:=]\s*['"(...)['"]` |
 | Encryption Key | Encryption key | `(?i)['"]?enc(?:ryption)?[_-]?key['"]?\s*[=:]\s*['"]([a-zA-Z0-9+/]{16,64})['"]` |
 | Signing Key | Signing key/secret | `(?i)['"]?sign(?:ing)?[_-]?(?:secret\|key)['"]?\s*[=:]\s*['"]([a-zA-Z0-9+/]{16,64})['"]` |
 
@@ -159,21 +162,33 @@ This list is continuously updated. Always refer to `secrethound --list-patterns`
 
 | Secret Type | Description | Example Pattern |
 |-------------|-------------|-----------------|
-| Private Key Content | Private key content | `-----BEGIN (?:RSA\|OPENSSH\|DSA\|EC\|PGP) PRIVATE KEY( BLOCK)?-----` |
+| Private Key Content | Private key with actual key data | `-----BEGIN (?:RSA \|OPENSSH \|...) PRIVATE KEY-----[\s]*[A-Za-z0-9+/=]{20,}` |
 
 ## CI/CD and DevOps
 
 | Secret Type | Description | Example Pattern |
 |-------------|-------------|-----------------|
-| Jenkins API Token | Jenkins API token | `(?i)(?:jenkins\|hudson).{0,5}(?:api)?.{0,5}(?:token).{0,5}['\"]([0-9a-zA-Z]{30,})['\"]` |
+| Jenkins API Token | Jenkins API token | `(?i)(?:jenkins\|hudson).{0,5}(?:api)?.{0,5}(?:token).{0,5}['"]([0-9a-zA-Z]{30,})['"]` |
 | NPM Access Token | NPM access token | `npm_[A-Za-z0-9]{36}` |
 | Docker Hub Personal Access Token | Docker Hub personal access token | `dckr_pat_[A-Za-z0-9_-]{56}` |
 | GitLab Runner Token | GitLab runner registration token | `glrt-[0-9a-zA-Z_\-]{20,}` |
 | GitLab Personal Token | GitLab personal access token | `glpat-[0-9a-zA-Z_\-]{20,}` |
+| Netlify Access Token | Netlify personal access token | `nf[pcfub]_[a-zA-Z0-9_\-]{36}` |
 
 ## Generic Secret Patterns
 
 | Secret Type | Description | Example Pattern |
 |-------------|-------------|-----------------|
 | Generic API Key | Generic API key format | `['"]?(?:api_?key\|api_?secret\|app_?key\|app_?secret)['"]?\s*[=:]\s*['"]([a-zA-Z0-9_\-\.]{16,64})['"]` |
 | API Key Assignment | API key assignment | `['"]?(?:api_?key\|api_?secret\|app_?key\|app_?secret)['"]?\s*[=:]\s*['"]([a-zA-Z0-9_\-\.]{16,64})['"]` |
+
+## PII (Personally Identifiable Information)
+
+| Secret Type | Description | Example Pattern |
+|-------------|-------------|-----------------|
+| Email Address | Email address (keyword-dependent) | `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}` |
+| Phone Number | Phone number (US format, keyword-dependent) | Requires keywords like `phone`, `mobile`, `tel` |
+| IPv4 Address | IPv4 address (keyword-dependent) | Requires keywords like `ip_addr`, `host_addr`, `server_ip` |
+| IPv6 Address | IPv6 address (keyword-dependent) | Requires keywords like `ipv6`, `ip6` |
+| MAC Address | MAC address (keyword-dependent, colon format) | Requires keywords like `mac_address`, `ethernet_addr`, `hw_addr` |
+| US ZIP Code | US ZIP code (keyword-dependent) | Requires keywords like `zip`, `postal`, `postcode` |

docs/USAGE.md

@@ -88,6 +88,7 @@ SecretHound supports the following options:
 | `-n, --concurrency` | Number of concurrent workers | 10 |
 | `-l, --rate-limit` | Requests per second per domain (0 = auto) | 0 |
 | `-H, --header` | Custom HTTP header (format: 'Name: Value') | - |
+| `--max-file-size` | Maximum file size to scan in MB (0 for no limit) | 10 |
 | `--insecure` | Disable SSL/TLS certificate verification | false |
 | `-v, --verbose` | Enable verbose output | false |