You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: rules/cross-platform/reconnaissance_web_server_unusual_spike_in_error_logs.toml
+32Lines changed: 32 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -17,6 +17,37 @@ interval = "10m"
17
17
language = "esql"
18
18
license = "Elastic License v2"
19
19
name = "Potential Spike in Web Server Error Logs"
20
+
note = """ ## Triage and analysis
21
+
22
+
> **Disclaimer**:
23
+
> This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
24
+
25
+
### Investigating Potential Spike in Web Server Error Logs
26
+
27
+
This detection flags spikes of web server error responses across HTTP/TLS and common server platforms, signaling active scanning or fuzzing that can expose misconfigurations or exploitable paths. A typical pattern is an automated scanner sweeping endpoints like /admin/, /debug/, /.env, /.git, and backup archives while mutating query parameters, producing repeated 404/403 and occasional 500 responses across multiple applications within minutes.
28
+
29
+
### Possible investigation steps
30
+
31
+
- Pivot on the noisy client IP(s) to build a minute-by-minute timeline across affected hosts showing request rate, status codes, methods, and top paths to distinguish automated scanning from a localized application failure.
32
+
- Enrich the client with ASN, geolocation, hosting/Tor/proxy reputation, historical sightings, and maintenance windows to quickly decide if it matches a known external scanner or an internal scheduled test.
33
+
- Aggregate the most requested URIs and verbs and look for telltale patterns such as /.env, /.git, backup archives, admin consoles, or unusual verbs like PROPFIND/TRACE, then correlate any 5xx bursts with application and server error logs and recent deploys or config changes.
34
+
- Hunt for follow-on success from the same client by checking for subsequent 200/302s to sensitive paths, authentication events and session creation, or evidence of file writes and suspicious child processes on the web hosts.
35
+
- If traffic traverses a CDN/WAF/load balancer, pivot to those logs to recover true client IPs, review rule matches and throttling, and determine whether similar patterns occurred across multiple edges or regions.
36
+
37
+
### False positive analysis
38
+
39
+
- Internal QA or integration tests that systematically crawl application routes after a deployment can generate bursts of 404/403 and occasional 500s from a single client IP, closely resembling active scanning.
40
+
- A transient backend outage or misconfiguration (broken asset paths or auth flows) can cause legitimate traffic to return many errors aggregated under a shared egress IP (NAT), pushing per-IP counts above the threshold without adversary activity.
41
+
42
+
### Response and remediation
43
+
44
+
- Immediately block or throttle the noisy client IPs at the WAF/CDN and load balancer by enabling per-IP rate limits and signatures for scanner patterns such as repeated hits to /.env, /.git, /admin, backup archives, or unusual verbs like PROPFIND/TRACE.
45
+
- If errors include concentrated 5xx responses from one web host, drain that node from service behind the load balancer, capture its web and application error logs, and roll back the most recent deploy or config change until error rates normalize.
46
+
- Remove risky exposures uncovered by the scan by denying access to environment files and VCS directories (.env, .git), disabling directory listing, locking down admin consoles, and rejecting unsupported HTTP methods at the web server.
47
+
- Escalate to Incident Response if the same client shifts from errors to successful access on sensitive endpoints (200/302 to /admin, /login, or API keys), if you observe file writes under the webroot or suspicious child processes, or if multiple unrelated clients show the same pattern across regions.
48
+
- Recover service by redeploying known-good builds, re-enabling health checks, running smoke tests against top routes, and restoring normal WAF/CDN policies while keeping a temporary blocklist for the offending IPs.
49
+
- Harden long term by tuning WAF/CDN to auto-throttle bursty 404/403/500 patterns, disabling TRACE/OPTIONS where unused, minimizing verbose error pages, and ensuring logs capture the true client IP via X-Forwarded-For or True-Client-IP headers.
name = "Web Server Potential Spike in Error Response Codes"
20
+
note = """ ## Triage and analysis
21
+
22
+
> **Disclaimer**:
23
+
> This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
24
+
25
+
### Investigating Web Server Potential Spike in Error Response Codes
26
+
27
+
This rule detects bursts of 5xx errors (500–504) from GET traffic, highlighting abnormal server behavior that accompanies active scanning or fuzzing and exposes fragile code paths or misconfigured proxies. Attackers sweep common and generated endpoints while mutating query params and headers—path traversal, template syntax, large payloads—to repeatedly force backend exceptions and gateway timeouts, enumerate which routes fail, and pinpoint inputs that leak stack traces or crash components for follow-on exploitation.
28
+
29
+
### Possible investigation steps
30
+
31
+
- Plot error rates per minute by server and client around the alert window to confirm the spike, determine scope, and separate a single noisy client from a platform-wide issue.
32
+
- Aggregate the failing URL paths and query strings from the flagged client and look for enumeration sequences, traversal encoding, template injection markers, or oversized inputs indicative of fuzzing.
33
+
- Examine User-Agent, Referer, header mix, and TLS JA3 for generic scanner signatures or reuse across multiple clients, and enrich the originating IP with reputation and hosting-provider attribution.
34
+
- Correlate the timeframe with reverse proxy/WAF/IDS and application error logs or stack traces to identify which routes threw exceptions or timeouts and whether they align with the client’s input patterns.
35
+
- Validate backend and dependency health (upstreams, databases, caches, deployments) to rule out infrastructure regressions, then compare whether only the suspicious client experiences disproportionate failures.
36
+
37
+
### False positive analysis
38
+
39
+
- A scheduled deployment or upstream dependency issue can cause normal GET traffic to fail with 502/503/504, and many users egressing through a shared NAT or reverse proxy may be aggregated as one source IP that triggers the spike.
40
+
- An internal health-check, load test, or site crawler running from a single host can rapidly traverse endpoints and induce 500 errors on fragile routes, mimicking scanner-like behavior without malicious intent.
41
+
42
+
### Response and remediation
43
+
44
+
- Immediately rate-limit or block the originating client(s) at the edge (reverse proxy/WAF) using the observed source IPs, User-Agent/TLS fingerprints, and the failing URL patterns generating 5xx bursts.
45
+
- Drain the origin upstream(s) showing repeated 500/502/503/504 on the probed routes, roll back the latest deployment or config change for those services, and disable any unstable endpoint or plugin that is crashing under input fuzzing.
46
+
- Restart affected application workers and proxies, purge bad cache entries, re-enable traffic gradually with canary percentage, and confirm normal response rates via synthetic checks against the previously failing URLs.
47
+
- Escalate to Security Operations and Incident Response if 5xx spikes persist after blocking or if error pages expose stack traces, credentials, or admin route disclosures, or if traffic originates from multiple global hosting ASNs.
48
+
- Deploy targeted WAF rules for path traversal and injection markers seen in the URLs, enforce per-IP and per-route rate limits, tighten upstream timeouts/circuit breakers, and replace verbose error pages with generic responses that omit stack details.
49
+
- Add bot management and IP reputation blocking at the CDN/edge, lock down unauthenticated access to admin/debug routes, and instrument alerts that trigger on sustained 5xx bursts per client and per route with automatic edge throttling.
0 commit comments