BotGuard

BotGuard (HTTP L7 Protection)

Type: Module Layer: L2 — HTTP bot behavior (L7 classification, L3/L4 enforcement) Since: v1.80.x Config: conf.d/botguard/main.conf (.local override) Daemon dependency: YES — daemon required for scoring and set population

---

Purpose

See also: Glossary | Health Model | Architecture | Known Limitations

BotGuard is the HTTP-focused protection module that classifies and reacts to suspicious bot-like behavior. It uses a kernel helper chain enforcing decisions based on six classification sets, with daemon-side scoring logic. The kernel chain enforces decisions (allow, throttle, ban, emergency drop) based on set membership. The daemon watches HTTP traffic patterns, scores source IPs, and populates the classification sets.

---

How It Works

BotGuard operates across two layers:

Kernel layer (enforcement)

The http_bot_guard helper chain is jumped to from the input chain for TCP ports 80 and 443. It checks source IPs against six classification sets and applies the corresponding verdict:

TCP dport {80, 443} → jump http_bot_guard
    │
    ├── IP in http_bot_allow     → ACCEPT (verified crawler)
    ├── IP in http_bot_ban       → DROP (denied bot)
    ├── IP in http_bot_emergency → DROP (emergency block)
    ├── IP in http_bot_grey      → throttle (5/sec burst 10)
    │   ├── within limit → ACCEPT
    │   └── exceeded     → DROP
    ├── IP in http_bot_pending   → throttle (15/sec burst 30)
    │   ├── within limit → ACCEPT
    │   └── exceeded     → DROP
    └── new connection > 30/sec  → mark as SUSPECT (add to suspect set)
        return to input chain

Daemon layer (classification)

The Go daemon (nftband) runs the BotGuard module which:

1. Monitors HTTP connection patterns via kernel set population
2. Scores source IPs based on request rate, user-agent, behavior signals
3. Moves IPs between classification sets based on score thresholds:
   • High rate, unknown → SUSPECT (kernel auto-marks via meter)
   • Suspect + verification failed → PENDING → GREY → BAN
   • Suspect + verification passed → ALLOW
   • Emergency conditions → EMERGENCY

The daemon is the writer. The kernel is the enforcer. If the daemon stops, existing set entries (with kernel-managed timeouts) continue enforcing, but no new classifications or promotions occur.

---

Kernel Objects

Chain (1 per family, required when ENABLED)

Chain	Rules (typical)	Purpose
http_bot_guard	8	Classification enforcement for HTTP traffic

The chain must have > 0 rules. A chain with 0 rules is DEGRADED (B80-3).

Jump rule (required when ENABLED)

tcp dport { 80, 443 } jump http_bot_guard

This jump must be present in the input chain and must be reachable (not shadowed by a broad accept before it).

Classification sets (6 per family, required when ENABLED)

Set	Type	Timeout	Purpose	Evidence class
http_bot_suspect	hash, timeout	5m	IPs under initial observation	OBSERVATIONAL
http_bot_pending	hash, timeout	varies	IPs awaiting verification	OBSERVATIONAL
http_bot_allow	hash, timeout	varies	Verified non-bots (crawlers, legit)	OBSERVATIONAL
http_bot_grey	hash, timeout	varies	Throttled IPs (suspicious but not banned)	ENFORCEMENT
http_bot_ban	hash, timeout	varies	Banned bots	ENFORCEMENT
http_bot_emergency	hash, timeout	varies	Emergency blocks (severe abuse)	ENFORCEMENT

IPv6 variants use 6 suffix: http_bot_suspect6, http_bot_pending6, etc.

Total required objects when ENABLED: 6 sets + 1 chain + 1 jump rule per family = 16 objects for dual-family.

No dedicated named counters

BotGuard does not have dedicated named counters in the nftables schema. The chain uses inline counters referencing total_input_drop for drop verdicts, but there is no botguard_drop named counter.

Effective state is derived exclusively from set population (kernel-resident data), not counter values:

Evidence type	Source	What it proves
Enforcement sets populated (ban, grey, emergency)	Kernel set query	BotGuard is ENFORCING
Observation sets populated (suspect, pending)	Kernel set query	BotGuard is OBSERVING
All sets empty	Kernel set query	NEUTRAL (valid IDLE — no bot traffic)

---

Configuration

Key	File	Default	Meaning
HTTP_BOTGUARD_ENABLED	conf.d/botguard/main.conf	"false"	Master enable/disable

Override via main.conf.local (survives upgrades).

Additional tuning parameters are in conf.d/botguard/ configuration files. These control scoring thresholds, verification methods, and timeout durations.

---

State Model (v1.81)

This module follows the 4-axis model:

• Config: ENABLED / DISABLED
• Structural: PRESENT / MISSING (6 sets + chain + jump exist per active family)
• Runtime: RUNNING / STOPPED / ERROR (daemon required for module operation)
• Effective: ENFORCING / OBSERVING / IDLE (from set population evidence)

Set precedence (chain rule order)

emergency > ban > grey > pending > suspect > allow

The chain checks sets in this order. A match in an earlier set takes precedence — an IP in both ban and allow is dropped (ban checked first).

Effective state derivation

Set population	Effective state
Ban, grey, or emergency set > 0 elements	ENFORCING
Suspect or pending set > 0 elements (enforcement sets empty)	OBSERVING
All 6 sets empty	IDLE (NEUTRAL — no bot traffic, valid per BUG-3 lesson)

BUG-3 context (v1.79.1): Empty sets were previously misinterpreted as "module broken." This was fixed — empty sets with BotGuard ENABLED and low HTTP traffic = correct IDLE behavior.

Truth table

Config	Structural	Runtime	Effective	System Contribution
DISABLED	—	—	—	skip
ENABLED	PRESENT	RUNNING	ENFORCING	PROTECTED
ENABLED	PRESENT	RUNNING	OBSERVING	PROTECTED
ENABLED	PRESENT	RUNNING	IDLE	IDLE
ENABLED	PRESENT	STOPPED	—	DEGRADED (daemon required)
ENABLED	MISSING	—	—	DEGRADED (structural failure)

Daemon stopped behavior

When nftband stops:

• Existing set entries continue enforcing (kernel-managed timeouts)
• Banned IPs remain banned until timeout expires
• Throttled IPs remain throttled
• No new classifications — kernel rate tracking continues, but set population (suspect, pending, etc.) requires the daemon. No scoring or promotion occurs.
• System is DEGRADED, not DOWN — kernel enforcement persists

Daemon restart behavior

After daemon restart, BotGuard sets may be temporarily empty until the module re-registers and scoring resumes. This is a transient IDLE state if it resolves within one scoring cycle. Persistent empty sets on a high-traffic HTTP host after daemon restart indicate a problem (module not registered, scoring failure).

---

CLI Commands

nftban botguard status       # Show module state and set populations
nftban botguard enable       # Enable BotGuard
nftban botguard disable      # Disable BotGuard

---

Verification (MANDATORY)

# Check if chain exists (structural)
nft list chain ip nftban http_bot_guard
# Expected: chain with 8 rules (allow/ban/emergency/grey/pending/suspect)
# Empty chain = DEGRADED

# Check jump rule in input chain
nft list chain ip nftban input | grep "bot_guard"
# Expected: "tcp dport { 80, 443 } jump http_bot_guard"

# Check all 6 sets exist (structural)
for s in http_bot_suspect http_bot_pending http_bot_allow \
         http_bot_grey http_bot_ban http_bot_emergency; do
    nft list set ip nftban $s >/dev/null 2>&1 && echo "$s: PRESENT" || echo "$s: MISSING"
done
# All 6 must be PRESENT when ENABLED

# Check set population (effective state evidence)
nft -j list set ip nftban http_bot_ban | jq '.nftables[].set.elem | length'
# > 0 = ENFORCING. 0 or missing = check other enforcement sets

# Check module state via validator
nftban-validate --json | jq '.modules.botguard'
# Expected when idle: {"config":"enabled","structural":"present","runtime":"running","effective":"idle"}
# Expected when enforcing: effective:"enforcing"

# Check daemon is running (runtime axis)
systemctl is-active nftband
# Expected: "active"

---

Failure Modes

DEGRADED: ENABLED but sets or chain missing

Symptom: Validator reports structural: "missing" for BotGuard. Cause: Module enabled but kernel objects not loaded (failed enable, or objects lost after rebuild without BotGuard enabled — the BOTGUARD-REBUILD-UX bug fixed in v1.79.4). Fix: nftban botguard enable or nftban firewall rebuild

DEGRADED: ENABLED + PRESENT but daemon STOPPED

Symptom: Validator reports runtime: "stopped". Cause: nftband daemon not running. Impact: Existing kernel bans persist. No new classifications possible. Fix: systemctl start nftband

All sets empty on a high-traffic HTTP host

Symptom: BotGuard ENABLED + RUNNING but all 6 sets show 0 elements on a host with significant HTTP traffic. Possible causes:

• BotGuard module not registered in daemon (check journal for module_start: botguard)
• Scoring thresholds too high (no IP exceeds the 30/sec suspect marking rate)
• Jump rule shadowed by an earlier accept for ports 80/443

Diagnosis:

# Check module registration
journalctl -u nftband | grep -i botguard | tail -5

# Check if HTTP traffic reaches BotGuard chain
# (no direct counter — observe anchor_detect and SYN meter)
nft list counter ip nftban anchor_detect

Sets present but module DISABLED

When HTTP_BOTGUARD_ENABLED="false" but BotGuard chain and sets still exist in kernel, this is a valid residual state (same pattern as DDoS/Portscan). The chain continues processing HTTP traffic based on existing set entries until they expire. Not DEGRADED.

---

Limitations

• No dedicated named counter. BotGuard enforcement evidence comes from set population, not from a botguard_drop counter. The chain uses inline counters referencing total_input_drop which is a shared aggregate — it cannot attribute drops to BotGuard specifically.
• Set population requires per-set query. The validator runs nft -j list set per enforcement-critical set. This is one exec per set (up to 6 calls). The main nft -j list ruleset does not include set elements.
• Daemon required for classification. Without nftband, the kernel-side rate tracking continues, but adding IPs to classification sets requires the daemon. No scoring, verification, or promotion occurs.
• Jump placement sensitivity. The BotGuard jump (tcp dport {80,443} jump http_bot_guard) must be reachable in the input chain. If a broad accept rule for TCP 80/443 appears before the jump, BotGuard never sees HTTP traffic (shadowed). The validator checks jump presence but full shadowing analysis is best-effort.
• Timeout-based set expiry. All classification sets use kernel-managed timeouts. A banned IP is released when its timeout expires, regardless of whether the daemon believes it should remain banned. The daemon must re-evaluate and re-ban if the IP returns.