AbuseIPDB Web Server IP Blocker

A high-performance, enterprise-grade IP blocklist manager for Apache and Nginx web servers. Blocks malicious IPs at the application level using efficient, web-server-specific methods.

Version 2.0.0 - Now with Nginx Support! Modular Architecture with Enhanced Security & Performance

Features

✅ Apache & Nginx Support - Auto-detects and configures for your web server ✅ Multiple Blocklist Sources - Aggregate IPs from multiple sources automatically
✅ Automatic Deduplication - Removes duplicate IPs across all sources
✅ CIDR Range Support - Automatically expands CIDR notation (192.0.2.0/24) to individual IPs
✅ Cloudflare Protection - Automatically whitelists Cloudflare IP ranges with caching
✅ Custom Whitelist - Protect your own IPs from being blocked
✅ Smart IP Extraction - Handles both structured and unstructured data
✅ IPv4 & IPv6 Support - Full CIDR matching for both IP versions
✅ High Performance - Uses binary DBM files for fast lookups (< 1ms)
✅ Scalable - Tested with 100,000+ IPs, can handle millions with proper configuration
✅ Comprehensive Logging - Track all operations with timestamps and verbosity levels
✅ Test Mode - Safely test blocking before going live
✅ Statistics Dashboard - View blocklist metrics at a glance
✅ Modular Architecture - Clean separation of concerns for maintainability
✅ Security Hardened - Input validation, secure downloads, proper file permissions
✅ Resource Monitoring - Memory/CPU usage tracking for large operations
✅ Parallel Processing - Multi-core optimization for faster updates
✅ Caching System - Smart caching of Cloudflare ranges and downloads

Requirements

Common Requirements (Both Apache & Nginx)

Bash 4.0+ (for CIDR range expansion) - pre-installed on all modern systems
Python 3.6+ (recommended for full IPv6 CIDR support and fast filtering)
Cloudflare proxy (or any proxy that sets X-Forwarded-For headers) - optional but recommended
Root/sudo access

Apache-Specific

Apache2 with mod_rewrite and mod_remoteip enabled
httxt2dbm utility (usually included with Apache/apache2-utils)

Nginx-Specific

Nginx with geo module (built-in by default)
ngx_http_realip_module (built-in by default)

Minimal external dependencies! Pure bash core with optional Python enhancement.

Installation

# One-line installer
bash <(curl -s https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/install.sh)

Option A: Quick Installation (Recommended)

# Create installation directory
mkdir -p ~/tools/abuseipdeny
cd ~/tools/abuseipdeny

# Download all script components
curl -o abuseip-blocker.sh \
  https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/abuseip-blocker.sh

curl -o config.sh \
  https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/config.sh

curl -o utils.sh \
  https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/utils.sh

curl -o whitelist.sh \
  https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/whitelist.sh

curl -o update.sh \
  https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/update.sh

curl -o main.sh \
  https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/main.sh

curl -o ipv6_filter.py \
  https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/ipv6_filter.py

curl -o ipv4_filter.py \
  https://raw.githubusercontent.com/mayankguptadotcom/abuseip-deny-script/main/ipv4_filter.py

# Make all scripts executable
chmod +x *.sh *.py

# Create symlink for system-wide access
sudo ln -sf ~/tools/abuseipdeny/abuseip-blocker.sh /usr/local/bin/abuseip-blocker

# Verify installation
abuseip-blocker

Option B: Git Clone (Best for Updates)

# Clone the repository
mkdir -p ~/tools
cd ~/tools
git clone https://github.com/mayankguptadotcom/abuseip-deny-script.git abuseipdeny
cd abuseipdeny

# Make scripts executable
chmod +x *.sh *.py

# Create symlink
sudo ln -sf ~/tools/abuseipdeny/abuseip-blocker.sh /usr/local/bin/abuseip-blocker

# Verify installation
abuseip-blocker

# To update in the future:
cd ~/tools/abuseipdeny && git pull

Option C: Development Mode (For Contributors)

If you're developing or testing changes:

# Clone your fork or the repo
cd ~/Documents/projects
git clone https://github.com/mayankguptadotcom/abuseip-deny-script.git
cd abuseip-deny-script

# Make scripts executable
chmod +x *.sh *.py

# Create symlink directly to your dev directory
sudo ln -sf ~/Documents/projects/abuseip-deny-script/abuseip-blocker.sh /usr/local/bin/abuseip-blocker

# Now any changes you make are immediately available
abuseip-blocker

# After making changes, test immediately
# No need to reinstall!

Note: The modular version provides better maintainability, security, and features.

Usage

Update the Blocklist

Downloads from all configured sources, deduplicates, filters whitelisted IPs, and updates Apache:

sudo abuseip-blocker update

Recommended: Set up a cron job to update daily:

# Edit root's crontab
sudo crontab -e

# Add this line to update at 3 AM daily
0 3 * * * /usr/local/bin/abuseip-blocker update

Test the Blocker

Temporarily block an IP to verify everything works:

sudo abuseip-blocker test

This will:

Ask for an IP to block (use your phone's IP)
Block it temporarily
Let you test from that device (you should see 403 Forbidden)
Restore the original list after you press Enter

Manage Whitelist

Add your IP to the whitelist:

sudo abuseip-blocker whitelist-add 203.0.113.45

Remove from whitelist:

sudo abuseip-blocker whitelist-remove 203.0.113.45

View all whitelisted IPs:

sudo abuseip-blocker whitelist-list

Note: After modifying the whitelist, run update to regenerate the blocklist.

View Statistics

sudo abuseip-blocker stats

Shows:

Number of blocked IPs
Number of whitelisted IPs
Cloudflare protected ranges
Last update time
Log file location

Configuration

The script uses a modular configuration system. Edit ~/tools/abuseipdeny/config.sh to customize settings:

Add Multiple Blocklist Sources

declare -a BLOCKLIST_SOURCES=(
    "https://raw.githubusercontent.com/borestad/blocklist-abuseipdb/refs/heads/main/abuseipdb-s100-30d.ipv4"
    "https://example.com/another-blocklist.txt"
    "https://example.com/spammer-ips.txt"
)

File Locations

BLOCKLIST_TXT_FILE="/etc/apache2/abuseipdb_blocklist.txt"      # Master text list
BLOCKLIST_DBM_FILE="/etc/apache2/abuseipdb_blocklist.dbm"      # Binary DBM file
WHITELIST_FILE="/etc/apache2/abuseipdb_whitelist.txt"          # Your custom whitelist
CLOUDFLARE_WHITELIST_FILE="/etc/apache2/cloudflare_ips.txt"    # Auto-updated CF ranges
LOG_FILE="/var/log/abuseipdb_blocker.log"                      # Operation logs

Performance & Security Settings

MAX_CIDR_SIZE=24              # Max CIDR prefix to expand (24 = up to 256 IPs)
ENABLE_PARALLEL_SORT=true     # Use parallel sorting for large lists
MAX_DOWNLOAD_SIZE=10485760    # Max download size (10MB)
CLOUDFLARE_CACHE_HOURS=24     # Cache Cloudflare ranges for 24 hours
VERBOSE=false                 # Enable detailed logging

CIDR Configuration

The script can expand CIDR ranges (e.g., 192.0.2.0/24) into individual IPs using pure bash (no Python or external tools required):

MAX_CIDR_SIZE=24  # Only expand /24 or smaller (256 IPs max per range)

Settings:

0 = Disable CIDR expansion (skip all CIDR ranges)
24 = Expand /24 and smaller (up to 256 IPs) - Recommended
20 = Expand /20 and smaller (up to 4,096 IPs) - Use with caution
16 = Expand /16 and smaller (up to 65,536 IPs) - Not recommended

Behavior Examples:

CIDR in Source	MAX_CIDR_SIZE	Result
`192.0.2.0/32`	24	✅ Expanded (1 IP)
`192.0.2.0/28`	24	✅ Expanded (16 IPs)
`192.0.2.0/24`	24	✅ Expanded (256 IPs)
`192.0.2.0/20`	24	❌ Skipped (too large)
`192.0.2.0/16`	24	❌ Skipped (too large)
`192.0.2.0/16`	16	✅ Expanded (65,536 IPs)

Important:

Ranges larger than MAX_CIDR_SIZE are skipped entirely
This prevents accidentally expanding huge ranges (e.g., /8 = 16 million IPs)
See PERFORMANCE.md for detailed performance implications

Web Server Support

The script automatically detects whether you're running Apache or Nginx and configures itself accordingly:

Apache Implementation

Uses Apache's RewriteMap with binary DBM files
O(1) hash-based lookups for maximum speed
Requires mod_rewrite and httxt2dbm

Nginx Implementation

Uses Nginx's geo module with radix tree lookups
O(log n) lookups with excellent memory efficiency
No external dependencies beyond Nginx itself

For detailed Nginx setup instructions, see: docs/NGINX_SETUP.md

How It Works

The script uses a modular architecture with specialized components:

Architecture Overview

abuseip-blocker.sh - Main entry point (backward compatible)
config.sh - All configuration settings
utils.sh - Shared utilities (logging, file management, IP functions)
whitelist.sh - Whitelist management and IP filtering
update.sh - Update logic and processing pipeline
main.sh - Command processing and orchestration
ipv6_filter.py - Dedicated IPv6 CIDR processing

Processing Pipeline

Initialization Phase:
- Load configuration and validate dependencies
- Check for required tools (httxt2dbm, python3, etc.)
- Set up temporary file cleanup traps
Download Phase:
- Fetches blocklists from all configured sources with security checks
- Updates Cloudflare's IP ranges (with smart caching)
- Validates downloads and handles failures gracefully
Extraction Phase:
- Uses regex to extract valid IPv4 and IPv6 addresses
- Handles both structured (one IP per line) and unstructured data
- Validates IP format and excludes invalid addresses
- Expands CIDR ranges using pure Bash mathematics
Deduplication Phase:
- Sorts and removes duplicate IPs across all sources (with parallel processing)
- Filters out whitelisted IPs using pre-loaded associative arrays
- Monitors resource usage for large datasets
Conversion Phase:
- Formats as Apache RewriteMap (IP blocked)
- Converts to binary DBM format for fast lookups
- Sets proper file permissions and ownership
Activation Phase:
- Completely stops and restarts Apache for clean reload
- Ensures new blocklist is active and functional

Security Features

Input Validation: Strict IP/CIDR format validation
Secure Downloads: SSL verification, size limits, and fail-fast behavior
File Permissions: Proper ownership and Apache-readable permissions
Dependency Checks: Validates all required tools and versions
Temporary Files: Secure cleanup and no predictable paths
Privilege Control: Root requirement with proper validation

Performance Optimizations

Parallel Processing: Multi-core sorting and deduplication
Memory Efficient: Streaming processing for large datasets
Caching: Cloudflare ranges cached to reduce API calls
Batch Operations: IPv6 processing done in batches for speed
Resource Monitoring: CPU/memory tracking with warnings

Performance

DBM Format: Binary hash table provides O(1) lookup time
Memory Efficient: Apache loads the DBM file into shared memory
Scalability: Handles 100,000+ IPs without performance degradation
Lookup Speed: < 1ms for up to 500,000 IPs
File Size: ~20 bytes per IP (100k IPs ≈ 2 MB DBM file)

Performance by Scale

IP Count	Lookup Time	Memory	Update Time	Status
100,000	< 1 ms	~10 MB	~30s	✅ Excellent
500,000	1-2 ms	~40 MB	~2 min	✅ Good
1,000,000	2-3 ms	~80 MB	~5 min	⚠️ Acceptable
5,000,000+	5-10 ms	~300+ MB	~30+ min	❌ Not Recommended

See PERFORMANCE.md for detailed analysis and optimization strategies.

Troubleshooting

General Debugging

Enable verbose logging for detailed operation information:

sudo abuseip-blocker update --verbose

Check the main log file for errors:

sudo tail -f /var/log/abuseipdb_blocker.log

Component-Specific Issues

Configuration Problems

Test configuration loading:

# Source config manually to check for syntax errors
source ~/tools/abuseipdeny/config.sh
echo "Config loaded successfully"

Whitelist Issues

Verify whitelist loading:

# Test whitelist functions
source ~/tools/abuseipdeny/utils.sh
source ~/tools/abuseipdeny/config.sh
source ~/tools/abuseipdeny/whitelist.sh
load_whitelists
echo "Whitelist loaded: ${#WHITELIST_IPV4[@]} IPv4, ${#WHITELIST_IPV6[@]} IPv6 entries"

Update Pipeline Problems

Debug the update process step-by-step:

# Test individual update components
source ~/tools/abuseipdeny/utils.sh
source ~/tools/abuseipdeny/config.sh
source ~/tools/abuseipdeny/whitelist.sh
source ~/tools/abuseipdeny/update.sh

# Test downloads
download_sources
echo "Downloads completed"

# Test filtering
apply_filters
echo "Filtering completed"

# Test DBM conversion
convert_to_dbm
echo "DBM conversion completed"

IPs are not being blocked

Verify mod_rewrite and mod_remoteip are enabled:

apache2ctl -M | grep -E 'rewrite|remoteip'

Check that Apache can read the DBM file:

ls -la /etc/apache2/abuseipdb_blocklist.dbm*
# Should show 644 permissions

Test if Apache sees the real IP:

# Add this temporarily to your VirtualHost
CustomLog /var/log/apache2/realip.log "%a %{X-Forwarded-For}i"
# Check the log - %a should show visitor's real IP

Verify the DBM file contains data:

# Check if DBM file has content
db_dump /etc/apache2/abuseipdb_blocklist.dbm | head -10

Cloudflare IPs are not being protected

Run update to fetch the latest Cloudflare ranges:

sudo abuseip-blocker update

Check the Cloudflare whitelist was downloaded:

cat /etc/apache2/cloudflare_ips.txt

Verify Cloudflare ranges are loaded in whitelist:

grep "cloudflare" /var/log/abuseipdb_blocker.log | tail -5

Apache fails to restart

Check Apache error logs:

sudo tail -f /var/log/apache2/error.log

Common issues:

Syntax error in VirtualHost configuration
DBM file permissions incorrect (should be 644)
DBM file corrupted (run update again)
Missing Apache modules (rewrite, remoteip)

IPv6 Issues

Test IPv6 processing specifically:

# Test IPv6 filter script
python3 /usr/local/bin/ipv6_filter.py --help

# Check IPv6 whitelist loading
source /usr/local/bin/abuseip-blocker-whitelist.sh
load_whitelists
echo "IPv6 whitelist entries: ${#WHITELIST_IPV6[@]}"

If Python is unavailable, the script falls back to exact IPv6 matching only.

Performance Issues

Monitor resource usage during updates:

# Enable performance monitoring
sudo abuseip-blocker update --verbose | grep -E "(Memory|Time|Processing)"

Check system resources:

# Monitor during update
top -p $(pgrep -f abuseip-blocker)

Module Loading Errors

If you see "command not found" errors:

# Verify all modules are installed
ls -la /usr/local/bin/abuseip-blocker*.sh
ls -la /usr/local/bin/ipv6_filter.py

# Check module permissions
ls -la /usr/local/bin/abuseip-blocker*.sh | grep -v rwxr-xr-x

Network/Download Issues

Test connectivity to blocklist sources:

# Test each source URL
curl -I https://raw.githubusercontent.com/borestad/blocklist-abuseipdb/main/abuseipdb-s100-all.ipv4
curl -I https://www.cloudflare.com/ips-v4
curl -I https://www.cloudflare.com/ips-v6

If downloads fail, check firewall/proxy settings and network connectivity.

Security Considerations

Cloudflare Bypass Protection: The script automatically protects Cloudflare's IP ranges, preventing site outages.
Custom Whitelist: Add your office IPs, monitoring services, or trusted partners to prevent accidental blocks.
Logging: All operations are logged to /var/log/abuseipdb_blocker.log for audit trails.
Fail-Safe: If download fails, the script aborts without touching the existing blocklist.

Advanced Usage

Manual DBM Inspection

View what's in the DBM file:

# Convert back to text for viewing
db_dump /etc/apache2/abuseipdb_blocklist.dbm

Check if Specific IP is Blocked

grep "192.0.2.1" /etc/apache2/abuseipdb_blocklist.txt

Extract Statistics from Logs

# Count updates
grep "Update complete" /var/log/abuseipdb_blocker.log | wc -l

# View last update details
grep "Statistics" -A 10 /var/log/abuseipdb_blocker.log | tail -11

IPv6 Support

This script uses an enhanced hybrid approach for IP handling:

IPv4: Full CIDR range matching (e.g., 192.0.2.0/24 matches all 256 IPs)
IPv6: Full CIDR matching for Cloudflare ranges using dedicated Python script
IPv6: Exact match for custom whitelist entries

How it works:

Cloudflare IPv6 ranges (e.g., 2606:4700::/32) are properly matched using Python's ipaddress module
Batch processing: All IPv6 IPs checked at once (50x faster than per-IP checks)
Dedicated script: ipv6_filter.py handles all IPv6 CIDR operations
Graceful fallback: If Python unavailable, uses exact match only
Custom whitelist: IPv6 addresses use exact match (add specific IPs you need to protect)

Requirements:

Python 3.6+ (recommended for full IPv6 CIDR support)
Without Python: Falls back to exact match only (limited protection)

Performance:

Batch processing: 50x faster than individual checks
Memory efficient: Processes all IPv6 at once
Error handling: Invalid IPv6 addresses are preserved but logged

For detailed explanation, see: IPv6_SUPPORT.md

Architecture

The script uses a clean modular architecture for better maintainability and security:

Core Components

Component	Purpose	Dependencies
`abuseip-blocker.sh`	Main entry point (backward compatible)	None
`config.sh`	Configuration settings	None
`utils.sh`	Shared utilities (logging, IP math, file ops)	None
`whitelist.sh`	Whitelist management and filtering	`utils.sh`, `config.sh`
`update.sh`	Update pipeline and processing	`utils.sh`, `config.sh`, `whitelist.sh`
`main.sh`	Command processing and orchestration	All modules
`ipv6_filter.py`	IPv6 CIDR processing	Python 3.6+

Benefits

Maintainability: Each module has a single responsibility
Security: Isolated components reduce attack surface
Testing: Individual modules can be unit tested
Upgrades: Components can be updated independently
Debugging: Issues can be isolated to specific modules

File Locations

After installation, the components are located at:

/usr/local/bin/
├── abuseip-blocker.sh          # Main script
├── abuseip-blocker-config.sh   # Configuration
├── abuseip-blocker-utils.sh    # Utilities
├── abuseip-blocker-whitelist.sh # Whitelist functions
├── abuseip-blocker-update.sh   # Update functions
├── abuseip-blocker-main.sh     # Command processing
└── ipv6_filter.py              # IPv6 processing

Configuration Files

Runtime configuration files are created at:

/etc/apache2/
├── abuseipdb_blocklist.txt     # Human-readable blocklist
├── abuseipdb_blocklist.dbm*    # Binary DBM files
├── abuseipdb_whitelist.txt     # Custom whitelist
└── cloudflare_ips.txt          # Cloudflare ranges

Log Files

/var/log/abuseipdb_blocker.log  # Operation logs

License

MIT License - Feel free to modify and distribute.

Credits

Blocklist source: AbuseIPDB Community Blocklist
Cloudflare IP ranges: Cloudflare IP Ranges

Contributing

Issues and pull requests welcome! Please test thoroughly before submitting.

Name		Name	Last commit message	Last commit date
Latest commit History 18 Commits
docs		docs
.gitignore		.gitignore
BLOCKLIST_SOURCES.example		BLOCKLIST_SOURCES.example
CHANGELOG.md		CHANGELOG.md
README.md		README.md
VERSION		VERSION
abuseip-blocker.sh		abuseip-blocker.sh
abuseipblock-original-dbm.sh		abuseipblock-original-dbm.sh
apache-config.example		apache-config.example
config.sh		config.sh
install.sh		install.sh
ipv4_filter.py		ipv4_filter.py
ipv6_filter.py		ipv6_filter.py
main.sh		main.sh
test-cidr.sh		test-cidr.sh
test-ipv6-batch.sh		test-ipv6-batch.sh
update.sh		update.sh
utils.sh		utils.sh
whitelist.sh		whitelist.sh

mayankguptadotcom/abuseip-deny-script

Folders and files

Latest commit

History

Repository files navigation

AbuseIPDB Web Server IP Blocker

Features

Requirements

Common Requirements (Both Apache & Nginx)

Apache-Specific

Nginx-Specific

Installation

Option A: Quick Installation (Recommended)

Option B: Git Clone (Best for Updates)

Option C: Development Mode (For Contributors)

Usage

Update the Blocklist

Test the Blocker

Manage Whitelist

View Statistics

Configuration

Add Multiple Blocklist Sources

File Locations

Performance & Security Settings

CIDR Configuration

Web Server Support

Apache Implementation

Nginx Implementation

How It Works

Architecture Overview

Processing Pipeline

Security Features

Performance Optimizations

Performance

Performance by Scale

Troubleshooting

General Debugging

Component-Specific Issues

Configuration Problems

Whitelist Issues

Update Pipeline Problems

IPs are not being blocked

Cloudflare IPs are not being protected

Apache fails to restart

IPv6 Issues

Performance Issues

Module Loading Errors

Network/Download Issues

Security Considerations

Advanced Usage

Manual DBM Inspection

Check if Specific IP is Blocked

Extract Statistics from Logs

IPv6 Support

Architecture

Core Components

Benefits

File Locations

Configuration Files

Log Files

License

Credits

Contributing

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Uh oh!

Languages

Packages