Skip to content

Commit e0986b6

Browse files
committed
feat: Add import functionality for IP and Domain blacklists
- Enhanced the blacklists.html template to include modals for importing IP and Domain blacklists. - Implemented JavaScript functions to handle the import process for both IP and Domain blacklists. - Created examples for importing blacklists in markdown format, detailing usage of the API. - Added a test script to validate the import functionality for both IP and Domain blacklists.
1 parent 8f2b50b commit e0986b6

File tree

6 files changed

+990
-332
lines changed

6 files changed

+990
-332
lines changed

README.md

Lines changed: 153 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -169,21 +169,53 @@ To use Secure Proxy as a transparent proxy:
169169

170170
### Extending Blacklists
171171

172-
Integrate with external threat intelligence:
172+
Integrate with external threat intelligence sources. The system supports importing plain text files (one entry per line) and JSON formats.
173173

174-
1. Import blacklists via the API:
175-
```bash
176-
curl -X POST http://localhost:8011/api/blacklists/import \
177-
-H "Content-Type: application/json" \
178-
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
179-
-d '{"url": "https://example.com/blacklist.txt", "type": "ip"}'
180-
```
174+
#### Import Domain Blacklists
181175

182-
2. Schedule automatic updates with the maintenance endpoint:
183-
```bash
184-
curl -X POST http://localhost:8011/api/maintenance/update-blacklists \
185-
-H "Authorization: Basic $(echo -n admin:admin | base64)"
186-
```
176+
```bash
177+
# Import from URL - supports plain text files with one domain per line
178+
curl -X POST http://localhost:8011/api/domain-blacklist/import \
179+
-H "Content-Type: application/json" \
180+
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
181+
-d '{"url": "https://example.com/domain-blacklist.txt"}'
182+
183+
# Import direct content
184+
curl -X POST http://localhost:8011/api/domain-blacklist/import \
185+
-H "Content-Type: application/json" \
186+
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
187+
-d '{"content": "example.com\n*.badsite.org\nmalicious.net"}'
188+
```
189+
190+
#### Import IP Blacklists
191+
192+
```bash
193+
# Import from URL - supports plain text files with one IP per line
194+
curl -X POST http://localhost:8011/api/ip-blacklist/import \
195+
-H "Content-Type: application/json" \
196+
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
197+
-d '{"url": "https://example.com/ip-blacklist.txt"}'
198+
199+
# Import direct content with CIDR notation support
200+
curl -X POST http://localhost:8011/api/ip-blacklist/import \
201+
-H "Content-Type: application/json" \
202+
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
203+
-d '{"content": "192.168.1.100\n10.0.0.5\n172.16.0.0/24"}'
204+
```
205+
206+
#### Supported File Formats
207+
208+
- **Plain Text**: One entry per line (recommended for most blacklists)
209+
- **JSON Array**: `["example.com", "malicious.net"]`
210+
- **JSON Objects**: `[{"domain": "example.com", "description": "Blocked site"}]`
211+
- **Comments**: Lines starting with `#` are ignored
212+
213+
#### Schedule Automatic Updates
214+
215+
```bash
216+
curl -X POST http://localhost:8011/api/maintenance/update-blacklists \
217+
-H "Authorization: Basic $(echo -n admin:admin | base64)"
218+
```
187219

188220
## 📊 Monitoring and Analytics
189221

@@ -301,30 +333,130 @@ To test if blacklisting works:
301333

302334
## 📘 API Documentation
303335

304-
Secure Proxy provides a comprehensive RESTful API for integration and automation:
336+
Secure Proxy provides a comprehensive RESTful API for integration and automation with support for plain text and JSON blacklist imports.
305337

306338
### Authentication
307339

340+
All API endpoints require Basic Authentication:
341+
308342
```bash
343+
# Login to get session (optional)
309344
curl -X POST http://localhost:8011/api/login \
310345
-H "Content-Type: application/json" \
311346
-d '{"username": "admin", "password": "admin"}'
347+
348+
# Or use Basic Auth directly (recommended for scripts)
349+
AUTH_HEADER="Authorization: Basic $(echo -n admin:admin | base64)"
312350
```
313351

314-
### Available Endpoints
352+
### 🚫 Blacklist Management
353+
354+
#### Import Domain Blacklists
355+
356+
Perfect for importing standard text files with one domain per line:
357+
358+
```bash
359+
# Import from URL (plain text file)
360+
curl -X POST http://localhost:8011/api/domain-blacklist/import \
361+
-H "Content-Type: application/json" \
362+
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
363+
-d '{"url": "https://example.com/domain-blacklist.txt"}'
364+
365+
# Import direct content
366+
curl -X POST http://localhost:8011/api/domain-blacklist/import \
367+
-H "Content-Type: application/json" \
368+
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
369+
-d '{"content": "malicious.com\n*.ads.example\nbadsite.org"}'
370+
```
371+
372+
**Example domain-blacklist.txt:**
373+
```
374+
malicious.com
375+
badsite.org
376+
*.ads.network
377+
phishing-site.net
378+
# Comments are ignored
379+
unwanted.domain
380+
```
381+
382+
#### Import IP Blacklists
383+
384+
```bash
385+
# Import from URL (plain text file)
386+
curl -X POST http://localhost:8011/api/ip-blacklist/import \
387+
-H "Content-Type: application/json" \
388+
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
389+
-d '{"url": "https://example.com/ip-blacklist.txt"}'
390+
391+
# Import with CIDR notation support
392+
curl -X POST http://localhost:8011/api/ip-blacklist/import \
393+
-H "Content-Type: application/json" \
394+
-H "Authorization: Basic $(echo -n admin:admin | base64)" \
395+
-d '{"content": "192.168.1.100\n10.0.0.0/8\n172.16.0.0/12"}'
396+
```
397+
398+
**Example ip-blacklist.txt:**
399+
```
400+
192.168.1.100
401+
10.0.0.5
402+
203.0.113.0/24
403+
# Malicious IP range
404+
198.51.100.0/24
405+
```
406+
407+
#### Supported File Formats
408+
409+
-**Plain Text**: One entry per line (most common)
410+
-**JSON Array**: `["entry1", "entry2"]`
411+
-**JSON Objects**: `[{"domain": "example.com", "description": "Blocked"}]`
412+
-**Comments**: Lines starting with `#` are ignored
413+
-**CIDR Notation**: For IP ranges (`192.168.1.0/24`)
414+
-**Wildcards**: For domains (`*.example.com`)
415+
416+
### 📋 Available Endpoints
315417

316418
| Endpoint | Method | Description |
317419
|----------|--------|-------------|
318420
| `/api/status` | GET | Get proxy service status |
319-
| `/api/settings` | GET | Get all proxy settings |
320-
| `/api/ip-blacklist` | GET/POST | Manage IP blacklist |
321-
| `/api/domain-blacklist` | GET/POST | Manage domain blacklist |
322-
| `/api/logs` | GET | Get proxy access logs |
421+
| `/api/settings` | GET/POST | Manage proxy settings |
422+
| `/api/ip-blacklist` | GET/POST/DELETE | Manage individual IP entries |
423+
| `/api/ip-blacklist/import` | POST | **Import IP blacklist from URL/content** |
424+
| `/api/domain-blacklist` | GET/POST/DELETE | Manage individual domain entries |
425+
| `/api/domain-blacklist/import` | POST | **Import domain blacklist from URL/content** |
426+
| `/api/blacklists/import` | POST | Generic import (requires type parameter) |
427+
| `/api/logs` | GET | Get proxy access logs with filtering |
323428
| `/api/logs/import` | POST | Import logs from Squid |
324429
| `/api/maintenance/clear-cache` | POST | Clear the proxy cache |
325-
| `/api/security/score` | GET | Get security assessment |
430+
| `/api/maintenance/reload-config` | POST | Reload proxy configuration |
431+
| `/api/security/score` | GET | Get security assessment score |
432+
433+
### 📊 Example API Responses
434+
435+
**Successful Import:**
436+
```json
437+
{
438+
"status": "success",
439+
"message": "Import completed: 150 entries imported",
440+
"imported_count": 150,
441+
"error_count": 0
442+
}
443+
```
444+
445+
**Import with Errors:**
446+
```json
447+
{
448+
"status": "success",
449+
"message": "Import completed: 145 entries imported, 5 errors",
450+
"imported_count": 145,
451+
"error_count": 5,
452+
"errors": [
453+
"Invalid domain format: not-a-domain",
454+
"Invalid IP format: 999.999.999.999"
455+
]
456+
}
457+
```
326458

327-
Full API documentation is available at `/api/docs` when the service is running.
459+
Full interactive API documentation is available at `/api/docs` when the service is running.
328460

329461
## 🔒 Security Best Practices
330462

0 commit comments

Comments
 (0)