Skip to content

Commit a526772

Browse files
Add MikroTik pagination script for smaller devices
- Add new pagination script section to MikroTik integration docs - Script fetches and imports blocklists in configurable chunks (default 15k entries) - Includes memory-efficient approach with automatic cleanup - Provides detailed logging and progress tracking - Ideal for devices with limited RAM or large blocklists - Maintains compatibility with existing integration format
1 parent c939261 commit a526772

File tree

1 file changed

+110
-0
lines changed

1 file changed

+110
-0
lines changed

crowdsec-docs/unversioned/integrations/mikrotik.mdx

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,116 @@ The script is now running and the blocklist is being imported into the Mikrotik
102102
Please be careful with the frequency of the scheduler and the blocklists size your subscribe to your integration, as it may cause performance issues on your Mikrotik device.
103103
:::
104104
105+
## Pagination Script for Smaller Devices
106+
107+
For smaller MikroTik devices that cannot handle large blocklists at once, you can use this pagination script that fetches and imports the blocklist in smaller chunks:
108+
109+
```bash
110+
# ========= CONFIG (v6) =========
111+
:local name "[crowdsec]"
112+
:local baseUrl "https://admin.api.crowdsec.net/v1/integrations/<integration_id>/content"
113+
:local pageSize 15000
114+
115+
# Basic Auth (v6 uses http-auth-scheme + user/password)
116+
:local user "<username>"
117+
:local pass "<password>"
118+
119+
# (Optional) clear these lists up-front (matches your .rsc lines)
120+
:local list4 "crowdsec-integration"
121+
:local list6 "crowdsec-integration"
122+
123+
# Fetch timeout
124+
:local fetchTimeout "30s"
125+
# ===============================
126+
127+
:log info "$name start (v6, pagination, per-page import)"
128+
129+
# Optional: start fresh
130+
:log info "$name clearing address-lists"
131+
/ip firewall address-list remove [ find where list=$list4 ]
132+
/ipv6 firewall address-list remove [ find where list=$list6 ]
133+
134+
:local page 1
135+
:local totalLines 0
136+
137+
:do {
138+
:local tmpname ("crowdsec_page_" . $page . ".rsc")
139+
140+
# Fetch one page to a file (no big RAM usage)
141+
/tool fetch \
142+
url=($baseUrl . "?page=" . $page . "&page_size=" . $pageSize) \
143+
mode=https dst-path=$tmpname keep-result=yes \
144+
http-auth-scheme=basic user=$user password=$pass \
145+
idle-timeout=$fetchTimeout
146+
# NOTE: You can optionally try gzip on newer v6 builds:
147+
# http-header-field="Accept-Encoding:gzip"
148+
149+
:if ([:len [/file find where name=$tmpname]] = 0) do={
150+
:log error "$name fetch failed for page $page"
151+
:break
152+
}
153+
154+
# Count non-empty lines to decide final page
155+
:local data [/file get $tmpname contents]
156+
:local itemsThisPage 0
157+
:while ([:len $data] > 0) do={
158+
:local nl [:find $data "\n"]
159+
:local line ""
160+
:if ($nl = nil) do={ :set line $data; :set data "" } else={
161+
:set line [:pick $data 0 $nl]
162+
:set data [:pick $data ($nl + 1) [:len $data]]
163+
}
164+
# Trim CR
165+
:if (([:len $line] > 0) && ([:pick $line ([:len $line]-1) [:len $line]] = "\r")) do={
166+
:set line [:pick $line 0 ([:len $line]-1)]
167+
}
168+
:if ([:len $line] > 0) do={ :set itemsThisPage ($itemsThisPage + 1) }
169+
}
170+
171+
:log info "$name page $page: lines=$itemsThisPage"
172+
173+
# Import this page immediately, then delete the temp file
174+
/import file-name=$tmpname
175+
/file remove $tmpname
176+
177+
:set totalLines ($totalLines + $itemsThisPage)
178+
179+
# Stop when this page is short
180+
:if ($itemsThisPage < $pageSize) do={
181+
:log info "$name final page reached (lines $itemsThisPage < $pageSize). Total imported: $totalLines"
182+
:break
183+
}
184+
185+
:set page ($page + 1)
186+
} while=true
187+
188+
:log info "$name done"
189+
```
190+
191+
:::info
192+
You need to replace `<integration_id>`, `<username>` and `<password>` with the values provided in the console.
193+
:::
194+
195+
:::warning
196+
Do not change `list="crowdsec-integration"` in the script, as it is used by the data you fetch from the CrowdSec API.
197+
:::
198+
199+
### Benefits of the Pagination Script
200+
201+
- **Memory Efficient**: Fetches and imports data in smaller chunks, reducing memory usage
202+
- **Better for Small Devices**: Ideal for MikroTik devices with limited RAM
203+
- **Configurable Page Size**: Adjust the `pageSize` variable based on your device's capabilities
204+
- **Progress Logging**: Provides detailed logs about import progress
205+
- **Automatic Cleanup**: Removes temporary files after each page import
206+
207+
### When to Use Pagination
208+
209+
Use the pagination script when:
210+
- Your MikroTik device has limited RAM
211+
- You're experiencing memory issues with the standard script
212+
- You're importing large blocklists (50,000+ entries)
213+
- Your device frequently runs out of memory during imports
214+
105215
## Format example
106216
107217
The CrowdSec blocklist will be in mikrotik format, with formatted data per line. Here is an example of how the blocklist will look:

0 commit comments

Comments
 (0)