Merge pull request #909 from crowdsecurity/hes-fix-integrations-size-limit

add lists size limit mgmt doc

Author: he2ss

crowdsec-docs/unversioned/console/service_api/integrations.md

@@ -19,11 +19,54 @@ For some constructors, the integrations can generate vendor-specific format, see
 
 For all the other providers, the `plain_text` format consists of one ip per line, and should be supported by most devices. If a specific format is missing, reach out to us and we'll help you support it!
 
-## Dealing with blocklist size limits
+## Managing integrations size limits with pagination
 
-Some providers have technical limits on the number of IPs they can pull at once. That's why we recommand to monitor the number of IPS returned by the integration and use the pagination feature if needed. For this, you can use the page and page_size query parameters in the URL.
+Some firewalls or security devices impose strict limits on how many IP addresses can be imported or processed from an external blocklist. When a blocklist exceeds these limits, it can lead to incomplete imports or failures during updates. To address this, CrowdSec integrations support pagination, allowing you to fetch IPs in manageable chunks.
 
-https://admin.api.crowdsec.net/v1/integrations/123/content?page=1&page_size=1500
+### Why pagination matters
 
-You can then use the page parameter to get the next page of IPs.
+Pagination ensures that large blocklists are retrieved and processed efficiently by splitting them into smaller segments. This helps:
 
+* Avoid exceeding the maximum number of entries a firewall can handle per list.
+* Maintain reliable updates without API timeouts.
+* Improve performance when synchronizing IPs from CrowdSec.
+
+### How pagination works
+
+You can control pagination using two query parameters in the integration API URL:
+
+* `page`: The current page number (starting from 1).
+* `page_size`: The number of IP addresses to include per page.
+
+Example request:
+
+```
+GET https://admin.api.crowdsec.net/v1/integrations/123/content?page=1&page_size=1500
+```
+
+* The above request retrieves the first 1,500 IPs in the list.
+* To fetch the next batch, increment the page parameter:
+
+```
+GET https://admin.api.crowdsec.net/v1/integrations/123/content?page=2&page_size=1500
+```
+
+Repeat this process until no new results are returned.
+
+### Example use case (Palo Alto firewall)
+
+A Palo Alto firewall may limit external dynamic lists between 50,000 and 150,000 entries depending on the model. If your CrowdSec blocklist exceeds this limit, you can set `page_size` to 50,000 and iterate through pages until all IPs are retrieved.
+
+1. Start with `page=1` and `page_size=50000`.
+2. Add the dynamic list to the firewall.
+3. Increment the `page` parameter and add the new dynamic list.
+4. Repeat until all IPs are processed.
+
+
+### Pro Tip
+
+When you know the maximum number of entries your device can handle, and you want to calculate the number of pages needed, you will also need to know the total number of IPs in your integration. You can get this information from the [integration details page](https://app.crowdsec.net/blocklists/integrations), where you can find the "Total IPs" count. Then, use the following formula:
+
+```
+number_of_pages = ceil(total_ips / page_size)
+```

crowdsec-docs/unversioned/integrations/checkpoint.mdx

@@ -98,6 +98,10 @@ Accessobserv2,192.168.38.188,IP,high,high,AB,C&C server IP
 The format is as follows: UNIQ-NAME , VALUE , TYPE , CONFIDENCE , SEVERITY , PRODUCT , COMMENT
 :::
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)

crowdsec-docs/unversioned/integrations/cisco.mdx

@@ -61,6 +61,11 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin
 192.168.38.186
 ```
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)

crowdsec-docs/unversioned/integrations/f5.mdx

@@ -63,6 +63,10 @@ The CrowdSec blocklist will be in F5 format, with formatted data per line. Here
 The format is as follows: IP, Mask, WL/BL, Category
 :::
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)

crowdsec-docs/unversioned/integrations/fortinet.mdx

@@ -60,6 +60,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin
 192.168.38.186
 ```
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)

crowdsec-docs/unversioned/integrations/juniper.mdx

@@ -108,6 +108,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin
 192.168.38.186
 ```
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)

crowdsec-docs/unversioned/integrations/mikrotik.mdx

@@ -112,6 +112,10 @@ The CrowdSec blocklist will be in mikrotik format, with formatted data per line.
 /ip6 firewall address-list add list=crowdsec-integration address=2001:0db8:85a3::/128 comment="crowdsec/mikrotik" timeout=48h;
 ```
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)

crowdsec-docs/unversioned/integrations/opnsense.mdx

@@ -78,6 +78,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin
 192.168.38.186
 ```
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)

crowdsec-docs/unversioned/integrations/paloalto.mdx

@@ -107,6 +107,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin
 192.168.38.186
 ```
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)

crowdsec-docs/unversioned/integrations/pfsense.mdx

@@ -78,6 +78,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin
 192.168.38.186
 ```
 
+## Manage integration size limits with pagination
+
+If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section.
+
 ## Contribute to this documentation
 
 Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)