diff --git a/crowdsec-docs/unversioned/console/service_api/integrations.md b/crowdsec-docs/unversioned/console/service_api/integrations.md index 3322243b0..3bb3674c6 100644 --- a/crowdsec-docs/unversioned/console/service_api/integrations.md +++ b/crowdsec-docs/unversioned/console/service_api/integrations.md @@ -19,11 +19,54 @@ For some constructors, the integrations can generate vendor-specific format, see For all the other providers, the `plain_text` format consists of one ip per line, and should be supported by most devices. If a specific format is missing, reach out to us and we'll help you support it! -## Dealing with blocklist size limits +## Managing integrations size limits with pagination -Some providers have technical limits on the number of IPs they can pull at once. That's why we recommand to monitor the number of IPS returned by the integration and use the pagination feature if needed. For this, you can use the page and page_size query parameters in the URL. +Some firewalls or security devices impose strict limits on how many IP addresses can be imported or processed from an external blocklist. When a blocklist exceeds these limits, it can lead to incomplete imports or failures during updates. To address this, CrowdSec integrations support pagination, allowing you to fetch IPs in manageable chunks. -https://admin.api.crowdsec.net/v1/integrations/123/content?page=1&page_size=1500 +### Why pagination matters -You can then use the page parameter to get the next page of IPs. +Pagination ensures that large blocklists are retrieved and processed efficiently by splitting them into smaller segments. This helps: +* Avoid exceeding the maximum number of entries a firewall can handle per list. +* Maintain reliable updates without API timeouts. +* Improve performance when synchronizing IPs from CrowdSec. + +### How pagination works + +You can control pagination using two query parameters in the integration API URL: + +* `page`: The current page number (starting from 1). +* `page_size`: The number of IP addresses to include per page. + +Example request: + +``` +GET https://admin.api.crowdsec.net/v1/integrations/123/content?page=1&page_size=1500 +``` + +* The above request retrieves the first 1,500 IPs in the list. +* To fetch the next batch, increment the page parameter: + +``` +GET https://admin.api.crowdsec.net/v1/integrations/123/content?page=2&page_size=1500 +``` + +Repeat this process until no new results are returned. + +### Example use case (Palo Alto firewall) + +A Palo Alto firewall may limit external dynamic lists between 50,000 and 150,000 entries depending on the model. If your CrowdSec blocklist exceeds this limit, you can set `page_size` to 50,000 and iterate through pages until all IPs are retrieved. + +1. Start with `page=1` and `page_size=50000`. +2. Add the dynamic list to the firewall. +3. Increment the `page` parameter and add the new dynamic list. +4. Repeat until all IPs are processed. + + +### Pro Tip + +When you know the maximum number of entries your device can handle, and you want to calculate the number of pages needed, you will also need to know the total number of IPs in your integration. You can get this information from the [integration details page](https://app.crowdsec.net/blocklists/integrations), where you can find the "Total IPs" count. Then, use the following formula: + +``` +number_of_pages = ceil(total_ips / page_size) +``` diff --git a/crowdsec-docs/unversioned/integrations/checkpoint.mdx b/crowdsec-docs/unversioned/integrations/checkpoint.mdx index 977108331..bb9e448f4 100644 --- a/crowdsec-docs/unversioned/integrations/checkpoint.mdx +++ b/crowdsec-docs/unversioned/integrations/checkpoint.mdx @@ -98,6 +98,10 @@ Accessobserv2,192.168.38.188,IP,high,high,AB,C&C server IP The format is as follows: UNIQ-NAME , VALUE , TYPE , CONFIDENCE , SEVERITY , PRODUCT , COMMENT ::: +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/cisco.mdx b/crowdsec-docs/unversioned/integrations/cisco.mdx index 32883a9d6..075283c28 100644 --- a/crowdsec-docs/unversioned/integrations/cisco.mdx +++ b/crowdsec-docs/unversioned/integrations/cisco.mdx @@ -61,6 +61,11 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin 192.168.38.186 ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/f5.mdx b/crowdsec-docs/unversioned/integrations/f5.mdx index 0455c314d..4211f0213 100644 --- a/crowdsec-docs/unversioned/integrations/f5.mdx +++ b/crowdsec-docs/unversioned/integrations/f5.mdx @@ -63,6 +63,10 @@ The CrowdSec blocklist will be in F5 format, with formatted data per line. Here The format is as follows: IP, Mask, WL/BL, Category ::: +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/fortinet.mdx b/crowdsec-docs/unversioned/integrations/fortinet.mdx index 52c0c2da5..87d44ad1b 100644 --- a/crowdsec-docs/unversioned/integrations/fortinet.mdx +++ b/crowdsec-docs/unversioned/integrations/fortinet.mdx @@ -60,6 +60,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin 192.168.38.186 ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/juniper.mdx b/crowdsec-docs/unversioned/integrations/juniper.mdx index 86f3e185c..a3214f166 100644 --- a/crowdsec-docs/unversioned/integrations/juniper.mdx +++ b/crowdsec-docs/unversioned/integrations/juniper.mdx @@ -108,6 +108,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin 192.168.38.186 ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/mikrotik.mdx b/crowdsec-docs/unversioned/integrations/mikrotik.mdx index 3bc4f0a31..0fb16a608 100644 --- a/crowdsec-docs/unversioned/integrations/mikrotik.mdx +++ b/crowdsec-docs/unversioned/integrations/mikrotik.mdx @@ -112,6 +112,10 @@ The CrowdSec blocklist will be in mikrotik format, with formatted data per line. /ip6 firewall address-list add list=crowdsec-integration address=2001:0db8:85a3::/128 comment="crowdsec/mikrotik" timeout=48h; ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/opnsense.mdx b/crowdsec-docs/unversioned/integrations/opnsense.mdx index 84382ca6a..d023db7bd 100644 --- a/crowdsec-docs/unversioned/integrations/opnsense.mdx +++ b/crowdsec-docs/unversioned/integrations/opnsense.mdx @@ -78,6 +78,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin 192.168.38.186 ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/paloalto.mdx b/crowdsec-docs/unversioned/integrations/paloalto.mdx index 93b5f5a12..bc80d96cf 100644 --- a/crowdsec-docs/unversioned/integrations/paloalto.mdx +++ b/crowdsec-docs/unversioned/integrations/paloalto.mdx @@ -107,6 +107,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin 192.168.38.186 ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/pfsense.mdx b/crowdsec-docs/unversioned/integrations/pfsense.mdx index 957e4f721..ce0014132 100644 --- a/crowdsec-docs/unversioned/integrations/pfsense.mdx +++ b/crowdsec-docs/unversioned/integrations/pfsense.mdx @@ -78,6 +78,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin 192.168.38.186 ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/rawiplist.mdx b/crowdsec-docs/unversioned/integrations/rawiplist.mdx index 84cd326c0..7e68ce7b3 100644 --- a/crowdsec-docs/unversioned/integrations/rawiplist.mdx +++ b/crowdsec-docs/unversioned/integrations/rawiplist.mdx @@ -64,6 +64,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin 192.168.38.186 ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs) diff --git a/crowdsec-docs/unversioned/integrations/sophos.mdx b/crowdsec-docs/unversioned/integrations/sophos.mdx index 688cd475c..231b8ab16 100644 --- a/crowdsec-docs/unversioned/integrations/sophos.mdx +++ b/crowdsec-docs/unversioned/integrations/sophos.mdx @@ -60,6 +60,10 @@ The CrowdSec blocklist will be in plain text format, with one IP address per lin 192.168.38.186 ``` +## Manage integration size limits with pagination + +If you want to learn how to manage integration size limits with pagination, please refer to the [Managing integrations size limits with pagination](console/service_api/integrations.md#managing-integrations-size-limits-with-pagination) section. + ## Contribute to this documentation Since CrowdSec is a community-driven project, we welcome contributions to this documentation. If you have any instructions or tips that you would like to share with the community, please feel free to open a pull request on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs)