Extending update ipsets

You can extend update-ipsets to download you IP lists.

There are 3 directories that are consulted for additional IP lists definitions:

1. /etc/firehol/ipsets.d
2. /usr/share/firehol/ipsets.d
3. ~/.update-ipsets/ipsets.d

In any of the above directories, you can place *.conf files, each describing one or more IP lists.

Here is what each of these files can contain:

IP list definitions

IP lists are defined with the update command. This is the template:

update name mins aggregation ipv keep \
    url \
    processor \
    category \
    info \
    maintainer maintainer_url \
    other options

name

Is the name of the IP list.

The name uniquely identifies the IP list and controls the filenames that will be generated.

mins

mins is the update frequency of the list, in minutes. update-ipsets will not attempt to download again an IP list if mins minutes have not passed since the modification time of the last downloaded file.

The modification time of the file at the maintainer's site is copied to the local file when it is downloaded. For example, if you download IP list named A, the file A.source that will be generated will have the modification time of the file as it is stored at the maintainer's site.

aggregation

aggregation is a list of durations in minutes that the ip list should be aggregated.

You can disable aggregation, by setting this parameter to 0.

Example values: "$[24*60] $[7*24*60] $[30*24*60]" to enable daily, weekly and monthly aggregations.

ipv

It can be ipv4 or ipv6.

Only ipv4 is supported at the moment.

keep

It can be:

• ip to keep only the single IPs found in the source file. If the source file includes also subnets, these will be filtered out.

  The generated file will have the suffix .ipset.

• net to keep only subnets found in the source file, ignoring single IPs.

  The generated file will the suffix .netset

• both to keep both single IPs and subnets found in the source file.

  The generated file will the suffix .netset

url

It is the URL to download the IP list from.

update-ipsets supports custom downloaders. The default supports downloading HTTP and HTTPS urls.

processor

The processor is a BASH function or external command that should input the source file and output the IPs or subnets it contains.

The processor is used the same way pipes work. Something like this:

cat source_file | processor > ips_and_subnets

# or

processor < source_file > ips_and_subnets

There are many processors already present in update-ipsets which you can use:

processor	description
trim	remove leading, trailing and multiple spaces and tabs from the file, remove empty lines
remove_comments	remove anything after a # and also trim
remove_comments_semi_colon	remove anything after a ; and also trim
gz_remove_comments	like remove_comments for gzip compressed files
snort_alert_rules_to_ipv4	extract IPs and subnets from snort alert rules
pix_deny_rules_to_ipv4	extract IPs (not subnets) from Cisco PIX access list deny rules
extract_ipv4_from_any_file	extract IPs (not subnets) from any text file
hostname_resolver	convert a list of hostnames to IPs by doing parallel DNS queries

You can find a lot more processors in the update-ipsets source. There is even an XML parser for BASH. You can check the parse_rss_rosinstrument, parse_rss_proxy, etc processors in the update-ipsets source.

category

You can define any category you like.

I have categorized all IP lists in these categories:

category	description
abuse	The IPs have been reported for some kind of abuse. Abuse is not an attack. It may be forum spamming, excessive or bad use for something, etc.
anonymizers	The IPs are used by other parties for hiding their real identity. Open proxies, VPN providers, etc fall into this category.
attacks	The IPs have been reported to actively try to crack, attack, take control, penetrate, of DDoS a remote system or service.
geolocation	The IP lists in this category are used for geolocation.
malware	The IPs listed are known for hosting malware, botnets, worms, drones, or somehow participating in related cyber crime activities.
organizations	The IP lists track the IP address space of organizations.
reputation	The IPs listed have somehow bad reputation. There is no clear evidence of abuse, attack or malware. IP lists that track connection failures to certain services using firewall or IDS logs are an example.
spam	The IPs listed are related to e-mail (SMTP) spamming.
unroutable	The IPs listed should not be routable in the Internet.