Blocklists not considering allowlisted IPs from ranges

[dotsam]

What would you like to be added?

/kind enhancement

Ranges on a blocklist should consider if allowlisted IP addresses fall within them and not distribute them to bouncers

Why is this needed?

We're an enterprise customer currently using the Service API to create a custom blocklist which we're then subscribing a distributed security engine to. We're also using centralized allowlists in the Console.

The issue that's come up is that in our custom blocklist, we're including some large IP blocks, while some of our allowlists contain a single IP address from the larger block that should be allowed.

We're currently only using the crowdsec-firewall-bouncer remediation component, and it seems like there's nothing that will factor a single IP out being included in a blocked IP set.

I see a few potential solutions here:

• Given that these are centralized allow and block lists, either the CAPI or the LAPI could spot that the allowed IP is in a blocked range and factor it out. In the example below though, this creates 17 ranges from a single range, so this may not scale
• Distribute the allowlist to bouncers for them to implement. In the case of the firewall bouncer, this could be implemented as a dedicated ipset where traffic is accepted, or the existing ipsets for blocklists could have an entry added with nomatch.

Here's an example, where 52.86.0.0/15 is on the custom blocklist, and 52.87.72.16 is on a centralized allowlist.

On the LAPI:

# cscli decisions list --all -o raw | grep '52\.86\.0'

368969241,lists,Ip:52.86.0.0/15,our-custom-blocklist,ban,,,0,-5h23m53s,false,1943512

# cscli allowlists check 52.87.72.16

52.87.72.16 is allowlisted by item 52.87.72.16 from uptimerobot (uptimerobot)

And on the host where the firewall bouncer is installed:

# ipset test crowdsec-blacklists-4 52.87.72.16

Warning: 52.87.72.16 is in set crowdsec-blacklists-4.

# ipset list crowdsec-blacklists-4 | grep '52\.86\.0'

52.86.0.0/15 timeout 64943

[github-actions]

@dotsam: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@dotsam: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[blotus]

Hello,

Crowdsec should already be matching the content of the blocklist it pulls with the content of the allowlists to prevent this exact issue.

If the allowlist is updated afterward, we will also apply the content on all existing decisions, so that should remove any existing decisions as well.

One important thing to note: as you said, just removing a single IP from a big range is going to split it into a lot of subranges, and will not scale very well:

• It will lead to a lot more decisions
• It will massively increase the complexity of sending decisions to the bouncers or require a massive rewrite of the way we track decisions internally: In your example, if the allowlist is ever removed, it means the range would need to be reaggregated, so LAPI would need to track more things (and it would probably get even harder if you had another allowlist that removes another IP from the range added afterwards or allow a range inside another range)

What we do currently (at least in theory, as it does not seem to work for you) is if an allowlist matches a decision, we entirely remove the decision, meaning in your case, the entire /15 range will not be distributed to the bouncer as there's a single entry inside it that is allowlisted (I do agree it's sub-optimal).

Anyway, I will try to reproduce your issue and see what is going on.

[dotsam]

/kind bug

@blotus Thanks, let me know if there's any other information I can provide to help troubleshoot

[blotus]

Your issue is likely not related to the allowlists.

Looking again at what you've pasted, we can see the decision was correctly expired:

368969241,lists,Ip:52.86.0.0/15,our-custom-blocklist,ban,,,0,-5h23m53s,false,1943512

You can see the -5h23m53s, which indicates the time left in the decision (a negative duration indicates that the decision was soft-deleted and has not yet been flushed from the database).
I also tried to reproduce it, but had no luck (decision was expired, and the bouncer removed it from the ipset).

The most likely explanation is that, for some reason, the bouncer missed the information that the decision was removed. If that's indeed the case, it's going to be hard to reproduce: the only way I can see this happening is for the bouncer to do a pull at exactly the same time the decision expires.

As a workaround for now, restarting the bouncer will force it to refetch everything from the DB, so the decision will not be applied.