Hundreds of IPSets Created on WAF but not used in Rule Groups CrowdSec Creates

[APhatak992]

When I deploy the AWS WAF Bouncer with my CorwdSec Deployment, I notice hundreds of IPSets getting created on the Web ACL I specify in the Bouncer config.

However, none of these IPSets are being used in the rule groups that the Bouncer creates.

Is the purpose of these IPSets to share a database of malicious IPs with us so that we can create rules for them separately, or is the Bouncer supposed to be creating rules that enforce actions against the IPSets that are created?

Currently, it seems that the bouncer will enforce actions for suspicious requests coming in rather than using the 1 milion plus IPs listed in the hundreds of IPSets that the bouncer adds.

[blotus]

Hello,

How many decisions do you have ? (cscli decisions list --all | wc -l)
How many ACLs have you configured in the bouncer ? Currently, the bouncer does not know how to share the sets between different ACLs, so the same IPs will end up in different sets.

The bouncer will create one ipset per 10k IPs (the maximum allowed by AWS) per IP type per remediation type (so different sets will be used for IPv4/IPv6 and ban/captcha remediation, which may increase the total by a few sets).

Also, the bouncer will delete every resources that it created on shutdown. If you stop the bouncer, do you see all sets being deleted ? If not, they might be remnant from previous runs (if the bouncer crashes, it won't be able to delete them for example), and you will need to delete them manually.

[APhatak992]

Hello,

cscli decisions list --all | wc -l returns 15000.

I have two configurations for the same web ACL (with different ip_header and ip_position values) in the bouncer.

I noticed the bouncer does not delete every resource it creates on shutdown. Prior to deploying the bouncer right now, I manually deleted all the ipsets that were previously created by it. After that, the bouncer created the 100+ ipsets I currently see.

From the initial time I started using the CrowdSec WAF Bouncer, I noticed this being an issue where even though the docs say the bouncer deletes resources it creates on WAF, it does not for my case.

Could you please help diagnosing

1. whether the 100+ ipsets the bouncer creates are supposed to be used by the bouncer itself to create rules referencing those upsets OR as a user do I need to create sepeate rules outside of what crowdsec creates that use the ipsets?
2. why the bouncer does not delete resources it creates on WAF
3. We have multiple bouncers deployed (1-2 per cluster) and multiple clusters in AWS using the same WAF. I think each bouncer is creating ipsets with the same ips. Is there a way for the clusters to share the ipsets?

[blotus]

100+ ipsets is definitely way too much, with 15k decisions, you should have maximum 4 sets (assuming all are ban decisions and there's a mix of IPv4/IPv6).

The only thing I can imagine preventing the deletion of the resources on shutdown is a permission issue on the role used by the bouncer.

Could you share the bouncer log file so that I try to see if there's something interesting in it ? (default location is /var/log/crowdsec/cs-aws-waf-bouncer.log)

[APhatak992]

I exec'd into the bouncer pod and ran cd /var/log/crowdsec/cs-aws-waf-bouncer.log but I get no such file or directory

[APhatak992]

For reference I have used these permissions for the bouncer (found at the bottom of this link)

https://docs.crowdsec.net/u/bouncers/aws_waf/

[blotus]

If you are using docker, then the logs are outputted to stderr, you can get them by running docker logs on the container.

[APhatak992]

All the logs in the bouncer look like this right now:

level=info msg="Updating IPSet crowdsec-ipset-a-IPV4-captcha-......" set=crowdsec-ipset-a-IPV4-captcha-.........

level=info msg="Creating IPSet crowdsec-ipset-a-IPV4-captcha-..............." set=crowdsec-ipset-a-IPV4-captcha-.............

level=error msg="Failed to update IPSets: failed to commit ipset changes: failed to create IPSet crowdsec-ipset-a-IPV4-captcha-<......>: operation error WAFV2: CreateIPSet, https response error StatusCode: 400, RequestID: ........, WAFLimitsExceededException: AWS WAF couldn’t perform the operation because you exceeded your resource limit NUM_IP_SETS_BY_ACCOUNT" acl=......... region=........ scope=REGIONAL

level=error msg="Failed to update IPSets: failed to commit ipset changes: failed to create IPSet crowdsec-ipset-a-IPV4-captcha-: operation error WAFV2: CreateIPSet, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: ......., api error ThrottlingException: Rate exceeded" acl=....... region=........scope=REGIONAL

[blotus]

Can you try to get the entire log output and upload it somewhere ? There must be something earlier in the logs that could give us a clue about what is happening.

[APhatak992]

Here is the log file from when the bouncer started

cs-bouncer-logs.txt

[blotus]

The bouncer immediately encouters an error when trying to create the 1st set.

Did you delete all the sets managed previously created before running the bouncer ? If not, you'll need to stop the bouncer, delete them all manually, and start the bouncer again and upload those logs.

Do you have any other sets in your account ? (not managed by the bouncer). There's a default limit of 100 ipsets per account.

[APhatak992]

Yes, I did delete all the ipsets managed by the bouncer.

I also have other ipsets other than the ones managed by the bouncer (less than 10 though).

Do we know why the bouncer is not deleting the resources it creates on WAF it when it is stopped / deleted?

[APhatak992]

Are IPSets gradually added by the bouncefr?

I noticed that yesterday I did not see any IPSets added from the CrowdSec community sourced list but in the morning I saw around 15000.

I re-deployed the bouncer after cleaning up WAF from the resources the earlier bouncer created and I am observing the same behavior. Rule groups are created but the community source IPs to block are no created in IPSets yet.

[blotus]

The sets are created and deleted on demand: the bouncer will look at what decisions it needs to apply, and create the appropriate sets per IP type/decision type. If decisions are deleted and a set becomes empty, it will be automatically deleted (and recreated later if needed).

I don't understand what is going on: I cannot reproduce the behavior you have, really my best bet is some kind of permissions issues on your side.

We have released a new version of bouncer 2 days ago, can you try it ?
I don't have a lot of hope, but it uses a more recent version of the AWS SDK, maybe it will help.

In any case, the new version will log a lot more when run with log_level: trace (it will log the full requests to AWS).
If the issue does not resolve itself by updating, could you run it in trace mode and send the logs to support[at]crowdsec[dot]net ? (note that they will contain data such as your AWS account id, the content of the rules you have enabled in your ACL and other sensitive information, but NO credentials).
Please make sure it's a full run of the bouncer from a clean state (all previous sets have been deleted): start it, let it fail, then stop it before grabbing the logs.

[APhatak992]

I think I know what is going on. I am deploying the WAF Bouncer as a pod through a helm chart. It seems that when the bouncer pod terminates and restarts with a new pod, it is not able to remove the ipsets that it initially created and when the new bouncer pod comes up, it brings in new ipsets to manage (which is why I am seeing a bunch of duplicated IPSets getting created).

Also, I am using the newest version of the bouncer (0.2.0). I can send the logs but they have pretty much the same info I provided in the full logs from above.

Do you know if the issue I am running into with duplicate IPsets being created for the same bouncer definition is a limitation of the WAF Bouncer with helm deployments when I directly use the bouncer image to deploy it as a pod? My guess for this comes after reading this - crowdsecurity/crowdsec#3663

[APhatak992]

Here is a log message that I get that says an IPSet could not be deleted after 3 attempts. Do you know why this can be happening? The behavior of the bouncer is inconsistent in that sometimes it deletes IPSets within 3 attempts and other times it does not.

level=error msg="could not delete set crowdsec-ipset-<………>: operation error WAFV2: DeleteIPSet, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: <……>, api error ThrottlingException: Rate exceeded" acl=<my-acl> component=ipset_manager region=<my-region> scope=REGIONAL

There seems to be an issue other than just this though. I see 6 different bouncer definitions (when I run a cscli bouncers list) and 19 IPSets even though I only defined 4 bouncers.

I expected there to be 3 IPSets / bouncer (12 total IPSets) based on the way the bouncer allocates the 15k IPs. But there are 6 duplicated IPSets (spread across the 4 bouncers).

And there is a 19th IPSet that didn't get deleted due to the above error message. This means there seems to be

1. A bug where the bouncer is not able to delete IPSets due to the throttling exception above
2. incompatibility with the WAF Bouncer when deployed in a kubernetes environment, which seems to be in line with IP address changes cause bouncers to be registered again crowdsec#3663.