Pangolin
Please Fix: Crowdsec Health Check Config #2119
Replies: 18 comments 30 replies
-
|
Hello, |
Beta Was this translation helpful? Give feedback.
-
|
What is the error that that you see when this happens? Does the crowdsec container remain unhealthy? |
Beta Was this translation helpful? Give feedback.
-
|
These are the container logs; after performing the check 3 times, it restarts in a loop. |
Beta Was this translation helpful? Give feedback.
-
|
You are most likely still rate-limited. As per crowdsecurity/crowdsec#4165, wait 24 hours or reach out to remove rate limit. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, I'll wait. Thanks for replying. |
Beta Was this translation helpful? Give feedback.
-
|
Wow, that was rough. I had to disable Crowdsec in the Pangolin/Traefik docker-compose.yml traefik-config.yml dynamic-config.yml to get Pangolin back up, just so I could use my password manager. So now I need to wait 24 hours to see if I'm no longer blocked by Crowdsec, in order to re-enable it? Ugh. |
Beta Was this translation helpful? Give feedback.
-
|
Have the Same Problem Cant reach my Site anymore |
Beta Was this translation helpful? Give feedback.
-
|
can this be resolved by removing crowdsec or something? |
Beta Was this translation helpful? Give feedback.
-
|
tried nothing changed HTTP ERROR 403 this is a big issue for me |
Beta Was this translation helpful? Give feedback.
-
yes, this is breaking my setup big time. |
Beta Was this translation helpful? Give feedback.
-
|
found a fix remove anything with crowdsec: |
Beta Was this translation helpful? Give feedback.
-
yep, i did this on both docker compose file + traefik config file. things are working again |
Beta Was this translation helpful? Give feedback.
-
This did not fix it for me. Still getting the 403. |
Beta Was this translation helpful? Give feedback.
-
|
It is my understanding that fixing the health-check won't immediately whitelist your IP on crowdsec's AWS WAF blacklist, you still need to wait the full 24hrs cooldown - or contact them via email. On a different note, if you are in a hurry I found that adding the following env variables for the crowdsec's service will momentarily skip any authentication and let the container start: Again, this is a temp fix and you should be able to resume normal operation, removing the config above, after the ban on your IP gets lifted. Hopefully this helps someone. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome thanks! That fixed it and hopefully I’m able to remove it after 24hours! |
Beta Was this translation helpful? Give feedback.
-
|
This fixed it for me while I'll wait for them to lift my IP ban. Already emailed them. Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
I wonder what people should do in this case with a dynamic IP |
Beta Was this translation helpful? Give feedback.
-
|
If you're facing container start issues after updating your health checks, look at the logs, you may see a 403 indicating a ban on Crowdstrikes side. Solution is to wait, or do what I did and email the address listed here: crowdsecurity/crowdsec#4165 As someone else said, their support is incredibly fast. I emailed at 9:06pm and got a reply back confirming the addition to allowlist at 10:00pm. Incredible support! 👏 |
Beta Was this translation helpful? Give feedback.
-
|
can confirm all pangolin instances with crowdsec back up after 24 hours (if you'd rather not send your ips over):
heads up:
|
Beta Was this translation helpful? Give feedback.
-
|
Hey all, I followed the advice here and
At first everything worked, but after 3 hours of the services running without issue again, CrowdSec went down with the exact same errors discussed in this thread. downloading https://hub-data.crowdsec.net/mmdb_update/GeoLite2-City.mmdb
Error: cscli hub upgrade: while downloading data for /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml: while getting data: bad HTTP code 403 for https://hub-data.crowdsec.net/mmdb_update/GeoLite2-City.mmdb
# ...
time="2025-12-21T05:52:04Z" level=info msg="Enabled feature flags: none"
time="2025-12-21T05:52:04Z" level=info msg="Crowdsec v1.7.4-db3efdbf"
time="2025-12-21T05:52:04Z" level=info msg="Loading CAPI manager"
time="2025-12-21T05:52:04Z" level=info msg="attempt 1 out of 2"
time="2025-12-21T05:52:04Z" level=info msg="attempt 2 out of 2"
time="2025-12-21T05:52:05Z" level=info msg="max attempts reached for status code 403"
time="2025-12-21T05:52:05Z" level=fatal msg="api server init: unable to run local API: authenticate watcher (8d576a5ffd9f4907bfaf975cc3239529Q8SCIIoRK71s6VtO): API error: Forbidden"Does the stack re-fetch the CAPI data while running? Is anyone else having issues with this? I really want CrowdSec in front of my services but I can't deal with constant outages like this. |
Beta Was this translation helpful? Give feedback.
-
|
update: services have been down and the |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I followed the recommended fix of removing capi from the health check. My docker compose logs have only capi references that look like this It seems to be holding up, I will write an update in a few hours / a day if its still holding up. Hope this helps, in case your initial logs would've looked something like this. |
Beta Was this translation helpful? Give feedback.
-
Can confirm it hasn't fallen over so far. So I reckon you may have had another source hammering the api / your config didn't actually remove CAPI from health check. |
Beta Was this translation helpful? Give feedback.
-
|
interesting. just to confirm, did you stop your crowdsec container for 24 hours after updating your healthchecks? |
Beta Was this translation helpful? Give feedback.
-
updated my healtcheck in docker-compose.yml |
Beta Was this translation helpful? Give feedback.
-
|
Having the same issue. Even after waiting for 24 hours for the rate limit to reset it ran fine for a few hours then got cut again |
Beta Was this translation helpful? Give feedback.
-
|
exactly the same for me too |
Beta Was this translation helpful? Give feedback.
-
|
Remove capi and wait 24h. |
Beta Was this translation helpful? Give feedback.
-
We have already done this. See my comment above. Even with the suggested changes the service will eventually come down again. |
Beta Was this translation helpful? Give feedback.
-
Could you post your current healthcheck from docker-compose please? |
Beta Was this translation helpful? Give feedback.
-
|
I have these logs |
Beta Was this translation helpful? Give feedback.
-
|
is this rate limiting or what |
Beta Was this translation helpful? Give feedback.
-
|
I have also experienced this even after applying the new settings. No choice but to disable Crowdsec for now. |
Beta Was this translation helpful? Give feedback.
-
|
check out
if the above is true and your crowdsec container is still running / not constantly restarting
if not you may have to reenroll again on the crowdsec console dashboard |
Beta Was this translation helpful? Give feedback.
-
|
After this changes my Traefik won't start anymore. I need to comment out the crowedsec health check for pangolin, than everything is up and running.
|
Beta Was this translation helpful? Give feedback.
-
|
ok, just ran also in this issue. had to disable everything in the configs (compose/traefik/dynamic) regarding crowdsec to get pangolin up and running again. i guess because i was already blocked? will try to enable everything tomorrow with the changes provided here. |
Beta Was this translation helpful? Give feedback.
-
|
I got Pangolin running by adding these two entries to docker-compose. I'm waiting 24 hours to retry the CrowdSec health check. |
Beta Was this translation helpful? Give feedback.
-
3 days later i tried it with the provided settings again and the stack is running again |
Beta Was this translation helpful? Give feedback.
-
same error here, seems that parser is no longer able to correctly interpret the logs and not ban, always worked, with the last update now i got 88 hits but 88 unparsed from the metrics... EDIT: fixed with crowdsecurity/hub#1643 |
Beta Was this translation helpful? Give feedback.
-
|
I've updated the healthcheck, but I'm having the issue of receiving this error everytime I restart my Pangolin stack. It could be up for weeks, but if I change a config and |
Beta Was this translation helpful? Give feedback.
-
|
I feel like I am getting the same thing. |
Beta Was this translation helpful? Give feedback.
-
|
as i wrote before i thought running it with the here provided paramaters work but it didn't. strange behaviours so i disabled all crowdsec related config again. will have closer look if i get some time.. |
Beta Was this translation helpful? Give feedback.
-
|
Same. My VPS was fine but ran out of space and now the crowdsec container won't come back up. That will be the variables needed, but it does mean you are unprotected from new threats. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I am having the same issue with crowdsec health check not letting pangolin stack start. |
Beta Was this translation helpful? Give feedback.
-
|
The instructions in the previous messages in this thread worked for me. Disable API for 24h: And update the health check to use lapi: |
Beta Was this translation helpful? Give feedback.
-
|
I ran into the same problem today after updating my vps and the instructions worked for me as well. Glad I found this discussion here. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Yesterday I also had the same problem (403) upgrading Pangolin from 1.14 to 1.15, so I followed instruction above: Today I revert the above, and now everything is working as usual. Thanks for all the suggestions. |
Beta Was this translation helpful? Give feedback.
-
|
I was just starting my stack for a new install and I also get hit with this rate limit. Maybe it is only happening on grabbing a large amount at a time? Going to wait and try again and see if it is able to finish this time. |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hey Everyone!
Due to a misconfiguration of the default Crowdsec install with the Pangolin installer we are hammering Crowdsec's API with health checks! If everyone could please update their installs as soon as possible that would really help out the team over there.
An update from Crowdsec: crowdsecurity/crowdsec#4165
Edit your
docker-compose.ymland update the health check section of the Crowdsec section to be the following:Then run
docker compose up -dto apply the changes.Note the change to
lapiand an increased interval.Beta Was this translation helpful? Give feedback.
All reactions