From 1032558edcb070a01bf39ee99f3c7932c3259112 Mon Sep 17 00:00:00 2001 From: Thibault Koechlin Date: Thu, 24 Jul 2025 17:45:35 +0200 Subject: [PATCH] moved a bit of content, updated some solutions --- .../unversioned/troubleshooting/intro.md | 177 +----------- .../troubleshooting/security_engine.mdx | 262 ++++++++++++------ 2 files changed, 184 insertions(+), 255 deletions(-) diff --git a/crowdsec-docs/unversioned/troubleshooting/intro.md b/crowdsec-docs/unversioned/troubleshooting/intro.md index 7332d0f7a..d1f5d3dee 100644 --- a/crowdsec-docs/unversioned/troubleshooting/intro.md +++ b/crowdsec-docs/unversioned/troubleshooting/intro.md @@ -11,11 +11,11 @@ You may see CrowdSec referred to as "Security Engine" and Bouncers referred to a We have extended our troubleshooting documentation to cover more common issues and questions. If you have any suggestions for this please open an [issue here](https://github.com/crowdsecurity/crowdsec-docs). -### [Security Engine](/troubleshooting/security_engine.mdx) +### [Security Engine Troubleshooting](/troubleshooting/security_engine.mdx) -### [Remediation Components](/troubleshooting/remediation_components.mdx) +### [Remediation Components Troubleshooting](/troubleshooting/remediation_components.mdx) -### [CTI](/troubleshooting/cti.mdx) +### [CTI Troubleshooting](/troubleshooting/cti.mdx) ## Community support @@ -26,33 +26,19 @@ Please try to resolve your issue by reading the documentation. If you're unable # FAQ -## Security Engine +## How to report a bug -### What is CrowdSec Security Engine ? - -CrowdSec Security Engine is a open-source security software. See the [overview](/docs/next/intro). - -### I've installed the Security Engine, it detects attacks but doesn't block anything ?! - -Yes, the Security Engine is in charge of detecting attacks, and [Remediation Component](/u/bouncers/intro) are applying decisions. -If you want to block the detected IPs, you should deploy a Remediation Component, such as the ones found on the [hub](https://hub.crowdsec.net/browse/#bouncers) ! - -### What language is it written in ? - -CrowdSec Security Engine is written in [Golang](https://golang.org/). - -### What resources are needed to run Security Engine ? - -Security Engine itself is rather light, and in a small to medium setup should use less than 100Mb of memory. +To report a bug, please open an issue on the affected component's repository: -During intensive logs processing, CPU is going to be the most used resource, and memory should grow slightly. +[CrowdSec Repo](https://github.com/crowdsecurity/crowdsec/issues/new/choose) -#### What is the performance impact ? +[CrowdSec Hub Repo](https://github.com/crowdsecurity/hub/issues/new/choose) -As the Security Engine only works on logs, it shouldn't impact your production. -When it comes to [remediation components](/u/bouncers/intro), please refer to their [documentation](/u/bouncers/intro). +:::info +CrowdSec Hub should be used when you have an issue with a parser, scenario or collection. +::: -### What license is provided ? +## What license is provided ? The Security Engine and Remediation Components are provided under [MIT license](https://en.wikipedia.org/wiki/MIT_License). @@ -62,95 +48,6 @@ The Security Engine can easily handle several thousands of events per second on If you need help for large scale deployment, please get in touch with us on the [Form](https://contact.crowdsec.net/business-request), we love challenges ;) -### How to set up proxy - -Setting up a proxy works out of the box, the [net/http golang library](https://golang.org/src/net/http/transport.go) can handle those environment variables: - -- `HTTP_PROXY` -- `HTTPS_PROXY` -- `NO_PROXY` - -For example: - -```bash -export HTTP_PROXY=http://: -``` - -#### Systemd variable - -On Systemd devices you have to set the proxy variable in the environment section for the CrowdSec service. To avoid overwriting the service file during an update, a folder is created in `/etc/systemd/system/crowdsec.service.d` and a file in it named `http-proxy.conf`. The content for this file should look something like this: - -```bash title="systemctl edit crowdsec.service" -[Service] -Environment=HTTP_PROXY=http://myawesomeproxy.com:8080 -Environment=HTTPS_PROXY=https://myawesomeproxy.com:443 -Environment=NO_PROXY=127.0.0.1,localhost,0.0.0.0 -``` - -After this change you need to reload the systemd daemon using: -`systemctl daemon-reload` - -Then you can restart CrowdSec like this: -`systemctl restart crowdsec` - -#### Sudo - -If you use `sudo cscli`, just add this line in `visudo` after setting up the previous environment variables: - -``` -Defaults env_keep += "HTTP_PROXY HTTPS_PROXY NO_PROXY" -``` - -### Tor - -You can configure `cscli` and `crowdsec` to use [tor](https://www.torproject.org/) to anonymously interact with our API. - -All (http) requests made to the central API to go through the [tor network](https://www.torproject.org/). - -With tor installed, setting `HTTP_PROXY` and `HTTPS_PROXY` environment variables to your socks5 proxy, as well as setting `NO_PROXY` to local addresses to prevent LAPI errors, will do the trick. - -#### Edit crowdsec systemd unit to push/pull via tor - -```bash title="systemctl edit crowdsec.service" -[Service] -Environment="HTTPS_PROXY=socks5://127.0.0.1:9050" -Environment="HTTP_PROXY=socks5://127.0.0.1:9050" -Environment="NO_PROXY=127.0.0.1,localhost,0.0.0.0" -``` - -#### Running the wizard with tor - -```bash -$ sudo HTTPS_PROXY=socks5://127.0.0.1:9050 HTTP_PROXY=socks5://127.0.0.1:9050 NO_PROXY=127.0.0.1,localhost,0.0.0.0 ./wizard.sh --bininstall -``` - -:::caution - -Do not use the wizard in interactive (`-i`) mode if you're concerned, as it will start the service at the end of the setup, leaking your IP address. - -::: - -#### cscli - -```bash -$ sudo HTTP_PROXY=socks5://127.0.0.1:9050 HTTPS_PROXY=socks5://127.0.0.1:9050 NO_PROXY=127.0.0.1,localhost,0.0.0.0 cscli capi register -``` - -### How to report a bug - -To report a bug, please open an issue on the affected component's repository: - -[CrowdSec Repo](https://github.com/crowdsecurity/crowdsec/issues/new/choose) - -[CrowdSec Hub Repo](https://github.com/crowdsecurity/hub/issues/new/choose) - -:::info -CrowdSec Hub should be used when you have an issue with a parser, scenario or collection. -::: - -### How to disable the central API - -To disable the central API, simply comment out the [`online_client` section of the configuration file](/docs/next/configuration/crowdsec_configuration#online_client). ### Why are some scenarios/parsers "tainted" or "custom" ? @@ -172,55 +69,3 @@ When using `cscli` to list your parsers, scenarios and collections, some might a ### Which information is sent to your services ? See [CAPI documentation](/docs/next/central_api/intro). - -### How to know if my setup is working correctly ? Some of my logs are unparsed, is it normal ? - -Yes, Security Engine parsers only parse the logs that are relevant for scenarios. - -Take a look at `cscli metrics` [and understand what do they mean](/docs/next/observability/cscli) to know if your setup is correct. - -You can take an extra step and use [`cscli explain` to understand what log lines are parsed, and how.](/docs/next/cscli/cscli_explain) : - -![cscli-explain](/img/cscli_explain.png) - -### Why are X logs not parsed in `cscli metrics` ? - -If you are facing logs that doesn't seem to be parsed correctly, please use [`cscli explain`](/docs/cscli/cscli_explain) : - -``` -# cscli explain --log "May 16 07:50:30 sd-126005 sshd[10041]: Invalid user git from 78.142.18.204 port 47738" --type syslog -line: May 16 07:50:30 sd-126005 sshd[10041]: Invalid user git from 78.142.18.204 port 47738 - β”œ s00-raw - | β”” 🟒 crowdsecurity/syslog-logs (first_parser) - β”œ s01-parse - | β”œ πŸ”΄ crowdsecurity/iptables-logs - | β”œ πŸ”΄ crowdsecurity/mysql-logs - | β”œ πŸ”΄ crowdsecurity/nginx-logs - | β”” 🟒 crowdsecurity/sshd-logs (+6 ~1) - β”œ s02-enrich - | β”œ 🟒 crowdsecurity/dateparse-enrich (+2 ~1) - | β”œ 🟒 crowdsecurity/geoip-enrich (+13) - | β”œ πŸ”΄ crowdsecurity/http-logs - | β”” 🟒 crowdsecurity/whitelists (unchanged) - β”œ-------- parser success 🟒 - β”œ Scenarios - β”œ 🟒 crowdsecurity/ssh-bf - β”œ 🟒 crowdsecurity/ssh-bf_user-enum - β”œ 🟒 crowdsecurity/ssh-slow-bf - β”” 🟒 crowdsecurity/ssh-slow-bf_user-enum -``` - -This command will allow you to see each parser behavior. - -:::warning -Do **not** use `cscli explain` on big log files, as this command will buffer a lot of information in memory to achieve this. -If you want to check crowdsec's behaviour on big log files, please see [replay mode](/u/user_guides/replay_mode/). -::: - -### I want to add collection X : how to add log files, and how to test if it works ? - -When adding a collection to your setup, the [hub](https://hub.crowdsec.net) will usually specify log files to add. - -Those lines need to be added in your acquisition file (`/etc/crowdsec/acquis.yaml` or `/etc/crowdsec/acquis.d/myfile.yaml`). - -After restart, `cscli metrics` will allow you to see if lines are read and/or parsed. diff --git a/crowdsec-docs/unversioned/troubleshooting/security_engine.mdx b/crowdsec-docs/unversioned/troubleshooting/security_engine.mdx index c79f14a89..a8ee7ca4d 100644 --- a/crowdsec-docs/unversioned/troubleshooting/security_engine.mdx +++ b/crowdsec-docs/unversioned/troubleshooting/security_engine.mdx @@ -16,34 +16,19 @@ Please try to resolve your issue by reading the documentation. If you're unable ### How to check the status +:::info + +If you have any doubt about your setup, please use the [HealthCheck Guide](/u/getting_started/health_check) + +::: + + -### How to check if LAPI is reachable - -```bash -cscli lapi status -``` - -
- -Example output - -```bash -time="27-11-2023 13:56:13" level=info msg="Loaded credentials from /etc/crowdsec/local_api_credentials.yaml" -time="27-11-2023 13:56:13" level=info msg="Trying to authenticate with username 6df9e06234fa428d9ba8e78f9b557c4axNHfFR9BmLcUlSfq on http://127.0.0.1:8080/" -time="27-11-2023 13:56:13" level=info msg="You can successfully interact with Local API (LAPI)" -``` - -
- -:::info -This command can be run on child nodes to check if they can reach the LAPI of the parent node. -::: - ### How to check if CAPI is reachable ```bash @@ -65,6 +50,55 @@ time="2024-01-08 14:08:22" level=info msg="You can successfully interact with Ce This command can should **ONLY** be run on the parent node. ::: + +### How to know if my setup is working correctly ? Some of my logs are unparsed, is it normal ? + +Yes, Security Engine parsers only parse the logs that are relevant for scenarios. + +Take a look at `cscli metrics` [and understand what do they mean](/docs/next/observability/cscli) to know if your setup is correct. + +You can take an extra step and use [`cscli explain` to understand what log lines are parsed, and how.](/docs/next/cscli/cscli_explain) : + +![cscli-explain](/img/cscli_explain.png) + +### Why are X logs not parsed in `cscli metrics` ? + +If you are facing logs that doesn't seem to be parsed correctly, please use [`cscli explain`](/docs/cscli/cscli_explain) : + +``` +# cscli explain --log "May 16 07:50:30 sd-126005 sshd[10041]: Invalid user git from 78.142.18.204 port 47738" --type syslog +line: May 16 07:50:30 sd-126005 sshd[10041]: Invalid user git from 78.142.18.204 port 47738 + β”œ s00-raw + | β”” 🟒 crowdsecurity/syslog-logs (first_parser) + β”œ s01-parse + | β”œ πŸ”΄ crowdsecurity/iptables-logs + | β”œ πŸ”΄ crowdsecurity/mysql-logs + | β”œ πŸ”΄ crowdsecurity/nginx-logs + | β”” 🟒 crowdsecurity/sshd-logs (+6 ~1) + β”œ s02-enrich + | β”œ 🟒 crowdsecurity/dateparse-enrich (+2 ~1) + | β”œ 🟒 crowdsecurity/geoip-enrich (+13) + | β”œ πŸ”΄ crowdsecurity/http-logs + | β”” 🟒 crowdsecurity/whitelists (unchanged) + β”œ-------- parser success 🟒 + β”œ Scenarios + β”œ 🟒 crowdsecurity/ssh-bf + β”œ 🟒 crowdsecurity/ssh-bf_user-enum + β”œ 🟒 crowdsecurity/ssh-slow-bf + β”” 🟒 crowdsecurity/ssh-slow-bf_user-enum +``` + +This command will allow you to see each parser behavior. + +:::warning +Do **not** use `cscli explain` on big log files, as this command will buffer a lot of information in memory to achieve this. +If you want to check crowdsec's behaviour on big log files, please see [replay mode](/u/user_guides/replay_mode/). +::: + +### I want to add collection X : how to add log files, and how to test if it works ? + +When adding a collection to your setup, the [hub](https://hub.crowdsec.net) will usually specify log files to add. + ## Decisions ### How to list current decisions @@ -86,25 +120,27 @@ Additional filtering is possible please read [cscli decisions delete](/docs/next The Security Engine is designed to be as flexible as possible, and allows you to control the granularity of decisions. -[Profiles](/docs/next/profiles/intro) allows you to control which decision will be applied to which alert. +[Profiles](/docs/next/local_api/profiles/intro) allows you to control which decision will be applied to which alert. ### How to add whitelists or prevent the Security Engine from banning a given IP -There are many ways to prevent the Security Engine from banning a given IP, depending on your use case. You can see the [whitelist section](/docs/next/whitelist/intro) on how to achieve this. +The recommanded ways to prevent specific IP being banned: + - [Use Centralized AllowLists](/docs/next/local_api/centralized_allowlists) - Works on local decisions, blocklists and can be controlled from the Console. + - [Custom Profiles](/docs/next/local_api/profiles/intro) - Applies to local alerts before Local API applies a decision. + - [Parser Whitelists](/docs/next/log_processor/whitelist/intro) - Applies directly on the log parsing pipeline. #### I need to whitelist an event pattern instead of a single IP address -You can whitelist a specific event pattern by following the [expression section](/docs/next/whitelist/create_expr). +You can exclude specific events [using whitelist parsers](/docs/next/log_processor/whitelist/intro). +It can be used to discard events based on a specific URL, source IP adress or any other condition. ## Scenarios ### Scenario X keeps triggering, it's a false trigger -To address a particular scenario that is causing concern, you can choose from several options: - -- Activate [simulation mode](/docs/next/cscli/cscli_simulation_enable): In this mode, you'll receive alerts, but no actual decisions will be implemented. - -- Completely [remove the scenario](/docs/next/cscli/cscli_scenarios_remove): By doing this, the scenario will be entirely disabled. +To avoid false positives from a known scenario, you can use: + - [Parser Whitelists](/docs/next/log_processor/whitelist/intro) - Applies directly on the log parsing pipeline. + - [Custom Profiles](/docs/next/local_api/profiles/intro) - Applies to local alerts before Local API applies a decision. ### Set a custom/tainted scenario in simulation mode @@ -116,7 +152,7 @@ For example, i have a scenario called `crowdsecurity/my-custom-scenario`Β locate Please see [configuration section](#where-is-configuration-stored) to see where the file will be located on your system. ::: -To enable the simulation mode for this scenario, i need to run: +To enable the simulation mode for this scenario, run: : +``` + +#### Systemd variable + +On Systemd devices you have to set the proxy variable in the environment section for the CrowdSec service. To avoid overwriting the service file during an update, a folder is created in `/etc/systemd/system/crowdsec.service.d` and a file in it named `http-proxy.conf`. The content for this file should look something like this: + +```bash title="systemctl edit crowdsec.service" +[Service] +Environment=HTTP_PROXY=http://myawesomeproxy.com:8080 +Environment=HTTPS_PROXY=https://myawesomeproxy.com:443 +Environment=NO_PROXY=127.0.0.1,localhost,0.0.0.0 +``` + +After this change you need to reload the systemd daemon using: +`systemctl daemon-reload` + +Then you can restart CrowdSec like this: +`systemctl restart crowdsec` + +#### Sudo + +If you use `sudo cscli`, just add this line in `visudo` after setting up the previous environment variables: + +``` +Defaults env_keep += "HTTP_PROXY HTTPS_PROXY NO_PROXY" +``` + +### Tor + +You can configure `cscli` and `crowdsec` to use [tor](https://www.torproject.org/) to anonymously interact with our API. + +All (http) requests made to the central API to go through the [tor network](https://www.torproject.org/). + +With tor installed, setting `HTTP_PROXY` and `HTTPS_PROXY` environment variables to your socks5 proxy, as well as setting `NO_PROXY` to local addresses to prevent LAPI errors, will do the trick. + +#### Edit crowdsec systemd unit to push/pull via tor + +```bash title="systemctl edit crowdsec.service" +[Service] +Environment="HTTPS_PROXY=socks5://127.0.0.1:9050" +Environment="HTTP_PROXY=socks5://127.0.0.1:9050" +Environment="NO_PROXY=127.0.0.1,localhost,0.0.0.0" +``` + +#### Running the wizard with tor + +```bash +$ sudo HTTPS_PROXY=socks5://127.0.0.1:9050 HTTP_PROXY=socks5://127.0.0.1:9050 NO_PROXY=127.0.0.1,localhost,0.0.0.0 ./wizard.sh --bininstall +``` + +:::caution + +Do not use the wizard in interactive (`-i`) mode if you're concerned, as it will start the service at the end of the setup, leaking your IP address. + +::: + +#### cscli + +```bash +$ sudo HTTP_PROXY=socks5://127.0.0.1:9050 HTTPS_PROXY=socks5://127.0.0.1:9050 NO_PROXY=127.0.0.1,localhost,0.0.0.0 cscli capi register +``` + -The community-blocklist that you receive is based on your installed scenarios, if they are neither tainted nor custom. +### How to disable the central API -For example, if your `crowdsecurity/ssh-bf` scenario is tainted, you will not receive IPs concerning this scenario in the `community-blocklist`. +To disable the central API, simply comment out the [`online_client` section of the configuration file](/docs/next/configuration/crowdsec_configuration#online_client).