From c0241905d574828f87e0360fb6fee06c5cff7e04 Mon Sep 17 00:00:00 2001 From: Thibault Koechlin Date: Mon, 2 Dec 2024 16:36:59 +0100 Subject: [PATCH 1/3] document events generated by the WAF and how they can be used by scenarios --- .../docs/appsec/alerts_and_scenarios.md | 147 ++++++++++++++++++ crowdsec-docs/sidebars.js | 2 + 2 files changed, 149 insertions(+) create mode 100644 crowdsec-docs/docs/appsec/alerts_and_scenarios.md diff --git a/crowdsec-docs/docs/appsec/alerts_and_scenarios.md b/crowdsec-docs/docs/appsec/alerts_and_scenarios.md new file mode 100644 index 000000000..1d92e510b --- /dev/null +++ b/crowdsec-docs/docs/appsec/alerts_and_scenarios.md @@ -0,0 +1,147 @@ +--- +id: alerts_and_scenarios +title: AppSec Alerts & Scenarios +sidebar_position: 5 +--- + +## Generated Events Layout + +When HTTP Requests trigger _InBand_ or _OutOfBand_ AppSec/WAF rules, an event is generated. Events can be used to create scenarios that will react (ban, alerts etc.) when AppSec/WAF rules are matched. + +The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is used to parse those events and turn them into events that are easier to process with scenarios. + + +The generated event looks like: + + - `evt.Meta.service` is set to `appsec` + - `evt.Meta.log_type`: + - `appsec-block` for blocked requests (_InBand_ rule matched for ex) + - `appsec-info` for non-blocked reuqests that triggered _OutOfBand_ rule + - `evt.Meta.source_ip` is set to the source (client) IP + - `evt.Meta.target_host` is set to the FQDN if present (`Host` header in the HTTP request) + - `evt.Meta.target_uri` is set to the full URI of the HTTP request + - `evt.Meta.rule_name` is set to the name of the triggered rule + - `evt.Meta.remediation_cmpt_ip` is set to the IP of the Remediation Component (Bouncer) that sent the HTTP request. + +:::info +The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is already part of the generic AppSec/WAF collections, and doesn't have to be manually installed. +::: + + +## Creating Scenario Based on AppSec/WAF Events + +### Triggering on InBand Rules + +A simple yet effective example is the [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) that will ban IPs triggering two distinct _InBand_ rules: + +```yaml title="/etc/crowdsec/scenarios/appsec-vpatch.yaml" +type: leaky +name: crowdsecurity/appsec-vpatch +filter: "evt.Meta.log_type == 'appsec-block'" +distinct: evt.Meta.rule_name +leakspeed: "60s" +capacity: 1 +groupby: evt.Meta.source_ip +... +``` + +:::info +The [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) is already part of the generic AppSec/WAF collections, and doesn't have to be manually installed. +::: + +### Triggering on OutOfBand Rules + +Let's try to solve an imaginary scenario: + +> We want to block users enumerating some URLs (`/foobar/*`) when a specific HTTP header is set (`something: *test*`). However, we only want to block/ban users that will try to access 2 or more distinct `/foobar/*` URLs while having this header set. + +:::info +Keep in mind that _OutOfBand_ rules will generate an event instead of blocking the HTTP Request. +::: + +#### The AppSec/WAF Rule + +This is our AppSec/WAF rule: + +```yaml title="/etc/crowdsec/appsec-rules/foobar-access.yaml" +name: crowdsecurity/foobar-access +description: "Detect access to foobar files with the something header set" +rules: + - zones: + - URI + transform: + - lowercase + match: + type: startsWith + value: /foobar/ + - zones: + - HEADERS + variables: + - something + transform: + - lowercase + match: + type: contains + value: test +``` + +Let ensure it's loaded as an _OutOfBand_ rule: + +```yaml title="/etc/crowdsec/appsec-configs/appsec-default.yaml" +name: crowdsecurity/appsec-default +default_remediation: ban +inband_rules: + - crowdsecurity/base-config + - crowdsecurity/vpatch-* + - crowdsecurity/generic-* +#Let's add our rule as an out-of-band rule +outofband_rules: + - crowdsecurity/foobar-access +``` + +#### The Scenario + +We can now create a scenario that will trigger when a single IPs triggers this rule on distinct URLs: + +```yaml title="/etc/crowdsec/scenarios/foobar-enum.yaml" +type: leaky +format: 3.0 +name: crowdsecurity/foobar-enum +description: "Ban IPs repeateadly triggering out of band rules" +filter: "evt.Meta.log_type == 'appsec-info' && evt.Meta.rule_name == 'crowdsecurity/foobar-access'" +distinct: evt.Meta.target_uri +leakspeed: "60s" +capacity: 1 +groupby: evt.Meta.source_ip +blackhole: 1m +labels: + remediation: true +``` + +:::info +The `filter` ensures only OutOfBand events generated by our scenario are picked up, while the `capacity: 1` and `distinct: evt.Meta.target_uri` will ensure that the IP has to trigger the rule on at least 2 distinct URLs to trigger the scenario. +::: + +#### Testing + +Let's now test our setup: + +```bash +$ curl -I localhost/foobar/1 -H 'something: test' +HTTP/1.1 404 Not Found + +$ curl -I localhost/foobar/2 -H 'something: test' +HTTP/1.1 404 Not Found + +$ curl -I localhost/foobar/3 -H 'something: test' +HTTP/1.1 403 Forbidden +``` + +And CrowdSec logs will show: + +``` +INFO[2024-12-02T15:28:16+01:00] Ip ::1 performed 'crowdsecurity/foobar-enum' (2 events over 4.780233613s) at 2024-12-02 14:28:16.858419797 +0000 UTC +INFO[2024-12-02T15:28:17+01:00] (test/crowdsec) crowdsecurity/foobar-enum by ip ::1 (/0) : 4h ban on Ip ::1 +``` + +As expected, the first two requests were processed without being blocked. The second one triggered the scenario, resulting in the third request being blocked by the bouncer. diff --git a/crowdsec-docs/sidebars.js b/crowdsec-docs/sidebars.js index 504a1f918..6471edd5b 100644 --- a/crowdsec-docs/sidebars.js +++ b/crowdsec-docs/sidebars.js @@ -1,3 +1,4 @@ + module.exports = { // By default, Docusaurus generates a sidebar from the docs folder structure //tutorialSidebar: [{type: 'autogenerated', dirName: '.'}], @@ -760,6 +761,7 @@ module.exports = { { type: "doc", id: "appsec/create_rules" }, ], }, + { type: "doc", id: "appsec/alerts_and_scenarios", label: "Alerts & Scenarios" }, { type: "doc", id: "appsec/installation" }, { type: "doc", id: "appsec/protocol", label: "Communication Protocol" }, { type: "doc", id: "appsec/benchmark", label: "Benchmark" }, From 7d4bb5da71e29901f562fbd3e9e3d272d3053bd8 Mon Sep 17 00:00:00 2001 From: Thibault Koechlin Date: Mon, 2 Dec 2024 17:27:30 +0100 Subject: [PATCH 2/3] up --- .../docs/appsec/alerts_and_scenarios.md | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/crowdsec-docs/docs/appsec/alerts_and_scenarios.md b/crowdsec-docs/docs/appsec/alerts_and_scenarios.md index 1d92e510b..c1efcb78e 100644 --- a/crowdsec-docs/docs/appsec/alerts_and_scenarios.md +++ b/crowdsec-docs/docs/appsec/alerts_and_scenarios.md @@ -16,7 +16,7 @@ The generated event looks like: - `evt.Meta.service` is set to `appsec` - `evt.Meta.log_type`: - `appsec-block` for blocked requests (_InBand_ rule matched for ex) - - `appsec-info` for non-blocked reuqests that triggered _OutOfBand_ rule + - `appsec-info` for reuqests that triggered _OutOfBand_ rule (not blocked) - `evt.Meta.source_ip` is set to the source (client) IP - `evt.Meta.target_host` is set to the FQDN if present (`Host` header in the HTTP request) - `evt.Meta.target_uri` is set to the full URI of the HTTP request @@ -85,20 +85,28 @@ rules: value: test ``` -Let ensure it's loaded as an _OutOfBand_ rule: +Let ensure it's loaded as an _OutOfBand_ rule, first by creating a new appsec-config: -```yaml title="/etc/crowdsec/appsec-configs/appsec-default.yaml" -name: crowdsecurity/appsec-default +```yaml title="/etc/crowdsec/appsec-configs/appsec-oob.yaml" +name: crowdsecurity/appsec-oob default_remediation: ban -inband_rules: - - crowdsecurity/base-config - - crowdsecurity/vpatch-* - - crowdsecurity/generic-* #Let's add our rule as an out-of-band rule outofband_rules: - crowdsecurity/foobar-access ``` +And then make sure this appsec-config is loaded: + +```yaml title="/etc/crowdsec/acquis.d/appsec.yaml" +appsec_configs: + - crowdsecurity/appsec-default + - crowdsecurity/appsec-oob +labels: + type: appsec +listen_addr: 127.0.0.1:7422 +source: appsec +``` + #### The Scenario We can now create a scenario that will trigger when a single IPs triggers this rule on distinct URLs: From c422236d39e55044a5d7d588a33ce53a313fe2f5 Mon Sep 17 00:00:00 2001 From: Thibault Koechlin Date: Tue, 3 Dec 2024 11:05:59 +0100 Subject: [PATCH 3/3] up --- .../docs/appsec/alerts_and_scenarios.md | 28 +++++++++---------- crowdsec-docs/docs/appsec/benchmark.md | 4 +-- crowdsec-docs/docs/appsec/configuration.md | 8 +++--- crowdsec-docs/docs/appsec/installation.md | 2 +- 4 files changed, 21 insertions(+), 21 deletions(-) diff --git a/crowdsec-docs/docs/appsec/alerts_and_scenarios.md b/crowdsec-docs/docs/appsec/alerts_and_scenarios.md index c1efcb78e..c05f9d5eb 100644 --- a/crowdsec-docs/docs/appsec/alerts_and_scenarios.md +++ b/crowdsec-docs/docs/appsec/alerts_and_scenarios.md @@ -6,33 +6,33 @@ sidebar_position: 5 ## Generated Events Layout -When HTTP Requests trigger _InBand_ or _OutOfBand_ AppSec/WAF rules, an event is generated. Events can be used to create scenarios that will react (ban, alerts etc.) when AppSec/WAF rules are matched. +HTTP requests that trigger _In-Band_ or _Out-Of-Band_ AppSec/WAF rules generate events. These events can trigger scenarios that react by banning or alerting when rules are matched. -The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is used to parse those events and turn them into events that are easier to process with scenarios. +The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is designed as a general-purpose tool to convert events into a format that is easier to process with scenarios. -The generated event looks like: +The generated event looks like this: - `evt.Meta.service` is set to `appsec` - `evt.Meta.log_type`: - - `appsec-block` for blocked requests (_InBand_ rule matched for ex) - - `appsec-info` for reuqests that triggered _OutOfBand_ rule (not blocked) + - `appsec-block` for blocked requests (_In-Band_ rule matched, for example) + - `appsec-info` for requests that triggered _Out-Of-Band_ rule (not blocked) - `evt.Meta.source_ip` is set to the source (client) IP - `evt.Meta.target_host` is set to the FQDN if present (`Host` header in the HTTP request) - - `evt.Meta.target_uri` is set to the full URI of the HTTP request + - `evt.Meta.target_uri` is set to the complete URI of the HTTP request - `evt.Meta.rule_name` is set to the name of the triggered rule - `evt.Meta.remediation_cmpt_ip` is set to the IP of the Remediation Component (Bouncer) that sent the HTTP request. :::info -The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is already part of the generic AppSec/WAF collections, and doesn't have to be manually installed. +The [`crowdsecurity/appsec-logs` parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-logs) is already part of the generic AppSec/WAF collections and doesn't have to be manually installed. ::: ## Creating Scenario Based on AppSec/WAF Events -### Triggering on InBand Rules +### Triggering on _In-Band_ Rules -A simple yet effective example is the [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) that will ban IPs triggering two distinct _InBand_ rules: +A simple yexample is the [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) that will ban IPs triggering two distinct _In-Band_ rules: ```yaml title="/etc/crowdsec/scenarios/appsec-vpatch.yaml" type: leaky @@ -49,14 +49,14 @@ groupby: evt.Meta.source_ip The [`crowdsecurity/appsec-vpatch` scenario](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/appsec-vpatch) is already part of the generic AppSec/WAF collections, and doesn't have to be manually installed. ::: -### Triggering on OutOfBand Rules +### Triggering on Out-Of-Band Rules Let's try to solve an imaginary scenario: -> We want to block users enumerating some URLs (`/foobar/*`) when a specific HTTP header is set (`something: *test*`). However, we only want to block/ban users that will try to access 2 or more distinct `/foobar/*` URLs while having this header set. +> We aim to prevent users from enumerating certain URLs (specifically, those that begin with `/foobar/*`) when a particular HTTP header is present (`something: *test*`). However, we want to impose this restriction only on users attempting to access two or more distinct `/foobar/*` URLs while this header is set. :::info -Keep in mind that _OutOfBand_ rules will generate an event instead of blocking the HTTP Request. +Keep in mind that _Out-Of-Band_ rules will generate an event instead of blocking the HTTP Request. ::: #### The AppSec/WAF Rule @@ -85,7 +85,7 @@ rules: value: test ``` -Let ensure it's loaded as an _OutOfBand_ rule, first by creating a new appsec-config: +Let ensure it's loaded as an _Out-Of-Band_ rule, first by creating a new appsec-config: ```yaml title="/etc/crowdsec/appsec-configs/appsec-oob.yaml" name: crowdsecurity/appsec-oob @@ -127,7 +127,7 @@ labels: ``` :::info -The `filter` ensures only OutOfBand events generated by our scenario are picked up, while the `capacity: 1` and `distinct: evt.Meta.target_uri` will ensure that the IP has to trigger the rule on at least 2 distinct URLs to trigger the scenario. +The `filter` ensures only _Out-Of-Band_ events generated by our scenario are picked up, while the `capacity: 1` and `distinct: evt.Meta.target_uri` will ensure that the IP has to trigger the rule on at least 2 distinct URLs to trigger the scenario. ::: #### Testing diff --git a/crowdsec-docs/docs/appsec/benchmark.md b/crowdsec-docs/docs/appsec/benchmark.md index 4eb3e47fa..307d91892 100644 --- a/crowdsec-docs/docs/appsec/benchmark.md +++ b/crowdsec-docs/docs/appsec/benchmark.md @@ -33,8 +33,8 @@ The benchmarks cover the following tests: Each test has been run with multiple cases: - Application Security Component enabled but without any rules -- Application Security Component enabled with 100 vpatch rules (in inband) -- Application Security Component enabled with all the CRS (in inband) +- Application Security Component enabled with 100 vpatch rules (in in-band) +- Application Security Component enabled with all the CRS (in in-band) On the system, we deployed: diff --git a/crowdsec-docs/docs/appsec/configuration.md b/crowdsec-docs/docs/appsec/configuration.md index 84ae7e584..d715de965 100644 --- a/crowdsec-docs/docs/appsec/configuration.md +++ b/crowdsec-docs/docs/appsec/configuration.md @@ -10,7 +10,7 @@ Configuring the AppSec Component usually requires the use of multiple files: - [AppSec rules](/appsec/rules_syntax.md) allow you to write a signature to detect and/or block malevolent requests. [You can find more information about the syntax here](/appsec/rules_syntax.md) - [acquisition configuration](/data_sources/appsec.md) indicates which port is the AppSec Component listening on, and which AppSec configuration it will use. - - AppSec configuration tells which rules are loaded in inband (blocking) and out-of-band (non-blocking) + - AppSec configuration tells which rules are loaded in in-band (blocking) and out-of-band (non-blocking) phases. [it as well allows you to tweak the behavior of the component via the powerful expr bindings](/appsec/rules_syntax.md) @@ -40,11 +40,11 @@ A supplementary list of rules can be loaded during the out-of-band phase. These ### `inband_rules` -An optional list of rules to be loaded in inband phase. In band rules are blocking and evaluated before answering the remediation component. Useful for virtual patching, rules with no/low false positives. +An optional list of rules to be loaded in in-band phase. In band rules are blocking and evaluated before answering the remediation component. Useful for virtual patching, rules with no/low false positives. ### `default_remediation` -An optional remediation for inband rules, defaults to `ban`. If set to `allow`, remediation component won't block the request (even if it matched rules). Any other value (including `captcha`) is passed as-is back to the remediation component. +An optional remediation for in-band rules, defaults to `ban`. If set to `allow`, remediation component won't block the request (even if it matched rules). Any other value (including `captcha`) is passed as-is back to the remediation component. ### `default_pass_action` @@ -84,6 +84,6 @@ See the [dedicated doc](/docs/appsec/hooks.md#on_match) ### `inband_options` and `outofband_options` -Subset of options that can be applied to the inband/outofband rules: +Subset of options that can be applied to the in-band/out-of-band rules: - `disable_body_inspection` : boolean, allows to disable HTTP body inspection - `request_body_in_memory_limit` : a number of byes indicating the maximum body size to be loaded in memory diff --git a/crowdsec-docs/docs/appsec/installation.md b/crowdsec-docs/docs/appsec/installation.md index ee3179260..1ed974081 100644 --- a/crowdsec-docs/docs/appsec/installation.md +++ b/crowdsec-docs/docs/appsec/installation.md @@ -24,7 +24,7 @@ To have a functional AppSec Component, you need: Before diving into the practical steps, it's crucial to familiarize yourself with the core configuration aspects of the AppSec Component: - **acquisition configuration**: Specifies how to acquire the AppSec Component stream of data -- **AppSec Component configuration**: Tells which rules are loaded in inband (blocking) and out-of-band (non-blocking) +- **AppSec Component configuration**: Tells which rules are loaded in in-band (blocking) and out-of-band (non-blocking) phases, [and allows tweaking the behavior of the component via the powerful expr bindings](/appsec/hooks.md). - **rules** allow writing a [signature to detect and/or block malevolent requests](/appsec/rules_syntax.md).