You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Security Engine runtime revolves around a few simple concepts:
10
+
# Log Processor (LP)
11
11
12
-
- It reads logs (defined via [datasources](/data_sources/introduction.md) configuration)
13
-
- Those logs are parsed via [parsers](/parsers/introduction.mdx) and eventually [enriched](/parsers/enricher.md)
14
-
- Those normalized logs are matched against the [scenarios](/scenarios/introduction.mdx) that the user has deployed
15
-
- When a scenario is "triggered", CrowdSec generates an [alert](/concepts.md#alerts) and eventually one or more associated [decisions](/concepts.md#decisions):
16
-
- The alert is here mostly for traceability and will stay even after the decision expires
17
-
- The decision, on the other hand, is short-lived and tells *what* action should be taken against the offending IP/range/user...
18
-
- This information (the signal, the associated decisions) is then sent to [Local API](/local_api/intro.md) and stored in the database
12
+
> The Log Processor is in charge of the detection of bad behaviors, based on your logs or your HTTP trafic.
19
13
20
-
As you might have guessed by now, the Security Engine itself does the detection part and stores those decisions.
21
-
Then, [remediation components](/u/user_guides/bouncers_configuration) can "consume" those decisions (via the very same [Local API](/local_api/intro.md) and apply the remediation.
14
+
The Log Processor (abreviated as `LP`) detects bad behaviors via two main functions:
15
+
-[Acquire](/log_processor/data_sources/introduction.md) logs, [parse](/log_processor/parsers/introduction.mdx), [enrich](/log_processor/parsers/enricher.md) and match them against [Scenarios](/log_processor/scenarios/introduction.mdx).
16
+
- Receive [HTTP Requests](/log_processor/data_sources/appsc.md) and match them against the [Appsec Rules](/appsec/intro.md).
22
17
23
-
## Crowd sourced aspect
18
+
Alerts resulting from Scenarios or Appsec Rules being triggered are sent to the `LAPI`.
Whenever the [Local API](/local_api/intro.md) receives an alert with associated decisions, it shares the alert's meta-information with our central API:
22
+
> The Local API is the middleman between the Log Processors, the Remediation Components and the Central API.
28
23
29
-
- The source IP address that triggered the alert
30
-
- The scenario that was triggered
31
-
- The timestamp of the attack
24
+
The Local API (abreviated as `LAPI`) has several functions:
25
+
- Receive alerts from Log Processors and create Decisions based on configured [Profiles](/local_api/profiles/intro.md)
26
+
- Expose Decisions to [Remediation Components](/u/bouncers/intro)
27
+
- Interact with the Central API to send Alerts receive Blocklists
32
28
33
-
This is the only data that is sent to our API, and it is processed on our side to be able to redistribute the relevant blocklists to all the participants. You can check the [central API documentation](central_api/intro) in the references link to have a comprehensive view of what might be shared between your instance and our services.
34
29
35
-
##Remediation Components
30
+
# Remediation Components (Bouncers)
36
31
37
-
[[References](/u/bouncers/intro)]
32
+
> The Remediation Components (also called `Bouncers`) are in charge of enforcing decisions.
38
33
39
-
Remediation Components are software packages in charge of acting upon decision's provided by the Security Engine.
40
-
To do so, the component queries the Local API to know if there is an existing decision against a given IP, range, username, etc. [You can find a list of existing Remediation Components on the hub](https://hub.crowdsec.net/browse/#bouncers)
34
+
Remediation Components rely on the Local API to receive decisions about malevolent IPs to be blocked.
41
35
36
+
Those Decisions can be based on behavioral detection made by the `LP` or from Blocklists.
42
37
43
-
# Configuration items
38
+
[Remediations components](https://app.crowdsec.net/hub/remediation-components) laverage existing components of your infrastructure to block malevolent IPs where it matters most.
44
39
45
-
## Acquisition
40
+
#Central API (CAPI)
46
41
47
-
[[References](/data_sources/introduction.md)]
42
+
> The Central API (CAPI) in CrowdSec serves as a pivotal component for aggregating and disseminating threat intelligence across its user community.
48
43
49
-
Acquisition configuration defines which streams of information CrowdSec must process.
50
44
51
-
A stream of information can be a file, a journald event log, a cloudwatch stream, and more or less any kind of stream, such as a Kafka topic.
45
+
The Central API (abreviated as `CAPI`) receives signal from Crowdsec instances and partner networks and will compute them to ultimately create [Cyber Threat Intelligence](/u/cti_api/intro) and [Blocklists](/u/blocklists/intro).
52
46
53
-
Acquisition configuration always contains a stream (ie. a file to tail) and a [tag](/data_sources/introduction.md) (ie. "these are in syslog format" "these are non-syslog nginx logs".md).
47
+
# Console
54
48
55
-
File acquisition configuration is defined as:
49
+
> The CrowdSec Console is a web-based interface that enhances the functionality of the CrowdSec security engine.
56
50
57
-
```yaml
58
-
filenames:
59
-
- /var/log/auth.log
60
-
labels:
61
-
type: syslog
62
-
```
51
+
The [Console](https://app.crowdsec.net) allows you to:
52
+
-[Manage alerts](/u/console/alerts/intro) of your security stack
53
+
-[Manage decisions](/u/console/decisions/decisions_intro) in real-time
54
+
- View and use [blocklists and integrations](/u/blocklists/intro)
55
+
- Manage your API keys ([CTI API](/u/cti_api/intro), [Service API](/u/service_api/getting_started))
63
56
64
-
The `labels` part is here to tag the incoming logs with a type. `labels.type` are used by the parsers to know which logs to process.
57
+
# Security Engine
65
58
66
-
## Application Security Component
59
+
> The Security Engine is a concept that encompasses the Log Processor and the Local API.
67
60
68
-
[[References](appsec/intro)]
69
-
70
-
The Application Security Component is a special datasource that allows the supported remediation components to forward HTTP requests to crowdsec for analysis, before they can reach the application.
71
-
Because CrowdSec can analyse the request and take a decision before it reaches the application, the combo of the Application Security Component and the remediation components acts as a Web Application Firewall (WAF).
72
-
73
-
## Stages
74
-
75
-
[[References](/parsers/introduction.mdx#stages)]
76
-
77
-
The concept of stages is central to data parsing in CrowdSec, as it allows to have various "steps" of parsing. All parsers belong to a given stage. While users can add or modify the stages order, the following stages exist:
78
-
79
-
- `s00-raw`: low-level parser, such as syslog
80
-
- `s01-parse`: most of the services' parsers (ssh, nginx, etc.)
81
-
- `s02-enrich`: enrichment that requires parsed events (ie. geoip-enrichment) or generic parsers that apply on parsed logs (ie. second stage HTTP parser)
82
-
83
-
84
-
Every event starts in the first stage, and moves to the next stage once it has been successfully processed by a parser that has the `onsuccess` directive set to `next_stage`, and so on until it reaches the last stage, when it's going to start to be matched against scenarios. Thus an sshd log might follow this pipeline:
85
-
86
-
- `s00-raw`: parsed by `crowdsecurity/syslog-logs` (will move the event to the next stage)
87
-
- `s01-raw`: parsed by `crowdsecurity/sshd-logs` (will move the event to the next stage)
88
-
- `s02-enrich`: parsed by `crowdsecurity/geoip-enrich` and `crowdsecurity/dateparse-enrich`
89
-
90
-
## Parsers
91
-
92
-
[[References](/parsers/introduction.mdx)]
93
-
94
-
For logs to be able to be exploited and analyzed, they need to be parsed and normalized, and this is where parsers are used.
95
-
96
-
A parser is a YAML configuration file that describes how a string is being parsed. The said string can be a log line, or a field extracted from a previous parser. While a lot of parsers rely on the **GROK** approach (a.k.a regular expression named capture groups), parsers can also reference enrichment modules to allow specific data processing.
97
-
98
-
A parser usually has a specific scope. For example, if you are using [Nginx](https://nginx.org), you will probably want to use the `crowdsecurity/nginx-logs` parser which allows your CrowdSec setup to parse Nginx's access and error logs.
99
-
100
-
Parsers are organized into stages to allow pipelines and branching in parsing.
101
-
102
-
See the [Hub](https://hub.crowdsec.net/browse/#configurations) to explore parsers, or see below some examples:
You can as well [write your own](/parsers/create.md)!
109
-
110
-
111
-
## Enrichers
112
-
113
-
[[References](/parsers/enricher.md)]
114
-
115
-
Enrichment is a parser that adds extra context to a log event so that CrowdSec can later take a better decision. In most cases, you should be able to find the relevant enrichers on our [Hub](https://hub.crowdsec.net/browse/#configurations).
116
-
117
-
A common/simple type of enrichment would be [geoip-enrich](https://github.com/crowdsecurity/hub/blob/master/parsers/s02-enrich/crowdsecurity/geoip-enrich.yaml) of an event (adding information such as origin country, origin Autonomous System and origin IP range to an event).
118
-
119
-
Once again, you should be able to find the ones you're looking for on the [Hub](https://hub.crowdsec.net/browse/#configurations)!
120
-
121
-
## Scenarios
122
-
123
-
[[References](/scenarios/introduction.mdx)]
124
-
125
-
A scenario is the expression of a heuristic that allows you to qualify a specific event (usually an attack). It is a YAML file that describes a set of events characterizing a scenario. Scenarios in CrowdSec gravitate around the [leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket) principle.
126
-
127
-
A scenario description includes at least:
128
-
129
-
- Event eligibility rules. For example, if we're writing an ssh brute-force detection, we only focus on logs of type `ssh_failed_auth`
130
-
- Bucket configuration such as the leak speed or its capacity (in our same ssh brute-force example, we might allow 1 failed auth per 10s and no more than 5 in a short amount of time: `leakspeed: 10s``capacity: 5`)
131
-
- Aggregation rules: per source IP or other criteria (in our ssh brute-force example, we will group per source ip)
132
-
133
-
The description allows for many other rules to be specified (blackhole, distinct filters, etc.), to allow rather complex scenarios.
134
-
135
-
See [Hub](https://hub.crowdsec.net/browse/#configurations) to explore scenarios and their capabilities, or see below some examples:
- [iptables port scan](https://github.com/crowdsecurity/hub/blob/master/scenarios/crowdsecurity/iptables-scan-multi_ports.yaml)
140
-
141
-
You can as well [write your own](/scenarios/create.md)!
142
-
143
-
## AppSec Rules
144
-
145
-
[[References](appsec/intro)]
146
-
147
-
An AppSec rule is a YAML configuration file that describe a rule for the [Application Security Component](appsec/intro).
148
-
149
-
An AppSec rule can match on various aspect of an HTTP request, such as the verb, the URI, the headers, the parameters, the body, etc.
150
-
You can use them to detect and block exploitation of specific vulnerabilities (virtual patching) or to write more generic WAF rules.
151
-
152
-
Contrary to scenarios, appsec rules can block a request before it reaches the application if configured in `in-band` mode.
153
-
154
-
## AppSec Configs
155
-
156
-
An Appsec Config file is a YAML configuration file that tell the [Application Security Component](appsec/intro) which rules should be loaded, and in which band (in-band or out-of-band).
157
-
158
-
The config also expose hooks allowing you to customize at runtime the behavior of the AppSec component (disable a rule on a specific URL, apply a captcha if a specific rule is triggered, etc.)
159
-
160
-
161
-
[[References](appsec/intro)]
162
-
163
-
## Collections
164
-
165
-
[[References](/collections/introduction.md)]
166
-
167
-
To make users' lives easier, "collections" are available, which are just a bundle of parsers and scenarios.
168
-
In this way, if you want to cover basic use-cases of let's say "Nginx", you can just install the `crowdsecurity/nginx` collection that is composed of `crowdsecurity/nginx-logs` parser, as well as generic HTTP scenarios such as `crowdsecurity/base-http-scenarios`.
169
-
170
-
As usual, these can be found on the [Hub](https://hub.crowdsec.net)!
171
-
172
-
## PostOverflows
173
-
174
-
[[References](/parsers/introduction.mdx)]
175
-
176
-
A postoverflow is a parser that is applied on overflows (scenario results) before the decision is written to the local DB or pushed to the API. Parsers in postoverflows are meant to be used for "expensive" enrichment/parsing processes that you do not want to perform on all incoming events, but rather on a decision that is about to be taken.
177
-
178
-
An example could be the slack/mattermost enrichment plugin that requires human confirmation before applying the decision or reverse-dns lookup operations.
179
-
180
-
181
-
# Runtime items
182
-
183
-
## Events
184
-
185
-
[[References](/expr/event.md)]
186
-
187
-
An `Event` is the runtime representation of an item being processed by CrowdSec: it can be a log line being parsed, or an Overflow being reprocessed.
188
-
189
-
The `Event` object is modified by parsers, scenarios, and directly via user [statics expressions](/parsers/format.md#statics) (for example).
190
-
191
-
192
-
<!--TBD: check ref-->
193
-
194
-
## Alerts
195
-
196
-
[[References](/expr/alert.md)]
197
-
198
-
An `Alert` is the runtime representation of a bucket overflow being processed by CrowdSec: it is embedded in an Event.
199
-
200
-
The `Alert` object is modified by post-overflows and [profiles](/profiles/intro.md).
201
-
202
-
## Decisions
203
-
204
-
[[References](/expr/decision.md)]
205
-
206
-
A `Decision` is the representation of the consequence of a bucket overflow: a decision against an IP address, an IP range, an AS, a Country, a User, a Session etc.
207
-
208
-
`Decisions`are generated by the Local API (LAPI) when an `Alert` is received, according to the existing [profiles](/profiles/intro.md)
61
+
The Security Engine is the generic term to describe a Log Processor coupled to a Local API.
0 commit comments