You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Security Engine runtime revolves around a few simple concepts:
10
+
#Security Engine
11
11
12
-
- It reads logs (defined via [datasources](/data_sources/introduction.md) configuration)
13
-
- Those logs are parsed via [parsers](/parsers/introduction.mdx) and eventually [enriched](/parsers/enricher.md)
14
-
- Those normalized logs are matched against the [scenarios](/scenarios/introduction.mdx) that the user has deployed
15
-
- When a scenario is "triggered", CrowdSec generates an [alert](/concepts.md#alerts) and eventually one or more associated [decisions](/concepts.md#decisions):
16
-
- The alert is here mostly for traceability and will stay even after the decision expires
17
-
- The decision, on the other hand, is short-lived and tells *what* action should be taken against the offending IP/range/user...
18
-
- This information (the signal, the associated decisions) is then sent to [Local API](/local_api/intro.md) and stored in the database
12
+
> The Security Engine is CrowdSec's IDS/IPS (Intrusion Detection System/Intrusion Prevention System)
13
+
> It is a rules and behavior detection engine comprised of Log Processor and the Local API.
19
14
20
-
As you might have guessed by now, the Security Engine itself does the detection part and stores those decisions.
21
-
Then, [remediation components](/u/user_guides/bouncers_configuration) can "consume" those decisions (via the very same [Local API](/local_api/intro.md) and apply the remediation.
15
+
A Security Engine can operate [independently](/intro#architecture) or in a [distributed manner](/intro#deployment-options), adapting to the specific needs and constraints of your infrastructure. For more information on CrowdSec's distributed approach, visit our documentation on collaborative operations and distributed deployments.
Whenever the [Local API](/local_api/intro.md) receives an alert with associated decisions, it shares the alert's meta-information with our central API:
20
+
> The Log Processor is the part of the Security Engine in charge of the detection of bad behaviors, based on your logs or your HTTP trafic.
28
21
29
-
- The source IP address that triggered the alert
30
-
-The scenario that was triggered
31
-
-The timestamp of the attack
22
+
The Log Processor (abreviated as `LP`) detects bad behaviors via two main functions:
23
+
-[Acquire](/log_processor/data_sources/introduction.md) logs, [parse](/log_processor/parsers/introduction.mdx), [enrich](/log_processor/parsers/enricher.md) and match them against [Scenarios](/log_processor/scenarios/introduction.mdx).
24
+
-Receive [HTTP Requests](/log_processor/data_sources/appsec.md) and match them against the [Appsec Rules](/appsec/intro.md).
32
25
33
-
This is the only data that is sent to our API, and it is processed on our side to be able to redistribute the relevant blocklists to all the participants. You can check the [central API documentation](central_api/intro) in the references link to have a comprehensive view of what might be shared between your instance and our services.
26
+
Alerts resulting from Scenarios or Appsec Rules being triggered are sent to the `LAPI`.
34
27
35
-
## Remediation Components
28
+
#Local API (LAPI)
36
29
37
-
[[References](/u/bouncers/intro)]
30
+
> The Local API is the part of the Security Engine acting as the middleman between the Log Processors, the Remediation Components and the Central API.
38
31
39
-
Remediation Components are software packages in charge of acting upon decision's provided by the Security Engine.
40
-
To do so, the component queries the Local API to know if there is an existing decision against a given IP, range, username, etc. [You can find a list of existing Remediation Components on the hub](https://hub.crowdsec.net/browse/#bouncers)
32
+
The Local API (abreviated as `LAPI`) has several functions:
33
+
- Receive alerts from Log Processors and create Decisions based on configured [Profiles](/local_api/profiles/intro.md)
34
+
- Expose Decisions to [Remediation Components](/u/bouncers/intro)
35
+
- Interact with the Central API to send Alerts receive Blocklists
41
36
42
37
43
-
# Configuration items
38
+
# Remediation Components (Bouncers)
44
39
45
-
## Acquisition
40
+
> The Remediation Components (also called `Bouncers`) are external components in charge of enforcing decisions.
46
41
47
-
[[References](/data_sources/introduction.md)]
42
+
Remediation Components rely on the Local API to receive decisions about malevolent IPs to be blocked *(or other supported types or remediations such as Captcha, supported by some of our Bouncers).*
43
+
*Note that they also support [CrowdSec's Blocklist as a Service](/u/integrations/intro).*
48
44
49
-
Acquisition configuration defines which streams of information CrowdSec must process.
45
+
Those Decisions can be based on behavioral detection made by the `LP` or from Blocklists.
50
46
51
-
A stream of information can be a file, a journald event log, a cloudwatch stream, and more or less any kind of stream, such as a Kafka topic.
47
+
Remediations components leverage existing components of your infrastructure to block malevolent IPs where it matters most. You can find them on our [Remediation Components' HUB](https://app.crowdsec.net/hub/remediation-components)
52
48
53
-
Acquisition configuration always contains a stream (ie. a file to tail) and a [tag](/data_sources/introduction.md) (ie. "these are in syslog format" "these are non-syslog nginx logs".md).
49
+
# Central API (CAPI)
54
50
55
-
File acquisition configuration is defined as:
51
+
> The Central API (CAPI) serves as the gateway for network participants to connect and communicate with CrowdSec's network.
56
52
57
-
```yaml
58
-
filenames:
59
-
- /var/log/auth.log
60
-
labels:
61
-
type: syslog
62
-
```
53
+
The Central API (abreviated as `CAPI`) receives attack signals from all participating Security Engines and signal partners, then re-distribute them curated community decisions ([Community Blocklist](/central_api/community_blocklist/)).
54
+
It's also at the heart of CrowdSec centralized [Blocklist services](/u/blocklists/intro).
63
55
64
-
The `labels` part is here to tag the incoming logs with a type. `labels.type` are used by the parsers to know which logs to process.
56
+
# Console
65
57
66
-
## Application Security Component
58
+
> The CrowdSec Console is a web-based interface providing reporting, alerting, management and QoL features to CrowdSec's products usages: from your park of Security Engines to the management of CTI related actions
67
59
68
-
[[References](appsec/intro)]
69
-
70
-
The Application Security Component is a special datasource that allows the supported remediation components to forward HTTP requests to crowdsec for analysis, before they can reach the application.
71
-
Because CrowdSec can analyse the request and take a decision before it reaches the application, the combo of the Application Security Component and the remediation components acts as a Web Application Firewall (WAF).
72
-
73
-
## Stages
74
-
75
-
[[References](/parsers/introduction.mdx#stages)]
76
-
77
-
The concept of stages is central to data parsing in CrowdSec, as it allows to have various "steps" of parsing. All parsers belong to a given stage. While users can add or modify the stages order, the following stages exist:
78
-
79
-
- `s00-raw`: low-level parser, such as syslog
80
-
- `s01-parse`: most of the services' parsers (ssh, nginx, etc.)
81
-
- `s02-enrich`: enrichment that requires parsed events (ie. geoip-enrichment) or generic parsers that apply on parsed logs (ie. second stage HTTP parser)
82
-
83
-
84
-
Every event starts in the first stage, and moves to the next stage once it has been successfully processed by a parser that has the `onsuccess` directive set to `next_stage`, and so on until it reaches the last stage, when it's going to start to be matched against scenarios. Thus an sshd log might follow this pipeline:
85
-
86
-
- `s00-raw`: parsed by `crowdsecurity/syslog-logs` (will move the event to the next stage)
87
-
- `s01-raw`: parsed by `crowdsecurity/sshd-logs` (will move the event to the next stage)
88
-
- `s02-enrich`: parsed by `crowdsecurity/geoip-enrich` and `crowdsecurity/dateparse-enrich`
89
-
90
-
## Parsers
91
-
92
-
[[References](/parsers/introduction.mdx)]
93
-
94
-
For logs to be able to be exploited and analyzed, they need to be parsed and normalized, and this is where parsers are used.
95
-
96
-
A parser is a YAML configuration file that describes how a string is being parsed. The said string can be a log line, or a field extracted from a previous parser. While a lot of parsers rely on the **GROK** approach (a.k.a regular expression named capture groups), parsers can also reference enrichment modules to allow specific data processing.
97
-
98
-
A parser usually has a specific scope. For example, if you are using [Nginx](https://nginx.org), you will probably want to use the `crowdsecurity/nginx-logs` parser which allows your CrowdSec setup to parse Nginx's access and error logs.
99
-
100
-
Parsers are organized into stages to allow pipelines and branching in parsing.
101
-
102
-
See the [Hub](https://hub.crowdsec.net/browse/#configurations) to explore parsers, or see below some examples:
You can as well [write your own](/parsers/create.md)!
109
-
110
-
111
-
## Enrichers
112
-
113
-
[[References](/parsers/enricher.md)]
114
-
115
-
Enrichment is a parser that adds extra context to a log event so that CrowdSec can later take a better decision. In most cases, you should be able to find the relevant enrichers on our [Hub](https://hub.crowdsec.net/browse/#configurations).
116
-
117
-
A common/simple type of enrichment would be [geoip-enrich](https://github.com/crowdsecurity/hub/blob/master/parsers/s02-enrich/crowdsecurity/geoip-enrich.yaml) of an event (adding information such as origin country, origin Autonomous System and origin IP range to an event).
118
-
119
-
Once again, you should be able to find the ones you're looking for on the [Hub](https://hub.crowdsec.net/browse/#configurations)!
120
-
121
-
## Scenarios
122
-
123
-
[[References](/scenarios/introduction.mdx)]
124
-
125
-
A scenario is the expression of a heuristic that allows you to qualify a specific event (usually an attack). It is a YAML file that describes a set of events characterizing a scenario. Scenarios in CrowdSec gravitate around the [leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket) principle.
126
-
127
-
A scenario description includes at least:
128
-
129
-
- Event eligibility rules. For example, if we're writing an ssh brute-force detection, we only focus on logs of type `ssh_failed_auth`
130
-
- Bucket configuration such as the leak speed or its capacity (in our same ssh brute-force example, we might allow 1 failed auth per 10s and no more than 5 in a short amount of time: `leakspeed: 10s``capacity: 5`)
131
-
- Aggregation rules: per source IP or other criteria (in our ssh brute-force example, we will group per source ip)
132
-
133
-
The description allows for many other rules to be specified (blackhole, distinct filters, etc.), to allow rather complex scenarios.
134
-
135
-
See [Hub](https://hub.crowdsec.net/browse/#configurations) to explore scenarios and their capabilities, or see below some examples:
- [iptables port scan](https://github.com/crowdsecurity/hub/blob/master/scenarios/crowdsecurity/iptables-scan-multi_ports.yaml)
140
-
141
-
You can as well [write your own](/scenarios/create.md)!
142
-
143
-
## AppSec Rules
144
-
145
-
[[References](appsec/intro)]
146
-
147
-
An AppSec rule is a YAML configuration file that describe a rule for the [Application Security Component](appsec/intro).
148
-
149
-
An AppSec rule can match on various aspect of an HTTP request, such as the verb, the URI, the headers, the parameters, the body, etc.
150
-
You can use them to detect and block exploitation of specific vulnerabilities (virtual patching) or to write more generic WAF rules.
151
-
152
-
Contrary to scenarios, appsec rules can block a request before it reaches the application if configured in `in-band` mode.
153
-
154
-
## AppSec Configs
155
-
156
-
An Appsec Config file is a YAML configuration file that tell the [Application Security Component](appsec/intro) which rules should be loaded, and in which band (in-band or out-of-band).
157
-
158
-
The config also expose hooks allowing you to customize at runtime the behavior of the AppSec component (disable a rule on a specific URL, apply a captcha if a specific rule is triggered, etc.)
159
-
160
-
161
-
[[References](appsec/intro)]
162
-
163
-
## Collections
164
-
165
-
[[References](/collections/introduction.md)]
166
-
167
-
To make users' lives easier, "collections" are available, which are just a bundle of parsers and scenarios.
168
-
In this way, if you want to cover basic use-cases of let's say "Nginx", you can just install the `crowdsecurity/nginx` collection that is composed of `crowdsecurity/nginx-logs` parser, as well as generic HTTP scenarios such as `crowdsecurity/base-http-scenarios`.
169
-
170
-
As usual, these can be found on the [Hub](https://hub.crowdsec.net)!
171
-
172
-
## PostOverflows
173
-
174
-
[[References](/parsers/introduction.mdx)]
175
-
176
-
A postoverflow is a parser that is applied on overflows (scenario results) before the decision is written to the local DB or pushed to the API. Parsers in postoverflows are meant to be used for "expensive" enrichment/parsing processes that you do not want to perform on all incoming events, but rather on a decision that is about to be taken.
177
-
178
-
An example could be the slack/mattermost enrichment plugin that requires human confirmation before applying the decision or reverse-dns lookup operations.
179
-
180
-
181
-
# Runtime items
182
-
183
-
## Events
184
-
185
-
[[References](/expr/event.md)]
186
-
187
-
An `Event` is the runtime representation of an item being processed by CrowdSec: it can be a log line being parsed, or an Overflow being reprocessed.
188
-
189
-
The `Event` object is modified by parsers, scenarios, and directly via user [statics expressions](/parsers/format.md#statics) (for example).
190
-
191
-
192
-
<!--TBD: check ref-->
193
-
194
-
## Alerts
195
-
196
-
[[References](/expr/alert.md)]
197
-
198
-
An `Alert` is the runtime representation of a bucket overflow being processed by CrowdSec: it is embedded in an Event.
199
-
200
-
The `Alert` object is modified by post-overflows and [profiles](/profiles/intro.md).
201
-
202
-
## Decisions
203
-
204
-
[[References](/expr/decision.md)]
205
-
206
-
A `Decision` is the representation of the consequence of a bucket overflow: a decision against an IP address, an IP range, an AS, a Country, a User, a Session etc.
207
-
208
-
`Decisions`are generated by the Local API (LAPI) when an `Alert` is received, according to the existing [profiles](/profiles/intro.md)
60
+
The [Console](https://app.crowdsec.net) allows you to:
61
+
-[Manage alerts](/u/console/alerts/intro) of your security stack
62
+
-[Manage decisions](/u/console/decisions/decisions_intro) in real-time
63
+
- View and use [blocklists and integrations](/u/blocklists/intro)
64
+
- Manage your API keys ([CTI API](/u/cti_api/intro), [Service API](/u/service_api/getting_started))
Copy file name to clipboardExpand all lines: crowdsec-docs/docs/getting_started/intro.md
+6-2Lines changed: 6 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,6 +18,10 @@ See [Version Matrix](/getting_started/versions_matrix.md) for a list of supporte
18
18
19
19
## Why is my Security Engine classed as a "log processor" within the console?
20
20
21
-
The `Security Engine` comes compiled with a number of optional features that can be enabled or disabled at runtime. One of these features is called the "LAPI" (Local API). If this feature is disabled at runtime, the Security Engine will be classed as a "log processor" within the console as it will only be able to process logs and forward the alerts to the local API you define in the configuration.
21
+
The `Security Engine` comes compiled with a number of core components that can be enabled or disabled at runtime.
22
+
One of these features is called the **"LAPI"** (Local API) and other one is the **"log processor"**.
22
23
23
-
Most commonly this is the case when you are running in a distributed setup, where you have a central server that is running the LAPI and a number of remote servers that are running the "Log processors".
24
+
A single `Security Engine` can run in autonomous mode where it handles reading logs (or requests) and processing them as well as handling alerts and other task on its own **"LAPI"**.
25
+
For **scaling** or **perimeter segmentation** reasons you might want to have a distributed setup. In that case multiple `Security Engines` process logs and send alerts to a central one's **"LAPI"**. In that distributed case you'll see **"log processors"** attached to the central `Security Engines` in the console UI.
26
+
27
+
Read more about the [Log Processor](log_processor/intro.mdx) and the [Local API](local_api/intro.md).
Copy file name to clipboardExpand all lines: crowdsec-docs/docs/intro.mdx
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -49,9 +49,9 @@ In addition to the core "detect and react" mechanism, CrowdSec is committed to s
49
49
50
50
Under the hood, the Security Engine has various components:
51
51
52
-
- The Log Processor is in charge of detection: it analyzes logs from [various data sources](data_sources/intro) or [HTTP requests](appsec/intro) from web servers.
52
+
- The [Log Processor](log_processor/intro.mdx) is in charge of detection: it analyzes logs from [various data sources](data_sources/intro) or [HTTP requests](appsec/intro) from web servers.
53
53
- The [Appsec](appsec/intro) feature is part of the Log Processor and filters HTTP Requests from the compatible web servers.
54
-
- The [Local API](/local_api/intro.md) acts as a middle man:
54
+
- The [Local API](local_api/intro.md) acts as a middle man:
55
55
- Between the [Log Processors](/docs/data_sources/intro) and the [Remediation Components](/u/bouncers/intro) which are in charge of enforcing decisions.
56
56
- And with the [Central API](/central_api/intro.md) to share alerts and receive blocklists.
57
57
- The [Remediation Components](/u/bouncers/intro) - also known as bouncers - block malicious IPs at your chosen level—whether via IpTables, firewalls, web servers, or reverse proxies. [See the full list on our CrowdSec Hub.](https://app.crowdsec.net/hub/remediation-components)
0 commit comments