You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Local API (LAPI) is a core component of CrowdSec to :
9
+
The Local API (LAPI) is one of the core component of the Security Engine to :
10
10
11
-
- Allow CrowdSec machines to push alerts & decisions to a database
12
-
- Allow bouncers to consume said alerts & decisions from database
13
-
- Allow `cscli` to view add or delete decisions
11
+
- Allow Log Processors to push alerts & decisions to a database
12
+
- Allow Remediation Components to consume said alerts & decisions from database
13
+
- Allow `cscli` to manage the database (list, delete, etc)
14
14
15
15
You can find the swagger documentation [here](https://crowdsecurity.github.io/api_doc/lapi/).
16
16
@@ -20,40 +20,54 @@ This allows you to create [multi-machines architectures](https://crowdsec.net/mu
20
20
21
21
There are three kinds of authentication to the Local API :
22
22
23
-
- Bouncers: they authenticate with a simple API key and can only read decisions
23
+
1. API Keys: they are used to authenticate Remediation Components (bouncers) and can only read decisions
24
+
2. Login/Password: they are used to authenticate Log Processors (machines) and can read, create and delete decisions
25
+
3. TLS client certificates:
26
+
- they are used to authenticate Remediation Components (bouncers) and Log Processors (machines)
27
+
- based on the OU field of the certificate, the Local API will determine what permissions the client has as per the restrictions above (log processor or remediation components)
28
+
- this allows the Local API to authenticate clients without generating the clients before hand if you have a dynamic environment
24
29
25
-
- Machines: they authenticate with a login and password and can not only read decisions, but create new ones too
30
+
For TLS authentication please see our [dedicated documentation](/local_api/tls_auth.md).
26
31
27
-
- TLS client certificates: it allows you to connect new bouncers or machines to the local API without registring them first
32
+
### Remediation Components (Bouncers)
28
33
29
-
30
-
### Bouncers
31
-
32
-
To register a bouncer to your API, you need to run the following command on the server where the API is installed:
34
+
To register a Remediation Component to your API, you need to run the following command on the server where the API is installed:
33
35
34
36
```bash
35
37
sudo cscli bouncers add testBouncer
36
38
```
37
39
38
-
and keep the generated API token to use it in your bouncers configuration file.
39
-
40
-
See [here](/local_api/tls_auth.md) for the documentation about TLS authentication.
40
+
and keep the generated API token to use it in your Remediation Component configuration file.
41
41
42
-
### Machines
42
+
### Log Processors (machines)
43
43
44
-
To allow a machine to communicate with the Local API, the machine needs to be validated by an administrator of the Local API.
44
+
To allow a log processor to communicate with the Local API, each instance will need it own set of credentials which is validated by an admin of the Local API.
45
45
46
46
There are two ways to register a CrowdSec to a Local API.
47
47
48
-
* You can create a machine directly on the API server that will be automatically validated by running the following command on the server where the API is installed:
48
+
1. You can generate credentials directly on the Local API server:
49
49
50
50
```bash
51
51
sudo cscli machines add testMachine
52
52
```
53
53
54
-
If your CrowdSec runs on the same server as the Local API, then your credentials file will be generated automatically, otherwise you will have to copy/paste them in your remote CrowdSec credentials file (`/etc/crowdsec/local_api_credentials.yaml`)
54
+
:::warning
55
+
if you are running this command on the local API server, most likely it will already have it own credentials file. If you are generating credentials for a remote machine you must pass the `-f` flag to generate the credentials to another file.
Upon installation of CrowdSec it will generate it own set of credentials to operate the log processor and local API server.
55
67
56
-
* You can use `cscli` to register to the API server:
68
+
If you are installing these credentials on a remote machine, you must replace the `local_api_credentials.yaml` file within the configuration directory, you can find the location of this directory [here](/u/troubleshooting/security_engine#where-is-configuration-stored) based on your operating system.
69
+
70
+
2. You can use `cscli` to send a registration request to the Local API server:
@@ -96,25 +108,87 @@ sudo cscli machines list # to get the name of the new registered machine
96
108
sudo cscli machines validate <machineName>
97
109
```
98
110
111
+
* Restart the CrowdSec service on the machine you registered once validated:
112
+
113
+
```bash
114
+
sudo systemctl restart crowdsec
115
+
```
116
+
117
+
#### Disable the registered machine Local API
118
+
119
+
On the machine you ran `cscli lapi register`, it optimal to disable the Local API component to save on resources since it is now forwarding all alerts/decisions to the Local API server.
120
+
121
+
Within the `config.yaml` file, set `enable` under `api.server` to `false`:
122
+
123
+
```yaml
124
+
api:
125
+
server:
126
+
enable: false
127
+
```
128
+
129
+
See where the `config.yaml` file is located on your operating system [here](/u/troubleshooting/security_engine#where-is-configuration-stored)
99
130
100
131
### Server
101
132
102
133
#### Configure listen URL
103
134
104
-
If you would like your Local API to be used by a remote CrowdSec installation, you will need to modify the URL it listens on.
105
-
Modify the [`listen_uri` option](/configuration/crowdsec_configuration.md#listen_uri) in the main configuration file.
106
-
Then see [how to configure your crowdsec to use a remote API](/u/user_guides/machines_mgmt).
135
+
If you would like your Local API to be used by a remote CrowdSec installation, you will need to modify the URL it listens on as by default it will listen on the loopback interface.
107
136
137
+
Modify the [`listen_uri`](/configuration/crowdsec_configuration.md#listen_uri) option in the `config.yaml`.
108
138
109
139
#### Enable SSL
110
140
111
-
The most common use case of the Local API is to listen on 127.0.0.1. In that case there's no need for
112
-
configuring any ssl layer. In some cases, the local API will listen for other CrowdSec installations that
113
-
will report their triggered scenarios. In that case the endpoint may be configured with ssl.
114
-
You can see how to configure SSL on your Local API [here](/configuration/crowdsec_configuration.md#tls).
141
+
If your Local API is exposed to the internet, it is recommended to enable SSL or at least use a reverse proxy with SSL termination to secure the communication between the Log Processors / Remediation Components and the Local API.
142
+
143
+
If your Log Processors and Remediation Components are apart of the same LAN or VPN, then this is not necessary step.
144
+
145
+
##### Local API SSL
146
+
147
+
You can configure the Local API to use SSL by setting the `tls` option under `api.server` in the `config.yaml` file.
148
+
149
+
```yaml
150
+
api:
151
+
server:
152
+
tls:
153
+
cert_path: "/path/to/cert.pem"
154
+
key_path: "/path/to/key.pem"
155
+
```
156
+
157
+
:::info
158
+
If you are using a self signed certificate on connecting Log Processors and Remediation Components you must enable `insecure_skip_verify` options.
159
+
:::
160
+
161
+
- Log Processors (machines)
162
+
163
+
```yaml
164
+
api:
165
+
client:
166
+
insecure_skip_verify: true
167
+
```
168
+
169
+
- Remediation Components (bouncers)
170
+
171
+
This can differ based on the configuration please refer to the documentation of the component you are using.
172
+
173
+
If you would like to read the full configuration options for TLS on the Local API please [see here](/configuration/crowdsec_configuration.md#tls).
115
174
116
175
You can also refer [here](/local_api/tls_auth.md) for the documentation about TLS authentication.
117
176
177
+
##### Reverse Proxy
178
+
179
+
We cannot cover all the reverse proxies available, please refer to the documentation of the reverse proxy you are using. However, the reverse proxy must send the connecting IP address as the `X-Forwarded-For` header to the Local API.
180
+
181
+
However, when the Local API is behind a reverse proxy you will need to configure the `trusted_proxies` and `use_forwarded_for_headers` options under `api.server` within the `config.yaml` file to be able to get the correct IP address within the database.
182
+
183
+
```yaml
184
+
api:
185
+
server:
186
+
use_forwarded_for_headers: true
187
+
trusted_proxies:
188
+
- "127.0.0.1" ## Change this to the proxy IP this is presuming the proxy is on the same machine
189
+
```
190
+
191
+
See where the `config.yaml` file is located on your operating system [here](/u/troubleshooting/security_engine#where-is-configuration-stored)
118
192
119
193
See the [Local API public documentation](https://crowdsecurity.github.io/api_doc/lapi/).
The Log Processor is one of the core component of the Security Engine to:
8
+
9
+
- Read logs from [Data Sources](log_processor/data_sources/introduction.md) in the form of Acquistions.
10
+
- Parse the logs and extract relevant information using [Parsers](log_processor/parsers/introduction.mdx).
11
+
- Enrich the parsed information with additional context such as GEOIP, ASN using [Enrichers](log_processor/parsers/enricher.md).
12
+
- Monitor the logs for patterns of interest known as [Scenarios](log_processor/scenarios/introduction.mdx).
13
+
- Push alerts to the Local API (LAPI) for alert/decisions to be stored within the database.
14
+
15
+
!TODO: Add diagram of the log processor pipeline
16
+
- Read logs from datasources
17
+
- Parse the logs
18
+
- Enrich the parsed information
19
+
- Monitor the logs for patterns of interest
20
+
21
+
22
+
## Introduction
23
+
24
+
The Log Processor is an internal core component of the Security Engine in charge of reading logs from Data Sources, parsing them, enriching them, and monitoring them for patterns of interest.
25
+
26
+
Once a pattern of interest is detected, the Log Processor will push alerts to the Local API (LAPI) for alert/decisions to be stored within the database.
27
+
28
+
## Data Sources
29
+
30
+
Data Sources are individual modules that can be loaded at runtime by the Log Processor to read logs from various sources. To define a Data Source, you will need to create an acquisition configuration file.
31
+
32
+
### Acquistions
33
+
34
+
Acquisitions are the configuration files that define how the Log Processor should read logs from a Data Source. Acquisitions are defined in YAML format and are loaded by the Log Processor at runtime.
35
+
36
+
We have two ways to define Acquisitions within the [configuration directory](/u/troubleshooting/security_engine#where-is-configuration-stored) :
37
+
38
+
-`acquis.yaml` file: This used to be only place to define Acquisitions prior to `1.5.0`. This file is still supported for backward compatibility.
39
+
-`acquis.d` folder: This is a directory where you can define multiple Acquisitions in separate files. This is useful when you want to auto generate files using an external application such as ansible.
40
+
41
+
```yaml title="Example Acquisition Configuration"
42
+
## /etc/crowdsec/acquis.d/file.yaml
43
+
source: file ## The Data Source module to use
44
+
filenames:
45
+
- /tmp/foo/*.log
46
+
- /var/log/syslog
47
+
labels:
48
+
type: syslog
49
+
```
50
+
51
+
For more information on Data Sources and Acquisitions, see the [Data Sources](log_processor/data_sources/introduction.md) documentation.
52
+
53
+
## Collections
54
+
55
+
Collections are used to group together Parsers, Scenarios, and Enrichers that are related to a specific application. For example the `crowdsecurity/nginx` collection contains all the Parsers, Scenarios, and Enrichers that are needed to parse logs from an NGINX web server and detect patterns of interest.
56
+
57
+
You can see all available collections on the [Hub](https://app.crowdsec.net/hub/collections).
58
+
59
+
### Parsers
60
+
61
+
The parsing pipeline is broken down into multiple stages:
62
+
63
+
- `s00-raw` : This is the first stage which aims to normalize the logs from various [Data Sources](log_processor/data_sources/introduction.md) into a predictable format for `s01-parse` and `s02-enrich` to work on.
64
+
- `s01-parse` : This is the second stage responsible for extracting relevant information from the normalized logs based on the application type to be used by `s02-enrich` and the [Scenarios](log_processor/scenarios/introduction.mdx).
65
+
- `s02-enrich` : This is the third stage responsible for enriching the extracted information with additional context such as GEOIP, ASN etc.
66
+
67
+
:::info
68
+
We will give a breif overview of each stage, however, for most users this documentation is not required to get started with CrowdSec but can be used to understand the inner workings of the Log Processor.
69
+
:::
70
+
71
+
#### `s00-raw`
72
+
73
+
This stage is responsible for normalizing logs from various [Data Sources](log_processor/data_sources/introduction.md) into a predictable format for `s01-parse` and `s02-enrich` to work on.
74
+
75
+
For example if you have a `syslog` Data Source and a `container` Data Source writing the same application log lines you wouldnt want `s01-parse` to handle this logic twice, since `s00-raw` can normalize the logs into a predictable format.
76
+
77
+
For most instances we have already created these `s00-raw` parsers for you are available to view on the [Hub](https://hub.crowdsec.net/).
78
+
79
+
#### `s01-parse`
80
+
81
+
The stage is responsible for extracting relevant information from the normalized logs based on the application type.
82
+
83
+
The application type is defined in different ways based on the Data Source. Please refer to the [Data Sources](log_processor/data_sources/introduction.md) documentation for more information.
84
+
85
+
We list all available applications we support on the [Hub](https://hub.crowdsec.net/) and within the readme of the collection our users provide an example Acquisition configuration.
86
+
87
+
#### `s02-enrich`
88
+
89
+
The aim of this stage is to enrich the extracted information with additional context such as GEOIP, ASN etc.
90
+
91
+
However, the stage can also be used to perform whitelist checks, however, we have dedicated documentation for this [here](log_processor/whitelist/introduction.md).
92
+
93
+
Currently we have a few enrichers available on the [Hub](https://hub.crowdsec.net/), that are installed by default so you dont need to worry about this stage unless you want to create your own.
94
+
95
+
For more information on Parsers, see the [Parsers](log_processor/parsers/introduction.mdx) documentation.
96
+
97
+
### Scenarios
98
+
99
+
Scenarios are the patterns of interest that the Log Processor is monitoring for. When a pattern of interest is detected, the Log Processor will push alerts to the Local API (LAPI) for alert/decisions to be stored within the database.
100
+
101
+
The patterns can be as simple as tracking the number of failed login attempts or as complex as tracking logging in from multiple countries within a short period of time which can be a indicator of a compromised account or VPN usage.
102
+
103
+
The community provides a number of scenarios on the [Hub](https://hub.crowdsec.net/) that you can install and use. If you would like to create your own, see the [Scenarios](log_processor/scenarios/introduction.mdx) documentation.
0 commit comments