You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: crowdsec-docs/docs/log_processor/data_sources/introduction.md
+57-8Lines changed: 57 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,55 @@ sidebar_position: 1
6
6
7
7
## Datasources
8
8
9
-
To be able to monitor applications, the Security Engine needs to access logs.
10
-
DataSources are configured via the [acquisition](/configuration/crowdsec_configuration.md#acquisition_path) configuration, or specified via the command-line when performing cold logs analysis.
9
+
To monitor applications, the Security Engine needs to read logs.
10
+
DataSources define where to access them (either as files, or over the network from a centralized logging service).
11
11
12
+
They can be defined:
13
+
14
+
- in [Acquisition files](/configuration/crowdsec_configuration.md#acquisition_path). Each file contains one or more DataSource definitions, separated by a line with three dashes (`---`).
15
+
- for cold log analysis, you can also specify acquisitions via the command line.
16
+
17
+
18
+
### Service detection (automated setup)
19
+
20
+
When CrowdSec is installed via a package manager on a fresh system, the package may run `cscli setup` in **unattended** mode.
21
+
22
+
The `cscli setup` command will:
23
+
24
+
- detect installed services and common log file locations
25
+
- install the related Hub collections
26
+
- generate acquisition files under `acquis.d/` as `setup.<service>.yaml` (e.g., `setup.linux.yaml`)
27
+
28
+
Generated files are meant to be managed by CrowdSec; don’t edit them in place. If you need changes, delete the generated file and create your own.
29
+
30
+
When upgrading or reinstalling CrowdSec, it detects non-generated or modified files and won’t overwrite your custom acquisitions.
31
+
32
+
:::caution
33
+
34
+
Make sure the same data sources are not ingested more than once: duplicating inputs can artificially increase scenario sensitivity.
35
+
36
+
:::
37
+
38
+
Examples:
39
+
40
+
- If an application logs to both `journald` and `/var/log/*`, you usually only need one of them.
41
+
- If an application writes to `/var/log/syslog` or `/var/log/messages`, it’s already acquired by `setup.linux.yaml` (since 1.7) or `acquis.yam`. You don’t need to add a separate acquisition for the same logs.
42
+
43
+
For config-managed deployments (e.g., Ansible), set the environment variable `CROWDSEC_SETUP_UNATTENDED_DISABLE` to any non-empty value to skip the automated setup.
44
+
In that case, ensure you configure at least one data source and install the OS collection (e.g., crowdsecurity/linux).
45
+
46
+
For more information on the automated configuration, see the command `cscli setup`.
47
+
48
+
### Assisted service detection (semi-automated setup)
49
+
50
+
If you installed new applications and want to detect the service detection again, running `cscli setup` yourself will guide you through the
51
+
automated setup, with confirmation prompts. You will receive a warning if you already configured some acquisition yourself but they won't be
52
+
modified by `cscli`.
53
+
54
+
Note that `cscli setup` will not remove any collection or acquisition file in `acquis.d/setup.<service>.yaml`, even if the service has been uninstalled since the file creation.
55
+
56
+
57
+
## Datasources modules
12
58
13
59
Name | Type | Stream | One-shot
14
60
-----|------|--------|----------
@@ -70,31 +116,34 @@ If not set, then crowdsec will think all logs happened at once, which can lead t
70
116
A map of labels to add to the event.
71
117
The `type` label is mandatory, and used by the Security Engine to choose which parser to use.
Copy file name to clipboardExpand all lines: crowdsec-docs/docs/log_processor/intro.mdx
-24Lines changed: 0 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -50,32 +50,8 @@ labels:
50
50
type: syslog
51
51
```
52
52
53
-
When CrowdSec is installed via a package manager on a fresh system, the package manager may run `cscli setup` in **unattended** mode.
54
-
It detects installed services and common log file locations, installs the related Hub collections, and generates acquisition files under `acquis.d/setup.<service>.yaml`, e.g. `setup.linux.yaml`).
55
-
56
-
Generated files are meant to be managed by crowdsec; don’t edit them in place. If you need changes, delete the generated file and create your own.
57
-
58
-
When upgrading or reinstalling crowdsec, it detects non-generated or modified files and won’t overwrite your custom acquisitions.
59
-
60
-
:::caution
61
-
62
-
Make sure the same data sources aren’t ingested more than once: duplicating inputs can artificially increase scenario sensitivity.
63
-
64
-
:::
65
-
66
-
Examples:
67
-
68
-
- If an application logs to both `journald` and `/var/log/*`, you usually only need one of them.
69
-
70
-
- If an application writes to `/var/log/syslog` or `/var/log/messages`, it’s already acquired by `setup.linux.yaml` (since 1.7) or `acquis.yam`. You don’t need to add a separate acquisition for the same logs.
71
-
72
-
For config-managed deployments (e.g., Ansible), set the environment variable `CROWDSEC_SETUP_UNATTENDED_DISABLE` to any non-empty value to skip the automated setup.
73
-
In that case, ensure you configure at least one data source and install the OS collection (e.g., crowdsecurity/linux).
74
-
75
53
For more information on Data Sources and Acquisitions, see the [Data Sources](log_processor/data_sources/introduction.md) documentation.
76
54
77
-
For more information on the automated configuration, see the command `cscli setup`.
78
-
79
55
## Collections
80
56
81
57
Collections are used to group together Parsers, Scenarios, and Enrichers that are related to a specific application. For example the `crowdsecurity/nginx` collection contains all the Parsers, Scenarios, and Enrichers that are needed to parse logs from an NGINX web server and detect patterns of interest.
Copy file name to clipboardExpand all lines: crowdsec-docs/unversioned/getting_started/post_installation/acquisition.mdx
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,13 +5,14 @@ title: Acquisition
5
5
6
6
# Acquisition
7
7
8
-
By default when CrowdSec is installed it will attempt to detect the running services and acquire the appropriate log sources and [Collections](https://docs.crowdsec.net/docs/next/collections/intro).
8
+
By default when CrowdSec is installed it will attempt to [detect the running services](/log_processor/data_sources#service-detection) and acquire the appropriate log sources and [Collections](https://docs.crowdsec.net/docs/next/collections/intro).
9
9
10
-
However, we should check that this detection worked or you may want to manually acquire additional [Collections](https://docs.crowdsec.net/docs/next/collections/intro) for other services that are not detected.
10
+
However, we should check that this detection worked and the log locations are correct.
11
+
You may want to manually acquire additional [Collections](https://docs.crowdsec.net/docs/next/collections/intro) for the services that were not detected.
11
12
12
13
## What log sources are already detected?
13
14
14
-
To find what log sources are already detected, you can use the `cscli` command line tool.
15
+
To find out which log sources are providing data to crowdsec, you can query the CrowdSec metrics with the `cscli` command line tool.
0 commit comments