Skip to content

Commit ecbdcd2

Browse files
committed
moved content
1 parent 8fb9881 commit ecbdcd2

File tree

3 files changed

+61
-35
lines changed

3 files changed

+61
-35
lines changed

crowdsec-docs/docs/log_processor/data_sources/introduction.md

Lines changed: 57 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,55 @@ sidebar_position: 1
66

77
## Datasources
88

9-
To be able to monitor applications, the Security Engine needs to access logs.
10-
DataSources are configured via the [acquisition](/configuration/crowdsec_configuration.md#acquisition_path) configuration, or specified via the command-line when performing cold logs analysis.
9+
To monitor applications, the Security Engine needs to read logs.
10+
DataSources define where to access them (either as files, or over the network from a centralized logging service).
1111

12+
They can be defined:
13+
14+
- in [Acquisition files](/configuration/crowdsec_configuration.md#acquisition_path). Each file contains one or more DataSource definitions, separated by a line with three dashes (`---`).
15+
- for cold log analysis, you can also specify acquisitions via the command line.
16+
17+
18+
### Service detection (automated setup)
19+
20+
When CrowdSec is installed via a package manager on a fresh system, the package may run `cscli setup` in **unattended** mode.
21+
22+
The `cscli setup` command will:
23+
24+
- detect installed services and common log file locations
25+
- install the related Hub collections
26+
- generate acquisition files under `acquis.d/` as `setup.<service>.yaml` (e.g., `setup.linux.yaml`)
27+
28+
Generated files are meant to be managed by CrowdSec; don’t edit them in place. If you need changes, delete the generated file and create your own.
29+
30+
When upgrading or reinstalling CrowdSec, it detects non-generated or modified files and won’t overwrite your custom acquisitions.
31+
32+
:::caution
33+
34+
Make sure the same data sources are not ingested more than once: duplicating inputs can artificially increase scenario sensitivity.
35+
36+
:::
37+
38+
Examples:
39+
40+
- If an application logs to both `journald` and `/var/log/*`, you usually only need one of them.
41+
- If an application writes to `/var/log/syslog` or `/var/log/messages`, it’s already acquired by `setup.linux.yaml` (since 1.7) or `acquis.yam`. You don’t need to add a separate acquisition for the same logs.
42+
43+
For config-managed deployments (e.g., Ansible), set the environment variable `CROWDSEC_SETUP_UNATTENDED_DISABLE` to any non-empty value to skip the automated setup.
44+
In that case, ensure you configure at least one data source and install the OS collection (e.g., crowdsecurity/linux).
45+
46+
For more information on the automated configuration, see the command `cscli setup`.
47+
48+
### Assisted service detection (semi-automated setup)
49+
50+
If you installed new applications and want to detect the service detection again, running `cscli setup` yourself will guide you through the
51+
automated setup, with confirmation prompts. You will receive a warning if you already configured some acquisition yourself but they won't be
52+
modified by `cscli`.
53+
54+
Note that `cscli setup` will not remove any collection or acquisition file in `acquis.d/setup.<service>.yaml`, even if the service has been uninstalled since the file creation.
55+
56+
57+
## Datasources modules
1258

1359
Name | Type | Stream | One-shot
1460
-----|------|--------|----------
@@ -70,31 +116,34 @@ If not set, then crowdsec will think all logs happened at once, which can lead t
70116
A map of labels to add to the event.
71117
The `type` label is mandatory, and used by the Security Engine to choose which parser to use.
72118

73-
## Acquisition configuration example
119+
## Acquisition configuration examples
74120

75-
```yaml title="/etc/crowdsec/acquis.yaml"
121+
```yaml title="/etc/crowdsec/acquis.d/nginx.yaml"
76122
filenames:
77123
- /var/log/nginx/*.log
78124
labels:
79125
type: nginx
80-
---
126+
```
127+
128+
```yaml title="/etc/crowdsec/acquis.d/linux.yaml"
81129
filenames:
82130
- /var/log/auth.log
83131
- /var/log/syslog
84132
labels:
85133
type: syslog
86-
---
134+
```
135+
136+
```yaml title="/etc/crowdsec/acquis.d/docker.yaml"
87137
source: docker
88138
container_name_regexp:
89139
- .*caddy*
90140
labels:
91141
type: caddy
92-
---
93-
...
94142
```
95143
96144
:::warning
97145
The `labels` and `type` fields are necessary to dispatch the log lines to the right parser.
98146

147+
You can define multiple datasources in the same yaml file, separated by the line `---`
99148
Also note between each datasource is `---` this is needed to separate multiple YAML documents (each datasource) in a single file.
100149
:::

crowdsec-docs/docs/log_processor/intro.mdx

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -50,32 +50,8 @@ labels:
5050
type: syslog
5151
```
5252
53-
When CrowdSec is installed via a package manager on a fresh system, the package manager may run `cscli setup` in **unattended** mode.
54-
It detects installed services and common log file locations, installs the related Hub collections, and generates acquisition files under `acquis.d/setup.<service>.yaml`, e.g. `setup.linux.yaml`).
55-
56-
Generated files are meant to be managed by crowdsec; don’t edit them in place. If you need changes, delete the generated file and create your own.
57-
58-
When upgrading or reinstalling crowdsec, it detects non-generated or modified files and won’t overwrite your custom acquisitions.
59-
60-
:::caution
61-
62-
Make sure the same data sources aren’t ingested more than once: duplicating inputs can artificially increase scenario sensitivity.
63-
64-
:::
65-
66-
Examples:
67-
68-
- If an application logs to both `journald` and `/var/log/*`, you usually only need one of them.
69-
70-
- If an application writes to `/var/log/syslog` or `/var/log/messages`, it’s already acquired by `setup.linux.yaml` (since 1.7) or `acquis.yam`. You don’t need to add a separate acquisition for the same logs.
71-
72-
For config-managed deployments (e.g., Ansible), set the environment variable `CROWDSEC_SETUP_UNATTENDED_DISABLE` to any non-empty value to skip the automated setup.
73-
In that case, ensure you configure at least one data source and install the OS collection (e.g., crowdsecurity/linux).
74-
7553
For more information on Data Sources and Acquisitions, see the [Data Sources](log_processor/data_sources/introduction.md) documentation.
7654
77-
For more information on the automated configuration, see the command `cscli setup`.
78-
7955
## Collections
8056
8157
Collections are used to group together Parsers, Scenarios, and Enrichers that are related to a specific application. For example the `crowdsecurity/nginx` collection contains all the Parsers, Scenarios, and Enrichers that are needed to parse logs from an NGINX web server and detect patterns of interest.

crowdsec-docs/unversioned/getting_started/post_installation/acquisition.mdx

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,14 @@ title: Acquisition
55

66
# Acquisition
77

8-
By default when CrowdSec is installed it will attempt to detect the running services and acquire the appropriate log sources and [Collections](https://docs.crowdsec.net/docs/next/collections/intro).
8+
By default when CrowdSec is installed it will attempt to [detect the running services](/log_processor/data_sources#service-detection) and acquire the appropriate log sources and [Collections](https://docs.crowdsec.net/docs/next/collections/intro).
99

10-
However, we should check that this detection worked or you may want to manually acquire additional [Collections](https://docs.crowdsec.net/docs/next/collections/intro) for other services that are not detected.
10+
However, we should check that this detection worked and the log locations are correct.
11+
You may want to manually acquire additional [Collections](https://docs.crowdsec.net/docs/next/collections/intro) for the services that were not detected.
1112

1213
## What log sources are already detected?
1314

14-
To find what log sources are already detected, you can use the `cscli` command line tool.
15+
To find out which log sources are providing data to crowdsec, you can query the CrowdSec metrics with the `cscli` command line tool.
1516

1617
```bash
1718
cscli metrics show acquisition

0 commit comments

Comments
 (0)