You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: crowdsec-docs/docs/log_processor/service-discovery-setup/intro.md
+48-30Lines changed: 48 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
2
id: intro
3
-
title: Service Discovery & Setup
3
+
title: Service Discovery
4
4
sidebar_position: 1
5
5
---
6
6
@@ -10,40 +10,43 @@ sidebar_position: 1
10
10
11
11
The main way to use the service discovery is with `cscli setup interactive` or `cscli setup unattended`.
12
12
13
-
By default, it will use the detection file provided by crowdsec stored in `/etc/crowdsec/detect.yaml`.
13
+
By default, it will use the detection file provided by crowdsec stored in `/var/lib/crowdsec/data/detect.yaml`.
14
14
15
15
In interactive mode, `cscli` will ask you to choose which service to configure based on those that were detected, and will require confirmation before any operation (installing hub items, generating acquisition config, ...).
16
16
17
-
If an `acquis.yaml` file exists, `cscli` will ask for confirmation before proceeding to avoid reading the same files multiple times.
17
+
It is your responsibility to check the compatibility of the generated acquisitions with the ones you add later or were already on the system.
18
18
19
-
It is your responsability to check the generated configuration to make sure each log file is only read once by crowdsec.
19
+
:::warning
20
20
21
-
As such, you should avoid putting your acquisition configuration in `/etc/crowdsec/acquis.yaml`, but instead create dedicated files in `/etc/crowdsec/acquis.d`.
21
+
While `cscli setup` validates the generated configuration files for syntax errors or invalid configuration, it does *not* check for duplicate acquisition.
22
22
23
-
When ran in unattended mode, `cscli` will automatically any hub item, but will refuse to run if:
24
-
-`acquis.yaml` exists and is not empty
25
-
- An automatically generated acquisition file in `/etc/crowdsec/acquis.d` has been modified
23
+
If using a custom `detect.yaml`, make sure no logs are read multiple times (with the same `type` label), as this could lead to false positives.
26
24
27
-
Linux packages (deb or rpm) will automatically call `cscli setup unattended` during installation.
25
+
:::
28
26
29
-
:::warning
30
27
31
-
While `cscli setup` will check the generated configuration files for syntax errors or invalid configuration, it does *not* check for duplicate acquisition.
28
+
`cscli` will ask for confirmation before proceeding if:
32
29
33
-
If using a custom `detect.yaml`, make sure no files are read multiple times (with the same `type` label), as this could lead to false positives.
30
+
- there is an `acquis.yaml`
31
+
- there is any non-generated file in `acquis.d`
32
+
- you modified the generated files in `acquis.d` (there is a checksum to detect modifications). Proceeding could overwrite them.
34
33
35
-
:::
34
+
Files composed by comments only are ignored.
36
35
37
-
### Generated acquisition files & coexistence with your own files
36
+
Linux packages (deb or rpm) will automatically call `cscli setup unattended` during installation. In the case above, instead of asking for confirmation, unattended mode will just skip the service detection.
38
37
39
-
When you generated the acquisition configuration with `cscli setup`, `cscli` writes one file per service as `setup.<name>.yaml` in the acquisition directory (typically `/etc/crowdsec/acquis.d`). The content is **prefixed with a header** that includes a truncated `cscli-checksum` and a comment stating it was generated by `cscli setup`.
40
38
41
-
- Files carrying a valid `cscli-checksum` are considered **generated** and may be overwritten by future runs.
42
-
- Files **without** a valid checksum are treated as **manually edited**; in interactive flows, `cscli` shows a colorized diff and asks before overwriting. In unattended flows, the command refuses to proceed if manual files are detected.
39
+
### Generated acquisition files & coexistence with your own files
40
+
41
+
When you generated the acquisition configuration with `cscli setup`, `cscli` writes one file per service as `setup.<name>.yaml` in the acquisition directory (typically `/etc/crowdsec/acquis.d`). The content is prefixed with a header that includes a checksum and a comment stating it was generated by `cscli setup`.
42
+
43
+
- Files carrying a valid checksum are considered generated and may be overwritten by future runs.
44
+
- Files without a valid checksum are treated as manually edited; in interactive mode, `cscli` shows a colorized diff and asks before overwriting. In unattended flows, the command refuses to proceed if manual files are detected.
43
45
- Either way, the safest practice is: **don’t edit generated files**. If you need changes, delete the generated `setup.<name>.yaml` and create your own hand‑managed file instead or use a custom `detect.yaml` to generate the proper configuration automatically.
44
46
45
47
> Tips
46
-
> - The actual on‑disk path is computed as `acquis.d/setup.<filename>` where `<filename>` comes from `acquisition_spec.filename`.
48
+
49
+
> - The actual on‑disk path is computed as `acquis.d/setup.<filename>.yaml` where `<filename>` comes from `acquisition_spec.filename`.
47
50
> - Use `--acquis-dir` to target a different directory.
48
51
> -`--dry-run` prints what would be created without writing files.
-`--yaml` – render the setup plan as YAML (easy to review/edit); default output is JSON.
67
71
-`--force <svc>` – pretend detection matched for `<svc>` (repeatable).
68
72
-`--ignore <svc>` – drop `<svc>` from the plan even if matched (repeatable).
69
-
-`--skip-systemd` – disable systemd‐based detection (useful in containers/chroots).
73
+
-`--skip-systemd` – disable systemd‐based detection (default if systemctl can't be run).
70
74
-`--list-supported-services` – print the service keys present in your file and exit.
71
75
72
76
You can see a list of all the available expr helpers in the [dedicated documentation](/log_processor/service-discovery-setup/expr.md).
73
77
74
-
For example, if you have configured nginx to log in a non-standard location, you can use a custom `detect.yaml` to automatically generate the configuration.
78
+
For example, if you have configured nginx to log in a non-standard location, you can use a custom `detect.yaml` to override it.
75
79
76
-
This example will generate an acquisition config for the file datasource with the pattern `/srv/logs/nginx/*.log` if the nginx service is installed OR if any file matches the glob pattern `/srv/logs/nginx/*.log`:
80
+
This example will generate an acquisition with the pattern `/srv/logs/nginx/*.log` if the nginx service is installed OR if any file matches the glob pattern `/srv/logs/nginx/*.log`:
77
81
78
82
```yaml
79
83
# detect.yaml
@@ -90,9 +94,9 @@ detect:
90
94
datasource:
91
95
source: file
92
96
filenames:
93
-
- /srv/logs/nginx/*.log# <- your path here
97
+
- /srv/logs/nginx/*.log
94
98
labels:
95
-
type: nginx
99
+
type: nginx
96
100
```
97
101
98
102
:::warning
@@ -109,28 +113,41 @@ detect:
109
113
hub_spec:
110
114
collections:
111
115
- crowdsecurity/linux
116
+
acquisition_spec:
117
+
filename: linux.yaml
118
+
datasource:
119
+
source: file
120
+
labels:
121
+
type: syslog
122
+
filenames:
123
+
- /var/log/messages
124
+
- /var/log/syslog
125
+
- /var/log/kern.log
112
126
```
127
+
113
128
:::
114
129
115
130
### Unattended installs with a custom detect file
116
131
117
132
Linux packages (deb or rpm) will automatically call `cscli setup unattended` during installation.
118
133
119
-
You can specify a custom detection file to use by setting the `CROWDSEC_SETUP_DETECT_CONFIG` environment variable.
134
+
You can specify a custom detection file to use by setting `CROWDSEC_SETUP_DETECT_CONFIG` before installing the package with `apt` or `dnf`.
120
135
121
-
Alternatively, if you want to skip the automatic detection (because you deploy the configuration with Ansible for example), you can set the env var `CROWDSEC_SETUP_UNATTENDED_DISABLE` to any value.
136
+
Alternatively, if you want to skip the automatic detection completely, you can set the env var `CROWDSEC_SETUP_UNATTENDED_DISABLE` to any value.
122
137
123
138
### End-to-end workflow
124
139
125
140
Behind the scenes, `cscli setup` use multiple steps to configure crowdsec:
126
-
- Generate a setup files that contains the detected services, their associated hub items and acquisition configuration
127
-
- Validate this file
128
-
- Install the hub items
129
-
- Write the acquisition config to disk
141
+
142
+
- Generate a YAML plan that contains the detected services, their associated hub items and acquisition configuration
143
+
- Validate this file
144
+
- Install the hub items
145
+
- Write the acquisition config to disk
130
146
131
147
If you wish, you can manually invoke any of those steps (if you only want to install the hub items for example).
132
148
133
149
`cscli setup detect` can be used to generate the setup file:
0 commit comments