Skip to content

Commit c3dfda2

Browse files
mmetcmmetc
committed
wip
1 parent 0e24c8f commit c3dfda2

File tree

3 files changed

+245
-0
lines changed

3 files changed

+245
-0
lines changed

‎crowdsec-docs/docs/log_processor/intro.mdx‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,3 +87,9 @@ You can see more information on Whitelists in the [documentation](log_processor/
8787
Alert Context is additional context that can sent with an alert to the LAPI. This context can be shown locally via `cscli` or within the [CrowdSec Console](https://app.crowdsec.net/signup) if you opt in to share context when you enroll your instance.
8888

8989
You can read more about Alert Context in the [documentation](log_processor/alert_context/intro.md).
90+
91+
### Service Discovery & Setup
92+
93+
On installation, CrowdSec can automatically detect existing services, download the relevant Hub collections, and generate acquisitions based on discovered log files.
94+
95+
You can [customize or override these steps](log_processor/service-discovery-setup/intro.md), for example when provisioning multiple systems or using configuration management tools.

‎crowdsec-docs/docs/log_processor/service-discovery-setup/detect-yaml.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
id: detect-yaml
3+
title: detect.yaml file format
4+
sidebar_position: 1
5+
---
6+
7+
# File layout: `detect.yaml`
8+
A minimal detection file is a YAML map with a top‐level `detect:` key. Under it, each entry describes one service plan:
9+
10+
```yaml
11+
# detect.yaml
12+
---
13+
detect:
14+
apache2-file-apache2:
15+
when:
16+
- Systemd.UnitInstalled("apache2.service") or len(Path.Glob("/var/log/apache2/*.log")) > 0
17+
hub_spec:
18+
collections:
19+
- crowdsecurity/apache2
20+
acquisition_spec:
21+
filename: apache2.yaml
22+
datasource:
23+
source: file
24+
filenames:
25+
- /var/log/apache2/*.log
26+
labels:
27+
type: apache2
28+
```
29+
30+
Fields
31+
32+
- `when`: a list of boolean expressions evaluated on the host. Examples include:
33+
- `Systemd.UnitInstalled("<unit>")`, `Windows.ServiceEnabled("<name>")`
34+
- `Host.OS == "linux"`, `Host.OS == "windows"`
35+
- `Path.Exists("/path/file")`, `len(Path.Glob("/path/*.log")) > 0`
36+
- `System.ProcessRunning("<binary>")`
37+
- `hub_spec`: which Hub items to install (collections/parsers/scenarios, etc.). Unknown item types are preserved and passed through.
38+
- `acquisition_spec`: how to generate a per‐service acquisition file:
39+
- `filename`: base name (no slashes). The actual path will be `acquis.d/setup.<filename>.yaml`.
40+
- `datasource`: a map validated against the selected `source` (e.g., `file`, `journalctl`, `docker`, `wineventlog`, `cloudwatch`, `kinesis`, …). Required fields vary per source; the CLI validates them for you.
41+
42+
Examples
43+
44+
Basic OS / Hub only:
45+
46+
```yaml
47+
detect:
48+
linux:
49+
when:
50+
- Host.OS == "linux"
51+
hub_spec:
52+
collections: [crowdsecurity/linux]
53+
```
54+
55+
`journalctl` source with a filter:
56+
57+
```yaml
58+
detect:
59+
caddy-journal:
60+
when:
61+
- Systemd.UnitInstalled("caddy.service")
62+
- len(Path.Glob("/var/log/caddy/*.log")) == 0
63+
hub_spec:
64+
collections: [crowdsecurity/caddy]
65+
acquisition_spec:
66+
filename: caddy.yaml
67+
datasource:
68+
source: journalctl
69+
labels: {type: caddy}
70+
journalctl_filter:
71+
- "_SYSTEMD_UNIT=caddy.service"
72+
```
73+
74+
Windows event log:
75+
76+
```yaml
77+
detect:
78+
windows_auth:
79+
when: [ Host.OS == "windows" ]
80+
hub_spec:
81+
collections: [crowdsecurity/windows]
82+
acquisition_spec:
83+
filename: windows_auth.yaml
84+
datasource:
85+
source: wineventlog
86+
event_channel: Security
87+
event_ids: [4625, 4623]
88+
event_level: information
89+
labels: {type: eventlog}
90+
```
91+

‎crowdsec-docs/docs/log_processor/service-discovery-setup/intro.md‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
---
2+
id: intro
3+
title: Service Discovery & Setup
4+
sidebar_position: 1
5+
---
6+
7+
8+
> Implementation notes & validation
9+
>
10+
> - The CLI can list supported services from your file (`--list-supported-services`). It also validates each datasource by type and errors out on unknown/misplaced fields (e.g., `source` missing; wrong keys for `journalctl`/`docker`; filename with slashes).
11+
>
12+
13+
---
14+
15+
# Use your custom `detect.yaml`
16+
You can provide your file in two ways:
17+
18+
```bash
19+
# Path flag (recommended)
20+
cscli setup detect --detect-config /path/to/detect.yaml
21+
22+
# Or via environment variable
23+
CROWDSEC_SETUP_DETECT_CONFIG=/path/to/detect.yaml cscli setup detect
24+
```
25+
26+
Helpful flags:
27+
- `--yaml` – render the setup plan as YAML (easy to review/edit); default output is JSON.
28+
- `--force <svc>` – pretend detection matched for `<svc>` (repeatable).
29+
- `--ignore <svc>` – drop `<svc>` from the plan even if matched (repeatable).
30+
- `--skip-systemd` – disable systemd‐based detection (useful in containers/chroots).
31+
- `--list-supported-services` – print the service keys present in your file and exit.
32+
33+
**End‑to‑end flow (typical)**
34+
```bash
35+
# 1) build a plan from your rules
36+
cscli setup detect --detect-config ./detect.yaml --yaml > setup.yaml
37+
38+
# 2) validate that plan (optional but recommended)
39+
cscli setup validate ./setup.yaml
40+
41+
# 3) install Hub items + write acquis files
42+
cscli setup install-hub ./setup.yaml
43+
cscli setup install-acquisition ./setup.yaml --acquis-dir /etc/crowdsec/acquis.d
44+
```
45+
46+
# Examples: override defaults (nginx path, etc.)
47+
If your logs live in non‑standard locations, just encode that in `acquisition_spec`.
48+
49+
```yaml
50+
# detect.yaml
51+
---
52+
detect:
53+
nginx-custom:
54+
when:
55+
- Systemd.UnitInstalled("nginx.service") or len(Path.Glob("/srv/logs/nginx/*.log")) > 0
56+
hub_spec:
57+
collections: [crowdsecurity/nginx]
58+
acquisition_spec:
59+
filename: nginx.yaml
60+
datasource:
61+
source: file
62+
filenames:
63+
- /srv/logs/nginx/*.log # <- your path here
64+
labels: {type: nginx}
65+
```
66+
67+
You can also define detection purely by process name when systemd isn’t a good signal:
68+
```yaml
69+
app-by-process:
70+
when: [ System.ProcessRunning("myappd") ]
71+
acquisition_spec:
72+
filename: myappd.yaml
73+
datasource:
74+
source: file
75+
filenames: [ /var/log/myappd/*.log ]
76+
labels: {type: myappd}
77+
```
78+
79+
---
80+
81+
# Generated acquisition files & coexistence with your own files
82+
When you install acquisition from a setup plan, the CLI writes one file per service as `setup.<name>.yaml` in the acquisition directory (typically `/etc/crowdsec/acquis.d`). The content is **prefixed with a header** that includes a truncated `cscli-checksum` and a comment stating it was generated by `cscli setup`.
83+
84+
- Files carrying a valid `cscli-checksum` are considered **generated** and may be overwritten by future runs.
85+
- Files **without** a valid checksum are treated as **manually edited**; in interactive flows, the CLI shows a colorized diff and asks before overwriting. In unattended flows, the command refuses to proceed if manual files are detected.
86+
- Either way, the safest practice is: **don’t edit generated files**. If you need changes, delete the generated `setup.<name>.yaml` and create your own hand‑managed file instead.
87+
88+
> Tips
89+
> - The actual on‑disk path is computed as `acquis.d/setup.<filename>` where `<filename>` comes from `acquisition_spec.filename`.
90+
> - Use `--acquis-dir` to target a different directory.
91+
> - `--dry-run` prints what would be created without writing files.
92+
93+
---
94+
95+
# Unattended installs with a custom detect file
96+
Package installers often call:
97+
98+
```bash
99+
cscli setup unattended
100+
```
101+
102+
This mode:
103+
- uses the same `--detect-config` and `--acquis-dir` flags;
104+
- never prompts for confirmation;
105+
- will skip itself entirely if `CROWDSEC_SETUP_UNATTENDED_DISABLE` is non‑empty (handy for Ansible/automation);
106+
- installs Hub items and writes `setup.*.yaml` files if and only if there are no conflicting manual acquisitions.
107+
108+
---
109+
110+
# Validation & troubleshooting
111+
- **Validate your setup plan** before writing files:
112+
```bash
113+
cscli setup validate ./setup.yaml
114+
```
115+
- Common validation errors (examples):
116+
- missing `datasource.source`
117+
- wrong keys for a source type (e.g., `filename` under `journalctl`)
118+
- missing mandatory fields (e.g., `journalctl_filter` for `journalctl`, `containers/services` for `docker`)
119+
- `acquisition_spec.filename` contains slashes/backslashes
120+
121+
---
122+
123+
# Reference snippets
124+
- Linux collection detection:
125+
```yaml
126+
detect:
127+
linux:
128+
when: [ Host.OS == "linux" ]
129+
hub_spec:
130+
collections: [crowdsecurity/linux]
131+
```
132+
- MariaDB/MySQL file detection with distro fallbacks:
133+
```yaml
134+
detect:
135+
mariadb:
136+
when:
137+
- Systemd.UnitInstalled("mariadb.service") or Path.Exists("/var/log/mariadb/mariadb.log")
138+
hub_spec: { collections: [crowdsecurity/mariadb] }
139+
acquisition_spec:
140+
filename: mariadb.yaml
141+
datasource:
142+
source: file
143+
labels: {type: mysql}
144+
filenames:
145+
- /var/log/mysql/error.log
146+
- /var/log/mariadb/mariadb.log
147+
```
148+

0 commit comments

Comments
 (0)