Skip to content

Commit 48d3cbd

Browse files
he2sshe2ss
authored
Merge pull request #647 from crowdsecurity/feat/add-http-datasource
add http datasource
2 parents 8f24d8b + 4b2a104 commit 48d3cbd

File tree

5 files changed

+385
-0
lines changed

5 files changed

+385
-0
lines changed

‎crowdsec-docs/docs/data_sources/http.md‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
---
2+
id: http
3+
title: HTTP
4+
---
5+
6+
This module allows the `Security Engine` to acquire logs from an HTTP endpoint.
7+
8+
## Configuration examples
9+
10+
To receive logs from an HTTP endpoint with basic auth:
11+
```yaml
12+
source: http
13+
listen_addr: 127.0.0.1:8080
14+
path: /test
15+
auth_type: basic_auth
16+
basic_auth:
17+
username: test
18+
password: test
19+
labels:
20+
type: mytype
21+
```
22+
23+
To receive logs from an HTTP endpoint with headers:
24+
```yaml
25+
source: http
26+
listen_addr: 127.0.0.1:8080
27+
path: /test
28+
auth_type: headers
29+
headers:
30+
MyHeader: MyValue
31+
labels:
32+
type: mytype
33+
```
34+
35+
To receive logs from an HTTP endpoint with TLS and headers:
36+
37+
```yaml
38+
source: http
39+
listen_addr: 127.0.0.1:8080
40+
path: /test
41+
auth_type: headers
42+
headers:
43+
MyHeader: MyValue
44+
tls:
45+
server_cert: server.crt
46+
server_key: server.key
47+
labels:
48+
type: mytype
49+
```
50+
51+
To receive logs from an HTTP endpoint with mTLS:
52+
53+
```yaml
54+
source: http
55+
listen_addr: 127.0.0.1:8080
56+
path: /test
57+
auth_type: mtls
58+
tls:
59+
server_cert: server.crt
60+
server_key: server.key
61+
ca_cert: ca.crt
62+
labels:
63+
type: mytype
64+
```
65+
66+
:::info
67+
If most of cases when the logs are sent in JSON format, you can use the [`transform`](https://docs.crowdsec.net/docs/next/data_sources/intro/#transform) expression to parse the logs.
68+
69+
For example, if the logs are sent in the following format:
70+
```json
71+
{
72+
"Records": [
73+
{
74+
"message": "test",
75+
"timestamp": "2021-01-01T00:00:00Z"
76+
}
77+
]
78+
}
79+
```
80+
81+
the `transform` expression can be:
82+
```yaml
83+
transform: map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#))
84+
```
85+
86+
87+
Look at the `configuration parameters` to view all supported options.
88+
89+
## Parameters
90+
91+
92+
### `listen_addr`
93+
94+
The address to listen on (e.g., `127.0.0.1:8088`).
95+
96+
Required.
97+
98+
### `path`
99+
100+
The endpoint path to listen on.
101+
102+
:::info
103+
The request method is always `POST`.
104+
:::
105+
106+
Optional, default is `/`.
107+
108+
### `auth_type`
109+
110+
The authentication type to use.
111+
112+
Can be `basic_auth`, `headers`, or `mtls`.
113+
114+
Required.
115+
116+
### `basic_auth`
117+
118+
The basic auth credentials.
119+
120+
### `basic_auth.username`
121+
122+
The basic auth username.
123+
124+
Optional, to use when `auth_type` is `basic_auth`.
125+
126+
### `basic_auth.password`
127+
128+
The basic auth password.
129+
130+
Optional, to use when `auth_type` is `basic_auth`.
131+
132+
### `headers`
133+
134+
The headers to send.
135+
136+
Optional, to use when `auth_type` is `headers`.
137+
138+
### `tls`
139+
140+
TLS configuration.
141+
142+
### `tls.server_cert`
143+
144+
The server certificate path.
145+
146+
Optional, to use when `auth_type` is `mtls`.
147+
148+
### `tls.server_key`
149+
150+
The server key path.
151+
152+
Optional, to use when `auth_type` is `mtls`.
153+
154+
### `tls.ca_cert`
155+
156+
The CA certificate path.
157+
158+
Optional, to use when `auth_type` is `mtls`.
159+
160+
### `custom_status_code`
161+
162+
The custom status code to return.
163+
164+
Optional.
165+
166+
### `custom_headers`
167+
168+
The custom headers to return.
169+
170+
Optional.
171+
172+
### `max_body_size`
173+
174+
The maximum body size to accept.
175+
176+
Optional.
177+
178+
### `timeout`
179+
180+
The timeout to read the body.
181+
182+
:::info
183+
The timeout is in duration format, e.g., `5s`.
184+
:::
185+
186+
Optional.
187+
188+
## DSN and command-line
189+
190+
This datasource does not support acquisition from the command line.
191+

‎crowdsec-docs/docs/data_sources/introduction.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Name | Type | Stream | One-shot
1818
[AWS S3](/data_sources/s3.md)| read logs from a S3 bucket | yes | yes
1919
[docker](/data_sources/docker.md) | read logs from docker containers | yes | yes
2020
[file](/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes
21+
[HTTP](/data_sources/http.md) | read logs from an HTTP endpoint | yes | no
2122
[journald](/data_sources/journald.md) | journald via filter | yes | yes
2223
[Kafka](/data_sources/kafka.md)| read logs from kafka topic | yes | no
2324
[Kubernetes Audit](/data_sources/kubernetes_audit.md) | expose a webhook to receive audit logs from a Kubernetes cluster | yes | no

‎crowdsec-docs/sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@
131131
"data_sources/s3",
132132
"data_sources/docker",
133133
"data_sources/file",
134+
"data_sources/http",
134135
"data_sources/journald",
135136
"data_sources/kafka",
136137
"data_sources/kubernetes_audit",

‎crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/http.md‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
---
2+
id: http
3+
title: HTTP
4+
---
5+
6+
This module allows the `Security Engine` to acquire logs from an HTTP endpoint.
7+
8+
## Configuration examples
9+
10+
To receive logs from an HTTP endpoint with basic auth:
11+
```yaml
12+
source: http
13+
listen_addr: 127.0.0.1:8080
14+
path: /test
15+
auth_type: basic_auth
16+
basic_auth:
17+
username: test
18+
password: test
19+
labels:
20+
type: mytype
21+
```
22+
23+
To receive logs from an HTTP endpoint with headers:
24+
```yaml
25+
source: http
26+
listen_addr: 127.0.0.1:8080
27+
path: /test
28+
auth_type: headers
29+
headers:
30+
MyHeader: MyValue
31+
labels:
32+
type: mytype
33+
```
34+
35+
To receive logs from an HTTP endpoint with TLS and headers:
36+
37+
```yaml
38+
source: http
39+
listen_addr: 127.0.0.1:8080
40+
path: /test
41+
auth_type: headers
42+
headers:
43+
MyHeader: MyValue
44+
tls:
45+
server_cert: server.crt
46+
server_key: server.key
47+
labels:
48+
type: mytype
49+
```
50+
51+
To receive logs from an HTTP endpoint with mTLS:
52+
53+
```yaml
54+
source: http
55+
listen_addr: 127.0.0.1:8080
56+
path: /test
57+
auth_type: mtls
58+
tls:
59+
server_cert: server.crt
60+
server_key: server.key
61+
ca_cert: ca.crt
62+
labels:
63+
type: mytype
64+
```
65+
66+
:::info
67+
If most of cases when the logs are sent in JSON format, you can use the [`transform`](https://docs.crowdsec.net/docs/next/data_sources/intro/#transform) expression to parse the logs.
68+
69+
For example, if the logs are sent in the following format:
70+
```json
71+
{
72+
"Records": [
73+
{
74+
"message": "test",
75+
"timestamp": "2021-01-01T00:00:00Z"
76+
}
77+
]
78+
}
79+
```
80+
81+
the `transform` expression can be:
82+
```yaml
83+
transform: map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#))
84+
```
85+
86+
87+
Look at the `configuration parameters` to view all supported options.
88+
89+
## Parameters
90+
91+
92+
### `listen_addr`
93+
94+
The address to listen on (e.g., `127.0.0.1:8088`).
95+
96+
Required.
97+
98+
### `path`
99+
100+
The endpoint path to listen on.
101+
102+
:::info
103+
The request method is always `POST`.
104+
:::
105+
106+
Optional, default is `/`.
107+
108+
### `auth_type`
109+
110+
The authentication type to use.
111+
112+
Can be `basic_auth`, `headers`, or `mtls`.
113+
114+
Required.
115+
116+
### `basic_auth`
117+
118+
The basic auth credentials.
119+
120+
### `basic_auth.username`
121+
122+
The basic auth username.
123+
124+
Optional, to use when `auth_type` is `basic_auth`.
125+
126+
### `basic_auth.password`
127+
128+
The basic auth password.
129+
130+
Optional, to use when `auth_type` is `basic_auth`.
131+
132+
### `headers`
133+
134+
The headers to send.
135+
136+
Optional, to use when `auth_type` is `headers`.
137+
138+
### `tls`
139+
140+
TLS configuration.
141+
142+
### `tls.server_cert`
143+
144+
The server certificate path.
145+
146+
Optional, to use when `auth_type` is `mtls`.
147+
148+
### `tls.server_key`
149+
150+
The server key path.
151+
152+
Optional, to use when `auth_type` is `mtls`.
153+
154+
### `tls.ca_cert`
155+
156+
The CA certificate path.
157+
158+
Optional, to use when `auth_type` is `mtls`.
159+
160+
### `custom_status_code`
161+
162+
The custom status code to return.
163+
164+
Optional.
165+
166+
### `custom_headers`
167+
168+
The custom headers to return.
169+
170+
Optional.
171+
172+
### `max_body_size`
173+
174+
The maximum body size to accept.
175+
176+
Optional.
177+
178+
### `timeout`
179+
180+
The timeout to read the body.
181+
182+
:::info
183+
The timeout is in duration format, e.g., `5s`.
184+
:::
185+
186+
Optional.
187+
188+
## DSN and command-line
189+
190+
This datasource does not support acquisition from the command line.
191+

0 commit comments

Comments
 (0)