Skip to content

Commit ef02da1

Browse files
AndrewChubatiuktruepele
authored andcommitted
deployment: restructure victorialogs examples (VictoriaMetrics#6971)
### Describe Your Changes - Use common compose.yaml file for all victorialogs setups to set version in a single place and override it on demand for each agent and protocol - Replaced multiple victorialogs instances in HA setup with single setup with `deploy.replica` parameter set - Added fluentd setup ### Checklist The following checks are **mandatory**: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).
1 parent fd7ef76 commit ef02da1

File tree

95 files changed

+696
-1273
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

95 files changed

+696
-1273
lines changed
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
services:
2+
victorialogs-2:
3+
deploy:
4+
replicas: 1
Lines changed: 27 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,55 +1,54 @@
11
services:
2-
telegraf:
3-
image: bitnami/telegraf:1.31.0
4-
restart: on-failure
5-
volumes:
6-
- type: bind
7-
source: /var/run/docker.sock
8-
target: /var/run/docker.sock
9-
- type: bind
10-
source: /var/lib/docker
11-
target: /var/lib/docker
12-
- ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
13-
command: --config /etc/telegraf/telegraf.conf
14-
depends_on:
15-
victorialogs:
16-
condition: service_healthy
17-
victoriametrics:
18-
condition: service_healthy
19-
20-
victorialogs:
2+
# meta service will be ignored by compose
3+
.victorialogs:
214
image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
22-
volumes:
23-
- victorialogs-vector-docker-vl:/vlogs
24-
ports:
25-
- '9428:9428'
265
command:
276
- -storageDataPath=/vlogs
287
- -loggerFormat=json
298
- -syslog.listenAddr.tcp=0.0.0.0:8094
9+
deploy:
10+
replicas: 0
3011
healthcheck:
3112
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9428/health"]
3213
interval: 1s
3314
timeout: 1s
3415
retries: 10
3516

17+
victorialogs:
18+
extends: .victorialogs
19+
ports:
20+
- '9428:9428'
21+
volumes:
22+
- victorialogs:/vlogs
23+
deploy:
24+
replicas: 1
25+
26+
# second replica is needed for HA setup and its replica count is set to 1 in compose-ha.yaml file
27+
victorialogs-2:
28+
extends: .victorialogs
29+
ports:
30+
- '9429:9428'
31+
volumes:
32+
- victorialogs-2:/vlogs
33+
deploy:
34+
replicas: 0
35+
3636
victoriametrics:
3737
image: victoriametrics/victoria-metrics:latest
3838
ports:
3939
- '8428:8428'
4040
command:
4141
- -storageDataPath=/vmsingle
42-
- -promscrape.config=/promscrape.yml
4342
- -loggerFormat=json
4443
volumes:
45-
- victorialogs-vector-docker-vm:/vmsingle
46-
- ./scrape.yml:/promscrape.yml
44+
- victoriametrics:/vmsingle
4745
healthcheck:
4846
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8428/health"]
4947
interval: 1s
5048
timeout: 1s
5149
retries: 10
5250

5351
volumes:
54-
victorialogs-vector-docker-vl:
55-
victorialogs-vector-docker-vm:
52+
victorialogs:
53+
victorialogs-2:
54+
victoriametrics:

deployment/docker/victorialogs/filebeat-syslog/README.md

Lines changed: 0 additions & 45 deletions
This file was deleted.

deployment/docker/victorialogs/filebeat-syslog/compose.yml

Lines changed: 0 additions & 23 deletions
This file was deleted.
Lines changed: 8 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
1-
# Docker compose Filebeat integration with VictoriaLogs for docker
1+
# Docker compose Filebeat integration with VictoriaLogs using listed below protocols:
2+
3+
* [syslog](./syslog)
4+
* [elasticsearch](./elasticsearch)
25

36
The folder contains the example of integration of [filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) with Victorialogs
47

5-
To spin-up environment run the following command:
8+
To spin-up environment `cd` to any of listed above directories run the following command:
69
```
710
docker compose up -d
811
```
@@ -16,39 +19,16 @@ docker compose rm -f
1619
The docker compose file contains the following components:
1720

1821
* filebeat - fileabeat is configured to collect logs from the `docker`, you can find configuration in the `filebeat.yml`. It writes data in VictoriaLogs
19-
* filebeat-exporter - it export metrics about the filebeat
2022
* VictoriaLogs - the log database, it accepts the data from `filebeat` by elastic protocol
2123
* VictoriaMetrics - collects metrics from `filebeat` via `filebeat-exporter`, `VictoriaLogs` and `VictoriaMetrics`
22-
* grafana - it comes with two predefined dashboards for `VictoriaLogs` and `VictoriaMetrics`
2324

2425
Querying the data
2526

2627
* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:9428/select/vmui`
2728
* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)
2829

29-
the example of filebeat configuration(`filebeat.yml`)
30-
31-
```yaml
32-
filebeat.autodiscover:
33-
providers:
34-
- type: docker
35-
hints.enabled: true
36-
37-
processors:
38-
- add_docker_metadata: ~
39-
40-
output.elasticsearch:
41-
hosts: [ "http://victorialogs:9428/insert/elasticsearch/" ]
42-
worker: 5
43-
parameters:
44-
_msg_field: "message"
45-
_time_field: "@timestamp"
46-
_stream_fields: "container.name"
47-
48-
http:
49-
enabled: true
50-
host: 0.0.0.0
51-
port: 5066
52-
```
30+
Filebeat configuration example can be found below:
31+
- [syslog](./syslog/filebeat.yml)
32+
- [elasticsearch](./elasticsearch/filebeat.yml)
5333

5434
Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.
Lines changed: 5 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
1+
include:
2+
- ../compose.yml
13
services:
24
filebeat-victorialogs:
3-
image: docker.elastic.co/beats/filebeat:8.8.1
5+
image: docker.elastic.co/beats/filebeat:8.15.0
46
restart: on-failure
57
volumes:
68
- type: bind
@@ -9,50 +11,8 @@ services:
911
- type: bind
1012
source: /var/lib/docker/containers/
1113
target: /var/lib/docker/containers/
12-
- ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
14+
- ${PWD}/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
1315
user: root
1416
command:
1517
- "--strict.perms=false"
16-
depends_on: [ victorialogs ]
17-
beat-exporter-victorialogs:
18-
image: trustpilot/beat-exporter:0.4.0
19-
command:
20-
- -beat.uri=http://filebeat-victorialogs:5066
21-
22-
victorialogs:
23-
image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
24-
volumes:
25-
- victorialogs-filebeat-docker-vl:/vlogs
26-
ports:
27-
- '9428:9428'
28-
command:
29-
- -storageDataPath=/vlogs
30-
- -loggerFormat=json
31-
grafana:
32-
container_name: grafana
33-
image: grafana/grafana:9.2.7
34-
depends_on:
35-
- "victoriametrics"
36-
ports:
37-
- 3000:3000
38-
volumes:
39-
- victorialogs-filebeat-docker-grafana:/var/lib/grafana
40-
- ./../../provisioning/:/etc/grafana/provisioning/
41-
- ./../../../../dashboards/victoriametrics.json:/var/lib/grafana/dashboards/vm.json
42-
- ./../../../../dashboards/victorialogs.json:/var/lib/grafana/dashboards/vl.json
43-
victoriametrics:
44-
image: victoriametrics/victoria-metrics:latest
45-
ports:
46-
- '8428:8428'
47-
command:
48-
- -storageDataPath=/vmsingle
49-
- -promscrape.config=/promscrape.yml
50-
- -loggerFormat=json
51-
volumes:
52-
- victorialogs-filebeat-docker-vm:/vmsingle
53-
- ./scrape.yml:/promscrape.yml
54-
55-
volumes:
56-
victorialogs-filebeat-docker-vl:
57-
victorialogs-filebeat-docker-vm:
58-
victorialogs-filebeat-docker-grafana:
18+
depends_on: [victorialogs]
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
include:
2+
- ../compose.yml
3+
name: filebeat-elasticsearch

deployment/docker/victorialogs/filebeat/scrape.yml

Lines changed: 0 additions & 16 deletions
This file was deleted.
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
include:
2+
- ../compose.yml
3+
name: filebeat-syslog

0 commit comments

Comments
 (0)