Skip to content

Commit a8cc860

Browse files
mrnicegyu11YuryHrytsuk
mrnicegyu11
and
YuryHrytsuk
authored
⚗️ Add vector-dev logging aggregator /w loki, graylog via host ports (#1233)
* wip * Add csi-s3 and have portainer use it * Change request @Hrytsuk 1GB max portainer volume size * Arch Linux Certificates Customization * Fix pgsql exporter failure * [Kubernetes] Introduce on-prem persistent Storage (Longhorn) 🎉 (#979) * Introduce longhorn chart * Further longhorn configuration * Longhorn: further settings configuration * Fix longhorn configuration bugs Extra: introduce longhorn pv vales for portainer * Add comment for deletion longhorn * Further longhorn configuration * Add README.md for Longhorn wit FAQ * Update Longhorn readme * Update readme * Futher LH configuration * Update LH's Readme * Update Longhorn Readme * Improve LH's Readme * LH: Reduce reserved default disk space to 5% Since we use a dedicated disk for LH, we can go ahead with 5% * Use values to set Longhorn storage class * Update LH's Readme * LH Readme: add requirements reference * PR Review: bring back portainer s3 pv * LH: decrease portinaer volume size * Experimental: Try to add tracing to simcore-traefik on master * Fixes ITISFoundation/osparc-simcore#7363 * Arch Linux Certificates Customization - 2 * Revert: disable loki & vector-dev, oldschool graylog logging (#1223) * wip * Add csi-s3 and have portainer use it * Change request @Hrytsuk 1GB max portainer volume size * Arch Linux Certificates Customization * Fix pgsql exporter failure * [Kubernetes] Introduce on-prem persistent Storage (Longhorn) 🎉 (#979) * Introduce longhorn chart * Further longhorn configuration * Longhorn: further settings configuration * Fix longhorn configuration bugs Extra: introduce longhorn pv vales for portainer * Add comment for deletion longhorn * Further longhorn configuration * Add README.md for Longhorn wit FAQ * Update Longhorn readme * Update readme * Futher LH configuration * Update LH's Readme * Update Longhorn Readme * Improve LH's Readme * LH: Reduce reserved default disk space to 5% Since we use a dedicated disk for LH, we can go ahead with 5% * Use values to set Longhorn storage class * Update LH's Readme * LH Readme: add requirements reference * PR Review: bring back portainer s3 pv * LH: decrease portinaer volume size * Experimental: Try to add tracing to simcore-traefik on master * Fixes ITISFoundation/osparc-simcore#7363 * Arch Linux Certificates Customization - 2 * Send docker logs directly to graylog * revert arch linux customization --------- Co-authored-by: Dustin Kaiser <[email protected]> Co-authored-by: YH <[email protected]> * Enable Chatbot for S4L products (#1221) * wip * Add csi-s3 and have portainer use it * Change request @Hrytsuk 1GB max portainer volume size * Arch Linux Certificates Customization * Fix pgsql exporter failure * [Kubernetes] Introduce on-prem persistent Storage (Longhorn) 🎉 (#979) * Introduce longhorn chart * Further longhorn configuration * Longhorn: further settings configuration * Fix longhorn configuration bugs Extra: introduce longhorn pv vales for portainer * Add comment for deletion longhorn * Further longhorn configuration * Add README.md for Longhorn wit FAQ * Update Longhorn readme * Update readme * Futher LH configuration * Update LH's Readme * Update Longhorn Readme * Improve LH's Readme * LH: Reduce reserved default disk space to 5% Since we use a dedicated disk for LH, we can go ahead with 5% * Use values to set Longhorn storage class * Update LH's Readme * LH Readme: add requirements reference * PR Review: bring back portainer s3 pv * LH: decrease portinaer volume size * Experimental: Try to add tracing to simcore-traefik on master * Fixes ITISFoundation/osparc-simcore#7363 * Arch Linux Certificates Customization - 2 * Remove frontend vendor chatbot service * wip --------- Co-authored-by: Dustin Kaiser <[email protected]> Co-authored-by: YH <[email protected]> * Kubernetes: fix global network policy (#1227) * Add authentication middleware to cahtbot vendor service * Revert "Kubernetes: fix global network policy (#1227)" This reverts commit 2d3adb1. * Add ACME DNS Resolver for gitlabCD and k8s (#1217) * wip * Add csi-s3 and have portainer use it * Change request @Hrytsuk 1GB max portainer volume size * Arch Linux Certificates Customization * Fix pgsql exporter failure * [Kubernetes] Introduce on-prem persistent Storage (Longhorn) 🎉 (#979) * Introduce longhorn chart * Further longhorn configuration * Longhorn: further settings configuration * Fix longhorn configuration bugs Extra: introduce longhorn pv vales for portainer * Add comment for deletion longhorn * Further longhorn configuration * Add README.md for Longhorn wit FAQ * Update Longhorn readme * Update readme * Futher LH configuration * Update LH's Readme * Update Longhorn Readme * Improve LH's Readme * LH: Reduce reserved default disk space to 5% Since we use a dedicated disk for LH, we can go ahead with 5% * Use values to set Longhorn storage class * Update LH's Readme * LH Readme: add requirements reference * PR Review: bring back portainer s3 pv * LH: decrease portinaer volume size * Experimental: Try to add tracing to simcore-traefik on master * Fixes ITISFoundation/osparc-simcore#7363 * Arch Linux Certificates Customization - 2 * wip * wip * this might work * k8s wip * wip * wip --------- Co-authored-by: Dustin Kaiser <[email protected]> Co-authored-by: YH <[email protected]> * Experimental: vectordev via host ports * revert * fix * fixes --------- Co-authored-by: Dustin Kaiser <[email protected]> Co-authored-by: YH <[email protected]>
1 parent 472cb6c commit a8cc860

File tree

3 files changed

+30
-15
lines changed

3 files changed

+30
-15
lines changed

‎services/logging/docker-compose.yml.j2‎

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ services:
1818
memory: 300M
1919
cpus: "0.1"
2020
networks:
21-
graylog:
21+
logging:
2222
aliases:
2323
- mongo # needed because of graylog configuration
2424

@@ -46,7 +46,7 @@ services:
4646
memory: 1G
4747
cpus: "0.1"
4848
networks:
49-
graylog:
49+
logging:
5050
# Graylog: https://hub.docker.com/r/graylog/graylog/
5151
graylog:
5252
image: graylog/graylog:6.0.5
@@ -71,12 +71,18 @@ services:
7171
networks:
7272
public:
7373
monitoring:
74-
graylog:
74+
logging:
7575
aliases:
7676
- graylog
7777
ports:
78-
- 12201:12201/udp
79-
- 12202:12202/udp
78+
- target: 12201
79+
published: 12203
80+
protocol: udp
81+
mode: host
82+
- target: 12202
83+
published: 12202
84+
protocol: udp
85+
mode: host
8086
deploy:
8187
replicas: 1
8288
restart_policy:
@@ -100,19 +106,23 @@ services:
100106
- traefik.http.middlewares.graylog_replace_regex.replacepathregex.replacement=/$${1}
101107
- traefik.http.routers.graylog.middlewares=ops_whitelist_ips@swarm, ops_gzip@swarm, graylog_replace_regex
102108
vector:
103-
image: timberio/vector:0.49.X-debian
109+
image: timberio/vector:0.50.X-debian
104110
ports:
105-
- "12200:12201/udp" # GELF input
111+
- target: 12201
112+
published: 12201
113+
protocol: udp
114+
mode: host
106115
volumes:
107116
- /var/run/docker.sock:/var/run/docker.sock:ro
108117
environment:
109118
- VECTOR_CONFIG=/etc/vector/vector.yaml
110119
- VECTOR_LOG=info
120+
- VECTOR_LOG_DESTINATION=${VECTOR_LOG_DESTINATION}
111121
configs:
112122
- source: vector_config
113123
target: /etc/vector/vector.yaml
114124
deploy:
115-
replicas: 0
125+
replicas: 1
116126
resources:
117127
limits:
118128
cpus: "1.0"
@@ -121,11 +131,15 @@ services:
121131
memory: 256M
122132
labels: []
123133
networks:
124-
- monitoring
125-
- graylog
134+
logging:
126135

127136
loki:
128-
image: grafana/loki:3.5.4
137+
image: grafana/loki:3.5.7
138+
ports:
139+
- target: 3100
140+
published: 12204
141+
protocol: tcp
142+
mode: host
129143
configs:
130144
- source: loki_config
131145
target: /etc/loki/loki.yaml
@@ -141,7 +155,7 @@ services:
141155
deploy:
142156
placement:
143157
constraints: []
144-
replicas: 0
158+
replicas: 1
145159
restart_policy:
146160
condition: any
147161
delay: 5s
@@ -172,7 +186,7 @@ volumes:
172186
graylog_journal:
173187

174188
networks:
175-
graylog:
189+
logging:
176190
public:
177191
external: true
178192
name: ${PUBLIC_NETWORK}

‎services/logging/template.env‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,4 @@ S3_FORCE_PATH_STYLE_LOKI=${S3_FORCE_PATH_STYLE_LOKI}
2424
S3_REGION_LOKI=${S3_REGION_LOKI}
2525
S3_SECRET_KEY_LOKI=${S3_SECRET_KEY_LOKI}
2626
STORAGE_DOMAIN=${STORAGE_DOMAIN}
27+
VECTOR_LOG_DESTINATION=${VECTOR_LOG_DESTINATION}

‎services/logging/vector.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ sinks:
5858
loki:
5959
type: loki
6060
inputs: ["process_logs"]
61-
endpoint: "http://loki:3100"
61+
endpoint: "http://${VECTOR_LOG_DESTINATION:?err}:12204"
6262
encoding:
6363
codec: json
6464
labels:
@@ -76,7 +76,7 @@ sinks:
7676
graylog:
7777
type: socket
7878
inputs: ["process_logs"]
79-
address: "logging_graylog:12201"
79+
address: "${VECTOR_LOG_DESTINATION:?err}:12203"
8080
mode: udp
8181
encoding:
8282
codec: gelf

0 commit comments

Comments
 (0)