[AWS Master] Kubernetes: add logging stack (#1063)

* Add dir for grafana loki chart

* Switch from loki to ELK stack

* Longhorn Readme. Fix typo

* Further configuration

* Final draft ELK configuration

* add victoria logs

* Introduce victoria logs and auth

* vl ha configuration with 2 charts + 1 vmauth chart

* Converge on a single replicated victoria logs chart

* Remove elastic stack chart

* Remove vector chart (already included in victora logs chart)

* Fix gui trailing slash issue with a href

Author: YuryHrytsuk

charts/aws-ebs-csi-driver/values.yaml.gotmpl

@@ -5,7 +5,7 @@ image:
     tag: "v1.38.1"
 
 storageClasses:
-  - name: "ebs-sc"
+  - name: "{{ .Values.ebsStorageClassName }}"
     parameters:
         type: "gp3"
     allowVolumeExpansion: true

charts/longhorn/README.md

@@ -2,7 +2,7 @@
 
 ### Can LH be used for critical services (e.g., Databases)?
 
-No (as of now). , we should not use it for volumes of critical services.
+No. We should not use it for volumes of critical services.
 
 As of now, we should avoid using LH for critical services. Instead, we should rely on easier-to-maintain solutions (e.g., application-level replication [Postgres Operators], S3, etc.). Once we get hands-on experience, extensive monitoring and ability to scale LH, we can consider using it for critical services.
 

charts/portainer/values.ebs-pv.yaml.gotmpl

@@ -1,4 +1,4 @@
 persistence:
   enabled: true
   size: "1Gi"  # minimal size for gp3 is 1Gi
-  storageClass: "ebs-sc"
+  storageClass: "{{ .Values.ebsStorageClassName }}"

charts/portainer/values.longhorn-pv.yaml.gotmpl

@@ -1,4 +1,4 @@
 persistence:
   enabled: true
   size: "300Mi" # cannot be lower https://github.com/longhorn/longhorn/issues/8488
-  storageClass: "{{.Values.longhornStorageClassName}}"
+  storageClass: "{{ .Values.longhornStorageClassName }}"

charts/traefik/values.secure.yaml.gotmpl

@@ -60,6 +60,27 @@ extraObjects:
       prefixes:
       - /longhorn
 
+  # a (href) links do not work properly without trailing slash
+- apiVersion: traefik.io/v1alpha1
+  kind: Middleware
+  metadata:
+    name: logs-append-slash
+    namespace: {{ .Release.Namespace }}
+  spec:
+    redirectRegex:
+      regex: "^(https?://[^/]+/logs)$"
+      replacement: "${1}/"
+
+- apiVersion: traefik.io/v1alpha1
+  kind: Middleware
+  metadata:
+    name: logs-strip-prefix
+    namespace: {{.Release.Namespace}}
+  spec:
+    stripPrefix:
+      prefixes:
+      - /logs
+
 - apiVersion: traefik.io/v1alpha1
   kind: Middleware
   metadata:

charts/victoria-logs/README.md

@@ -0,0 +1,2 @@
+Highly Available Configuration with Helm:
+* https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9076

charts/victoria-logs/values.yaml.gotmpl

@@ -0,0 +1,59 @@
+# https://github.com/VictoriaMetrics/helm-charts/blob/victoria-logs-single-0.11.2/charts/victoria-logs-single/values.yaml
+
+vector:
+  # by default it will generate sink per statefulset's pod
+  # each pod has a separate PV, so the data is replicated
+  enabled: true
+
+server:
+  # HA trough multiple replicas
+  # https://github.com/VictoriaMetrics/VictoriaMetrics/issues/9076
+  replicaCount: 2
+
+  retentionPeriod: 30d
+
+  ingress:
+    enabled: true
+    annotations:
+        namespace: "{{ .Release.Namespace }}"
+        cert-manager.io/cluster-issuer: "cert-issuer"
+        traefik.ingress.kubernetes.io/router.entrypoints: websecure
+        traefik.ingress.kubernetes.io/router.middlewares: traefik-logs-append-slash@kubernetescrd,traefik-logs-strip-prefix@kubernetescrd,traefik-traefik-basic-auth@kubernetescrd # namespace + middleware name
+    tls:
+      - hosts:
+          - {{ requiredEnv "K8S_MONITORING_FQDN" }}
+        secretName: monitoring-tls
+    hosts:
+      - name: {{ requiredEnv "K8S_MONITORING_FQDN" }}
+        path:
+          - /logs
+        pathType: Prefix
+
+  persistentVolume:
+    enabled: true
+    storageClassName: "{{ .Values.ebsStorageClassName }}"
+    size: 10Gi
+
+  nodeSelector:
+    ops: "true"
+
+  # Schedule pods on different nodes if possible (HA)
+  # https://stackoverflow.com/a/64958458/12124525
+  topologySpreadConstraints:
+    - maxSkew: 1
+      topologyKey: "kubernetes.io/hostname"
+      whenUnsatisfiable: DoNotSchedule
+      # hardcoded due to https://github.com/VictoriaMetrics/helm-charts/issues/2219
+      labelSelector:
+        matchLabels:
+          app: server
+          app.kubernetes.io/instance: victoria-logs
+          app.kubernetes.io/name: victoria-logs-single
+
+  resources:
+    limits:
+      cpu: 500m
+      memory: 512Mi
+    requests:
+      cpu: 500m
+      memory: 512Mi