Skip to content

Commit 54a1084

Browse files
ssorichessoriche
committed
feat(loki): Add Loki project for logs
Initial commit of Loki that collects log data and stores them in an object store, for display/querying in Grafana. The installation uses the Loki project (https://github.com/grafana/loki), manifests generation uses the `Makefile`, with application using kustomize. As the upstream project using helm for configuration, most configuration options are in the values.yaml file. Changes to the settings, requires rerunning the `make manifests` to create the new manifests. Commit the results to the metacpan-k8s repository.
1 parent 817660a commit 54a1084

File tree

8 files changed

+1945
-0
lines changed

8 files changed

+1945
-0
lines changed

‎platform/loki/Makefile‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
.DEFAULT_GOAL := help
2+
3+
# Default Loki version
4+
DEFAULT_VERSION := 6.22.0
5+
LOKI_VERSION ?= $(DEFAULT_VERSION)
6+
LOKI_OUTPUT_YAML := vendor/loki.yaml
7+
VALUES_FILE := $(CURDIR)/values.yaml
8+
9+
.PHONY: all clean manifests check-tools
10+
11+
all: check-tools clean manifests
12+
13+
.PHONY: init
14+
init:
15+
helm repo add grafana https://grafana.github.io/helm-charts
16+
helm repo update
17+
18+
.PHONY: manifests
19+
manifests:
20+
@echo "Removing previous $(LOKI_OUTPUT_YAML)..."
21+
rm -f $(LOKI_OUTPUT_YAML)
22+
@echo "Pulling Loki YAML (Version: $(LOKI_VERSION)) from Helm chart..."
23+
helm repo update
24+
helm template loki --values $(VALUES_FILE) --namespace loki grafana/loki --version $(LOKI_VERSION) > $(LOKI_OUTPUT_YAML)
25+
26+
.PHONY: upgrade
27+
upgrade:
28+
@echo "Removing previous $(LOKI_OUTPUT_YAML)..."
29+
rm -f $(LOKI_OUTPUT_YAML)
30+
@echo "Pulling Loki YAML (Version: $(LOKI_VERSION)) from Helm chart..."
31+
helm repo update
32+
helm upgrade --values $(VALUES_FILE) --namespace loki grafana/loki > $(LOKI_OUTPUT_YAML)
33+
34+
.PHONY: check-tools
35+
check-tools:
36+
@command -v helm >/dev/null 2>&1 || { echo >&2 "helm is not installed. Aborting."; exit 1; }
37+
38+
# Help target to display usage information
39+
.PHONY: help
40+
help:
41+
@echo "Makefile for pulling Loki YAML"
42+
@echo ""
43+
@echo "Targets:"
44+
@echo " init : Add Loki helm chart repository"
45+
@echo " manifests : Pull Loki YAML from Helm chart with specified values.yaml and remove the old Loki YAML"
46+
@echo " upgrade : Pull Loki YAML from Helm chart with specified values.yaml and upgrade the old Loki YAML"
47+
@echo " check-tools : ensure required tools are installed"
48+
@echo " help : Display this help message"

‎platform/loki/README.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Loki
2+
3+
The [Loki](https://github.com/grafana/loki) project is a log aggregation system
4+
designed to store and query logs from multiple sources. It's part of the
5+
[Grafana](https://github.com/grafana/grafana) ecosystem and optimizing for
6+
cost-effective and efficient log storage. Loki's design is to work seamlessly
7+
with Grafana for visualization and Prometheus for metrics.
8+
9+
## Makefile Overview
10+
11+
This Makefile manages the Loki Helm chart, which simplifies the deployment and
12+
management of Loki in a Kubernetes environment. The Makefile provides targets
13+
to automate common tasks such as pulling the latest Loki YAML configuration,
14+
upgrading Loki, and ensure required tools are present.
15+
16+
## Prerequisites
17+
18+
Before using the Makefile, ensure you have the following tools installed:
19+
20+
- `helm`
21+
22+
### Makefile Targets
23+
24+
- **all**: This is the default target that runs `check-tools`, `clean`, and
25+
`manifests` targets in sequence.
26+
- **init**: Adds the Grafana Helm repository and updates the Helm repository
27+
index.
28+
- **manifests**: Removes the previous Loki YAML file and pulls the latest Loki
29+
YAML configuration from the Helm chart using the specified `values.yaml` file.
30+
- **upgrade**: Removes the previous Loki YAML file and upgrades the Loki
31+
deployment using the latest configuration from the Helm chart.
32+
- **check-tools**: Ensures that the required tools (e.g., Helm) are available
33+
on the system.
34+
- **help**: Displays usage information for the Makefile targets.
35+
36+
### Usage
37+
38+
1. **Initialize Helm Repository**
39+
40+
Before using the Makefile, you need to initialize the Helm repository:
41+
42+
```sh
43+
make init
44+
```
45+
46+
2. **Generate Loki Manifests**
47+
48+
To generate the Loki YAML configuration file, run:
49+
50+
```sh
51+
make manifests
52+
```
53+
54+
This will remove the previous `loki.yaml` file and pull the latest
55+
configuration from the Helm chart.
56+
57+
3. **Upgrade Loki**
58+
59+
To upgrade the Loki deployment with the latest configuration, run:
60+
61+
```sh
62+
make upgrade
63+
```
64+
65+
This will remove the previous `loki.yaml` file and upgrade the Loki deployment.
66+
67+
4. **Check Required Tools**
68+
69+
Ensure the installation of required tools by running:
70+
71+
```sh
72+
make check-tools
73+
```
74+
75+
This will check that Helm is available on your system.
76+
77+
5. **Display Help**
78+
79+
To display the help message with information about the Makefile targets, run:
80+
81+
```sh
82+
make help
83+
```
84+
85+
### Default Variables
86+
87+
- **DEFAULT_VERSION**: The default version of Loki to use (e.g., `6.22.0`).
88+
- **LOKI_VERSION**: The version of Loki to use, which can be overridden by
89+
setting this variable.
90+
- **LOKI_OUTPUT_YAML**: The output file for the Loki YAML configuration
91+
(default: `vendor/loki.yaml`).
92+
- **VALUES_FILE**: The path to the `values.yaml` file used for Helm configuration.
93+
94+
### Example
95+
96+
To generate the Loki manifests using a specific version and values file, you
97+
can run:
98+
99+
```sh
100+
make LOKI_VERSION=2.4.1 VALUES_FILE=/path/to/your/values.yaml manifests
101+
```
102+
103+
This will pull the specified version of Loki and use the provided `values.yaml`
104+
file for configuration.

‎platform/loki/do/application.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
apiVersion: argoproj.io/v1alpha1
2+
kind: Application
3+
metadata:
4+
name: loki
5+
namespace: argocd
6+
spec:
7+
project: default
8+
source:
9+
repoURL: https://github.com/metacpan/metacpan-k8s
10+
targetRevision: main
11+
path: platform/loki/do
12+
destination:
13+
server: https://kubernetes.default.svc
14+
namespace: loki
15+
syncPolicy:
16+
automated:
17+
prune: true
18+
selfHeal: false

‎platform/loki/do/kustomization.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
resources:
5+
- ./application.yaml
6+
- ./sealedsecret-loki-secrets.yaml
7+
- ../vendor/

‎platform/loki/do/sealedsecret-loki-secrets.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
apiVersion: bitnami.com/v1alpha1
3+
kind: SealedSecret
4+
metadata:
5+
creationTimestamp: null
6+
name: loki-secrets
7+
namespace: loki
8+
spec:
9+
encryptedData:
10+
grafana-loki-s3-accessKeyId: AgAby4tq0CSboTv68HMUQlaM2JuArvmt35EMw1GTp/px3j5z5eY3eDJX8p9GLOTW4tTTgpvCUTwUYIbe1NC932iCkTDVo7DlJ6cpPTH8IJn0Bw/iCcX0ZWfYP5s/tQu2b7jrtyA4+TehhervYiSYS18Dh4LRZYs/CJIAQeSnX+XttZAsl2Kzfm9VT+7cXSzLVqgH/BXjvaVgTfx1Ijp+unRH+hj5fs7FPG7eQTcsGb5439r03f3QL59izhKMfa9FjSebnUhnLzmXVoic/o3vVn58/rVbJ+gSab1o7472Hwnd2Hx0uDkdzxsau/DjQPHXxjXtwtkROl2ida4MIYaVxT2RPDQPOoa8iphVLRGDo70R+2AXYv5rjP4iQcsLuTK9A9bG/Hlb2sZHG0CLJvMHBIHIQMtokN+ZJsRuCvR3sgWYBFhxStWxtM07PE+epbbEdOZK9V+WkbQEtgxU6Sfg4EG++r/nz2aXe/aji9QfuJpwCo8bLb3NtuOf2H5jrueOyicgGEv1IUTc+4mML50ZFdFU/JF/zU+y03cUoO8qzGJjrwL3epRfxwHapsVwaOLvcp5mAL4wR5Yfp6C5bTGQrq0mLrCDwEH+eyB1r//EqCAUUMUhesjSurb16CjTjthBEEX6LsS2wz7rXmj19gm44/G1OwPEpnwh6UHJRV9Aq6hNWhprAgesBEArAyCozLdWhk/7rsosnLHhkad3O8yi10FOX1RzJg==
11+
grafana-loki-s3-endpoint: AgBTcLzWaCgc9SFaqQVbjUJDM2aFFI4zmHj0jgDiKgdPn8HlvKwhUU7b+ztyBHBczLY5FMF9PgMOHX8M+SXRn6aAXIvLQXns+poP9bipID4DghptoG1HE9NvxQ68KKJXCc+siJka8oXTFYSj9PXYEG7ntoRlfFhem858NyoMfCMQ6t2cnAbw3Qu8+boIBlmv9gh6fFm5LC9Hm9QF/zOUqkM/J/OVOh+SXEnbbxukDnINoeFybnl5ZOCfhfrXyaWruU2J+FxKBYq9th6VUJVcr3pTd7GZQ0Z5DZY/FMO4ecpDG397MrjysHu2oBstt/7Gzd2iWSMdvBTkUWxs1gktXT48FMS94hBlj1KhGnc5Qg1Y3xkAfaEA6DWQRwfJGWiueVyOQVG1ikpiog0fJ8Ga/vLsM+TtHZizIjioSfu1H4DmUsR9NlS5nOPkFQUqzqFqkxW9Da9xtGNDFuUX4kONUJL1P2WO0UmDnzG9OQYZnIOEq+qrQ01qXnzs3TjV3q6N0Cvpocn8ZDfj3G5fpg4GNMftiMInZZoTLXktiFpTteRhWaG8Ku0s6iGb8lTFrOEpcR1RgJMB97pyNGbN03VcDIW46YXc4CT0HGMcDzgILSISP/5ajLjUX2vHUfSkKXRe1aC+sTc46lld7Q7lRNfoO56mFhuneINF2pf0kS402GVpFFwZ+DUhW/xMVl1gbBUSHtsYVFLm+Fg2D+L4HC/D5Qh8xIdcSz2BPu2xhPQ=
12+
grafana-loki-s3-secretAccessKey: AgCSPYP8neHgp8XNdAnB7OmvhLAgLvrGO3sa8cgC/kPsWltiDd1X+L+RE15WVeR/EHwWVkaAe8UQvLKcftpd9nW+vShPVJgECtg9HaMc4WVWx4cioNR8g9MHqV8XFuld/FBbrS+eurnUnmgb0anUZdc33creZW4XcqZ0REbGcPua50aRbrRSCoKHJFQbzHRakf/ek99gAP11qEUPt8pqDFABU9fnn13CjNVuI+QQhnt3FMmGaZnBXeszo9fNpU0eir4eTcSOw4xSHg+xVgVDa5i678ThMIXf3SUjYxC12TJysrW9Cnq1Vp5emYzCeNqhTPLC31SrID0h/e2FKizaRQdHkUqvzG44fGHt36EKbcA1JfjDoK0EWqGMkRhh2EeeM8vj1sUfS8i4nKJoBsHB9jHCJB5Fzm6/uHI8DY+A1RRMsiKoKQnh/RWTkbYW724Jz3wxEbmDK0UiBFuZcTWj/orLTAHK/yLriy/Bt9WGkAh76cYCoZvCSnoCUL4QShrs8ez3DAk/Fg2hfoUYOJNB/6Fxb6upd1j8OivngTNrwghMvhU6bUs42IKvmAYzuMIFiJPkmKrIG03D7GqxbjHtOmBJEv8pZEQNk5xBoyTCY12aIxmz2c9c8s8G42YyKV62dEZyfFPb3Gac6ppNxMqheVhgYKLOBsKCDBDRuVQpEHyWvpfK/G0QVutpa/PF87gKWOCrKs1llOGYU6NrX/A9qjl9r8yDx8BtoTAHqBetj4v5seE5e8jYoV6712Vz
13+
template:
14+
metadata:
15+
creationTimestamp: null
16+
name: loki-secrets
17+
namespace: loki
18+
type: Opaque

‎platform/loki/values.yaml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
loki:
2+
auth_enabled: false
3+
schemaConfig:
4+
configs:
5+
- from: "2024-04-01"
6+
store: tsdb
7+
object_store: s3
8+
schema: v13
9+
index:
10+
prefix: loki_index_
11+
period: 24h
12+
storage_config:
13+
aws:
14+
region: fra1
15+
bucketnames: mc-do-logs
16+
s3forcepathstyle: true
17+
pattern_ingester:
18+
enabled: true
19+
limits_config:
20+
allow_structured_metadata: true
21+
volume_enabled: true
22+
retention_period: 672h # 28 days retention
23+
querier:
24+
max_concurrent: 4
25+
26+
storage:
27+
type: s3
28+
bucketNames:
29+
chunks: mc-do-logs
30+
ruler: mc-do-logs
31+
admin: mc-do-logs
32+
s3:
33+
# AWS endpoint URL
34+
endpoint: "${GRAFANA_LOKI_S3_ENDPOINT}"
35+
# AWS region where the S3 bucket is located
36+
region: fra1
37+
# AWS secret access key
38+
secretAccessKey: "${GRAFANA_LOKI_S3_SECRETACCESSKEY}"
39+
# AWS access key ID
40+
accessKeyId: "${GRAFANA_LOKI_S3_ACCESKEYID}"
41+
# AWS signature version (e.g., v2 or v4)
42+
signatureVersion: v4
43+
# Forces the path style for S3 (true/false)
44+
s3ForcePathStyle: true
45+
# Allows insecure (HTTP) connections (true/false)
46+
# HTTP configuration settings
47+
48+
deploymentMode: SimpleScalable
49+
50+
backend:
51+
replicas: 3
52+
extraArgs:
53+
- "-config.expand-env=true"
54+
extraEnv:
55+
- name: GRAFANA_LOKI_S3_ENDPOINT
56+
valueFrom:
57+
secretKeyRef:
58+
name: loki-secrets
59+
key: grafana-loki-s3-endpoint
60+
- name: GRAFANA_LOKI_S3_ACCESKEYID
61+
valueFrom:
62+
secretKeyRef:
63+
name: loki-secrets
64+
key: grafana-loki-s3-accessKeyId
65+
- name: GRAFANA_LOKI_S3_SECRETACCESSKEY
66+
valueFrom:
67+
secretKeyRef:
68+
name: loki-secrets
69+
key: grafana-loki-s3-secretAccessKey
70+
read:
71+
replicas: 3
72+
extraArgs:
73+
- "-config.expand-env=true"
74+
extraEnv:
75+
- name: GRAFANA_LOKI_S3_ENDPOINT
76+
valueFrom:
77+
secretKeyRef:
78+
name: loki-secrets
79+
key: grafana-loki-s3-endpoint
80+
- name: GRAFANA_LOKI_S3_ACCESKEYID
81+
valueFrom:
82+
secretKeyRef:
83+
name: loki-secrets
84+
key: grafana-loki-s3-accessKeyId
85+
- name: GRAFANA_LOKI_S3_SECRETACCESSKEY
86+
valueFrom:
87+
secretKeyRef:
88+
name: loki-secrets
89+
key: grafana-loki-s3-secretAccessKey
90+
write:
91+
replicas: 3
92+
extraArgs:
93+
- "-config.expand-env=true"
94+
extraEnv:
95+
- name: GRAFANA_LOKI_S3_ENDPOINT
96+
valueFrom:
97+
secretKeyRef:
98+
name: loki-secrets
99+
key: grafana-loki-s3-endpoint
100+
- name: GRAFANA_LOKI_S3_ACCESKEYID
101+
valueFrom:
102+
secretKeyRef:
103+
name: loki-secrets
104+
key: grafana-loki-s3-accessKeyId
105+
- name: GRAFANA_LOKI_S3_SECRETACCESSKEY
106+
valueFrom:
107+
secretKeyRef:
108+
name: loki-secrets
109+
key: grafana-loki-s3-secretAccessKey
110+
111+
chunksCache:
112+
replicas: 1
113+
allocatedMemory: 768
114+
# Disable minio storage
115+
minio:
116+
enabled: false

‎platform/loki/vendor/kustomization.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
resources:
5+
- ./loki.yaml

0 commit comments

Comments
 (0)