feat(post): Added Grafana Loki for Kubernetes

Author: timothystewart6

_posts/2021-11-20-grafana-loki-kubernetes.md

@@ -0,0 +1,101 @@
+---
+layout: post
+title: "Installing Grafana Loki with Helm on Kubernetes"
+date: 2021-11-20 7:00:00 -0500
+categories: kubernetes
+tags: homelab proxmox grafana logging promtail prometheus kubernetes helm
+---
+
+In my previous video ([Meet Grafana LOKI, a log aggregation system for everything](https://img.youtube.com/vi/h_GGd7HfKQ8/0.jpg) and [post](https://techno-tim.github.io/posts/grafana-loki/), I promised that I would also explain how to install Granfana Loki on Kubernetes using `helm`.  If you're looking to set this up in `docker-compose`, be sure to check out this [video](https://www.youtube.com/watch?v=h_GGd7HfKQ8)
+
+## Installing helm
+
+Think of `helm` as a package manager for kubernetes. It'a an easy way to bundle and deploy config to kubernetes with versioning.  If you need to install `helm` visit [helm.sh](https://helm.sh/docs/intro/install/)
+
+## Installing Loki Stack
+
+This command will:
+
+* install grafana
+* install prometheus
+* install loki
+* enable persistance for your stack and create a PVC
+
+```bash
+helm upgrade --install loki grafana/loki-stack  --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=nfs-client,loki.persistence.size=5Gi
+```
+
+You'll want to set `loki.persistence.storageClassName=nfs-client` to your `StorageClass`  
+In this example, I am using `nf-client` which is the [Kubernetes NFS Subdir External Provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner)
+
+## Accessing the Grafana Dashboard
+
+To access your Grafana dashboard you can run
+
+```bash
+kubectl port-forward --namespace <YOUR-NAMESPACE> service/loki-grafana 3000:80
+```
+
+To get the password for the `admin` user run
+
+```bash
+kubectl get secret --namespace <YOUR-NAMESPACE> loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
+```
+
+This should print out your password
+
+You can now access your dashboard on `http://localhost:3000`
+
+## Ingress with Traefik
+
+If you want to create an `IngressRoute` and you are using traefik can you apply the following
+
+`ingress.yml`
+
+```yml
+apiVersion: traefik.containo.us/v1alpha1
+kind: IngressRoute
+metadata:
+  name: loki-grafana-ingress
+  annotations: 
+    kubernetes.io/ingress.class: traefik-internal # change with your value
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`grafana.example.com`) # change with your value
+      kind: Rule
+      services:
+        - name: loki-grafana
+          port: 80
+```
+
+```bash
+kubectl apply -f ingress.yml
+```
+
+You should now be able to access your dashboard on `https://grafana.example.com`
+
+## LogQL sample queries
+
+Query all logs from the `container` label
+
+```sql
+{container="uptime-kuma"} 
+```
+
+query all logs from the `container` stream and filter on `error`
+
+```sql
+{container="uptime-kuma" |= "error"}
+
+```
+
+query all logs from the `pod` label of `uptime-kuma-8d45g32fd-lk8rl`
+
+```sql
+{pod="uptime-kuma-8d45g32fd-lk8rl"}
+
+```
+
+Read more about LogQL [here](https://grafana.com/docs/loki/latest/logql/)

_posts/2021-11-20-grafana-loki.md

@@ -10,7 +10,6 @@ tags: homelab proxmox grafana logging promtail prometheus
 
 I've been on a quest to find a new logging system.  I've use quite a few in the past, some open source, some proprietary, and some home grown, but recently I've decided to switch.  I've switched to Grafana Loki for all of my logs for all of my systems - this includes machines, devices, docker systems and hosts, and my all of my kubernetes clusters.  If you're thinking of using Grafana and are also looking for a fast way to log all of your systems, join me as we discuss and configure Grafana Loki.
 
-
 [Watch Video](https://www.youtube.com/watch?v=h_GGd7HfKQ8)
 
 (see video description for links to gear, discord, and other ways to connect.)
@@ -74,7 +73,9 @@ services:
       - loki
 ```
 
-```
+## Loki Config
+
+```bash
 touch nano loki/loki-config.yml
 ```
 
@@ -113,9 +114,10 @@ ruler:
   alertmanager_url: http://localhost:9093
 ```
 
+## Promtail Config
 
-```
-touch nano promtail/loki-config.yml
+```bash
+touch nano promtail/promtail-config.yml
 ```
 
 `promtail-config.yml`
@@ -132,6 +134,7 @@ clients:
   - url: http://loki:3100/loki/api/v1/push
 
 # local machine logs
+
 scrape_configs:
 - job_name: local
   static_configs:
@@ -141,7 +144,8 @@ scrape_configs:
       job: varlogs
       __path__: /var/log/*log
 
-## docker logs  
+## docker logs
+
 # scrape_configs:
 #   - job_name: docker 
 #     pipeline_stages:
@@ -152,6 +156,7 @@ scrape_configs:
 #           __path__: /var/lib/docker/containers/*/*-json.log
 
 ## syslog target
+
 # scrape_configs:
 #   - job_name: syslog
 #     syslog:
@@ -165,9 +170,9 @@ scrape_configs:
 #         target_label: 'host'
 ```
 
+## Loki Docker Driver
 
-
-```
+```bash
 sudo nano /etc/daemon.json
 ```
 
@@ -185,4 +190,28 @@ sudo nano /etc/daemon.json
 
 ```bash
  sudo systemctl restart docker
-```
+```
+
+## LogQL sample queries
+
+Query all logs from the `varlogs` stream
+
+```sql
+{job="varlogs"} 
+```
+
+query all logs from the `varlogs` stream and filter on  `docker`
+
+```sql
+{job="varlogs" |= "docker"}
+
+```
+
+query all logs from the `container_name` label of `uptime-kuma` and filter on `host` of `juno`
+
+```sql
+{container_name="uptime-kuma", host="juno"}
+
+```
+
+Read more about LogQL [here](https://grafana.com/docs/loki/latest/logql/)