Loki doc updates (#491)

* Update Grafana images and associated text

* Add some query tips in Loki doc section

* Downsize Grafana screenshots

* Separate Loki section from monitoring config

* Fix broken link

* Add Additional Loki links

* add details on how logging with loki works

* fix spelling and run yarn linter

* Apply suggestions from code review

Co-authored-by: Kim Pevey <[email protected]>
Co-authored-by: Vinicius D. Cerutti <[email protected]>

* fix link

* Add permissions note and typo fix

---------

Co-authored-by: Kim Pevey <[email protected]>
Co-authored-by: Vinicius D. Cerutti <[email protected]>

Author: 3 people

docs/docs/how-tos/access-logs-loki.md

@@ -0,0 +1,76 @@
+---
+title: Access system logs (Loki) via Grafana
+description: Access common system logs on Nebari
+---
+
+# How to access system logs (Loki) via Grafana
+
+Below is a step-by-step walkthrough of how to view JupyterHub pod logs in Grafana [Loki](https://grafana.com/docs/loki/latest/). As similar approach can be used to find other commonly accessed logs (summarized at the end of this document).
+
+To view the Loki logs via Grafana in Nebari, you'll need to have [set up monitoring](/docs/how-tos/setup-monitoring) on your deployment.
+
+## Getting Started
+
+Access the Monitoring UI for your Nebari installation at https://{your-nebari-domain}/monitoring/.
+
+:::note
+The **Explore** functionality shown below is only available to users who have `grafana_admin` permissions.  In Nebari's default configuration, only the users in the `admin` Keycloak user group will have this role.  `grafana_admin` is a client role in Keycloak which can be assigned to other groups or users.  See [Configure Keycloak](/docs/how-tos/configuring-keycloak#in-depth-look-at-roles-and-groups) for more information.
+:::
+
+First, click "Explore".
+
+![Grafana Explore Page](/img/how-tos/1_grafana-explore.png)
+
+Select Loki as the data source at the top:
+
+![Grafana Select Loki](/img/how-tos/2_grafana-select-loki.png)
+
+Select a Label to search. For this example, we will select `pod`:
+
+![Grafana Select Loki](/img/how-tos/3_grafana-log-browser-pod.png)
+
+Begin typing the name of the desired pod. In this case we are looking
+for a JupyterHub pod. The exact name will vary on each deployment but it will begin with `hub-` followed by a unique identifier.
+
+![Grafana Select Loki](/img/how-tos/4_grafana-log-search-pod.png)
+
+Select the pod from the list of pods and then click on "Run Query":
+
+![Grafana Select Loki](/img/how-tos/5_grafana-log-select-pod.png)
+
+After clicking on "Run Query", you should be able to see logs for JupyterHub pod as shown below:
+
+![Grafana Select Loki](/img/how-tos/6_grafana-view-pod-logs.png)
+
+You can also filter by time by clicking on the time filter on top right (next to "Run query").
+
+## Common Queries
+
+The approach above can be used to access a wide variety of logs on Nebari. Below are some common queries.
+
+### JupyterLab Server Logs
+
+Each user will have their own JupyterLab instance (which may or may not be running at any given time) which will contain its own set of logs. To view the logs for a JupyterLab server, use `pod` label and begin typing the username of interest. The pod name will be `jupyter-{username}`.
+
+### Conda-Store Logs
+
+Conda-store runs multiple pods on the backend. The conda-store server runs continuously while conda-store-workers are only started when an environment is being built. Therefore, worker pods may not always exist. Generally, user requests and access logs will be in the server pod, while logs related to environment builds will be in a worker pod. In the list of conda-store pods shown below, you'll also see pods for `minio`, `postgresql`, and `redis`. These are used internally by conda-store are not likely to have any logs of interest.
+
+- `nebari-conda-store-server-[id]`
+- `nebari-conda-store-worker-[id]`
+- `nebari-conda-store-minio-[id]`
+- `nebari-conda-store-postgresql-postgresql-0`
+- `nebari-conda-store-redis-master-0`
+
+### Deployed app logs (via jhub-apps)
+
+If you have `jhub-apps` enabled on your deployment, you can view the logs to debug deployed apps.
+
+To see the logs from **all deployed apps**, use the label filter `container` = `notebook`. The main container in each deployed app pod is named `notebook`.
+
+To see **logs from a specific app**, use the `pod` label and begin typing either the name of the user running the app or the app name to find the correct pod. App pods are named with the convention `jupyter-[username]--[app_name]-[pod_id]`.
+
+## Additional Information
+
+- [Understand Log Query Structure](https://grafana.com/docs/loki/latest/query/log_queries/)
+- [Use the Query Editor](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/#choose-a-query-editing-mode)

docs/docs/how-tos/monitoring.md renamed to docs/docs/how-tos/setup-monitoring.md

@@ -1,4 +1,4 @@
-# Monitoring
+# How to Set Up Monitoring on Nebari
 
 In Nebari, we've integrated Grafana, Prometheus, and Loki to provide robust monitoring capabilities for
 your data science platform. This integration allows you to visualize metrics, monitor system health, and
@@ -28,37 +28,9 @@ metrics from configured targets, stores them efficiently, and allows querying th
 
 [Loki](https://grafana.com/docs/loki/latest/) is a horizontally-scalable, highly available, multi-tenant log
 aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it
-does not index the contents of the logs, but rather a set of labels for each log stream. Below is a step-by-step
-walkthrough of how to view JupyterHub pod logs in Grafana Loki:
+does not index the contents of the logs, but rather a set of labels for each log stream.
 
-Go to explore section of monitoring of your Nebari installation at:
-
-https://{your-nebari-domain}/monitoring/
-
-![Grafana Explore Page](/img/how-tos/1_grafana-explore.png)
-
-Select Loki Data source at the top:
-
-![Grafana Select Loki](/img/how-tos/2_grafana-select-loki.png)
-
-Click on the Log browser and select labels to search in, in this case we have selected pod:
-
-![Grafana Select Loki](/img/how-tos/3_grafana-log-browser-pod.png)
-
-Type the pod name initials for the pod you're looking to search logs for, in this case we are looking
-for hub pod:
-
-![Grafana Select Loki](/img/how-tos/4_grafana-log-search-pod.png)
-
-Select the pod from the list of pods and then click on "Show logs":
-
-![Grafana Select Loki](/img/how-tos/5_grafana-log-select-pod.png)
-
-After clicking on "Show logs", you should be able to see logs for JupyterHub pod as shown below:
-
-![Grafana Select Loki](/img/how-tos/6_grafana-view-pod-logs.png)
-
-You can also filter by time, by clicking on the time filter on top right, next to "Run query".
+See [How to access system logs (Loki) via Grafana][access-logs-loki] for more information on using Loki in Nebari.
 
 ## Terraform Overrides
 
@@ -147,3 +119,30 @@ monitoring:
         compactor:
           retention_enabled: false
 ```
+
+## Logging architecture
+
+The architecture diagram below shows a simplified, high level explanation of the logging components on Nebari.
+
+![Grafana](/img/how-tos/grafana-loki-promtail-architecture.png)
+
+`Grafana` is the dashboarding user interface which allows us to use `Loki` as the data source for our logs. `Loki` connects to [`promtail`](https://grafana.com/docs/loki/latest/send-data/promtail/) as it's source.
+
+The `promtail` component scrapes logs from various pods on the kubernetes nodes. The `kube api server` provides the API endpoints which `promtail` uses for for discovering and scraping its targeted resources
+
+End users viewing the logs in `Grafana` will create queries using `Loki` as the data source, typically querying based on `labels`. However, it's important to note that Grafana labels differ from Kubernetes labels, as their main goal is to act as an aggregation layer of logs from multiple matching resources into a single "stream," allowing users to easily access a collection of logs from various Kubernetes resources with just a single logical label.
+
+:::note
+Loki's "labels" are used to filter collections of logs from the available [kubernetes_sd](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#kubernetes_sd_config) API endpoints, in a similar way as to how Prometheus handles metrics. These labels are configured through Promtail, which is the agent responsible for collecting and shipping logs to Loki, based on the defined [targets](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#scrape_configs) and scraping configurations.
+:::
+
+For details on how to view specific logs in Loki, check out the document ["How to access system logs (Loki) via Grafana"](access-logs-loki)
+
+## References
+
+[More information on promtail configurations](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/)
+[Understanding labels in Loki](https://grafana.com/docs/loki/latest/get-started/labels/#understand-labels)
+
+<!-- Internal links -->
+
+[access-logs-loki]: /how-tos/access-logs-loki.md

docs/sidebars.js

@@ -78,7 +78,8 @@ module.exports = {
         "how-tos/idle-culling",
         "how-tos/nebari-extension-system",
         "how-tos/telemetry",
-        "how-tos/monitoring",
+        "how-tos/setup-monitoring",
+        "how-tos/access-logs-loki",
         "how-tos/use-gpus",
         "how-tos/fine-grained-permissions",
       ],