Merge branch 'main' into rook-ceph

Author: Adam-D-Lewis

.github/workflows/test-website.yml

@@ -33,5 +33,8 @@ jobs:
       - name: Lint 🔍
         run: yarn run lint
 
+      - name: Format 🥇
+        run: yarn run format
+
       - name: Build site 🔨
         run: yarn run build

docs/docs/community/decision-making.md

@@ -40,7 +40,6 @@ Learn more in [Gitpod's documentation on decision making][gitpod-rfd].
 
 Minor updates to the codebase and documentation can be discussed in GitHub issues or in pull requests during code review. Contributors are expected to (informally) follow the consent-based decision making philosophy in these discussions.
 
-
 <!-- Reusable links -->
 
 [nebari-team]: /docs/community/team-structure

docs/docs/community/plugins.md

@@ -12,5 +12,6 @@ You can learn to use and create Nebari extensions in the [extension mechanism do
 
 The following community-developed extensions are recognized and verified by the Nebari development team.
 
-* [nebari-mlflow-aws](https://github.com/MetroStar/nebari-mlflow-aws)
-* [nebari-label-studio](https://github.com/MetroStar/nebari-label-studio)
+- The **[ML Flow extension for AWS](https://github.com/MetroStar/nebari-mlflow-aws)** is designed to integrate [MLFlow](https://mlflow.org/) into Nebari deployments utilizing AWS (Amazon Web Services) as the provider. It provides a robust, collaborative environment for AI/ML professionals to manage experiments, track metrics, and deploy models.
+- The **[Label Studio](https://github.com/MetroStar/nebari-label-studio)** integrates [Label Studio](https://labelstud.io/) into the Nebari platform, allowing seamless labeling functionality within Nebari. Utilizing Python, Terraform, Kubernetes, and Helm charts, the plugin provides a configurable deployment and authentication through Keycloak.
+- The **[Self Registration](https://github.com/nebari-dev/nebari-self-registration)** extension allows potential new users of a Nebari deployment to self-register through a coupon code. A new self-registration page is generated on the Nebari server where users can input their information and a coupon code. Once the form is validated, the new user will be auto-generated on the Nebari deployment. It can also be configured to give new users a set expiration date.

docs/docs/community/team-structure.md

@@ -57,9 +57,9 @@ Nebari follows a nomination process to add new team members[^2], as detailed in
 
 [^2]: All except the [Emeritus core](#emeritus-core) team are decided through nominations.
 
-| Team        | Requirements for nomination                                                                                         | Nominators                                                          | Approvers                                                                                                               |
-| ----------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
-| Triage      | Engage with (create or comment on) one or more issue/PR/discussion in [`nebari-dev`](https://github.com/nebari-dev) | Any community member (including self)                                        | Any Core team member                                                                                                    |
-| Contributor | Two or more [contributions](./#how-to-contribute) to Nebari with intention to continue contributing regularly                                          | Any community member (including self)                                    | Any Core team member                                                                                |
-| Core        | [Contributors team](#contributors) member with a record of regular, valuable, and high-quality contributions to Nebari for at least one month  | Any community member (including self) and one Core team member | Core team makes a [consent-based decision](https://www.sociocracyforall.org/consent-decision-making/) |
-| Conduct     | Member of the Contributor or Core Team with adequate training to handle CoC reports         | Any community member (including self)                                    | Core team makes a [consent-based decision](https://www.sociocracyforall.org/consent-decision-making/)                                                                                 |
+| Team        | Requirements for nomination                                                                                                                   | Nominators                                                     | Approvers                                                                                             |
+| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
+| Triage      | Engage with (create or comment on) one or more issue/PR/discussion in [`nebari-dev`](https://github.com/nebari-dev)                           | Any community member (including self)                          | Any Core team member                                                                                  |
+| Contributor | Two or more [contributions](./#how-to-contribute) to Nebari with intention to continue contributing regularly                                 | Any community member (including self)                          | Any Core team member                                                                                  |
+| Core        | [Contributors team](#contributors) member with a record of regular, valuable, and high-quality contributions to Nebari for at least one month | Any community member (including self) and one Core team member | Core team makes a [consent-based decision](https://www.sociocracyforall.org/consent-decision-making/) |
+| Conduct     | Member of the Contributor or Core Team with adequate training to handle CoC reports                                                           | Any community member (including self)                          | Core team makes a [consent-based decision](https://www.sociocracyforall.org/consent-decision-making/) |

docs/docs/explanations/advanced-custom-settings.md

@@ -276,6 +276,8 @@ One way to achieve this is by creating a Virtual Machine (VM) inside the virtual
 Select the virtual network and subnet name under the networking settings of your cloud provider while creating the VM
 and then follow the usual deployment instructions as you would deploy from your local machine.
 
+=======
+
 #### Conda store worker
 
 You can use the following settings to change the defaults settings (shown) used for Conda store workers.
@@ -292,3 +294,55 @@ conda_store:
 :::note Note
 Current `conda_store.worker_resources` defaults are set at the minimum recommended resources for conda-store-workers - (conda-store [docs](https://conda.store/conda-store/references/faq#what-are-the-resource-requirements-for-conda-store-server))
 :::
+
+## Helm Extensions
+
+Nebari provides a way for any user to expand the infrastructure available by default by using the `helm_extensions` attribute. This attribute allows for the management and customization of Kubernetes applications through Helm charts. The helm_extensions is a configuration construct that specifies a list of Helm charts and their respective settings.
+
+### Overview
+
+Each entry in the `helm_extensions` list represents a single Helm chart configuration, allowing you to define the chart source, version, and specific overrides or settings for that chart. When Nebari is deployed, it will install the specified Helm charts using the provided settings.
+
+### Structure
+
+Each entry in the helm_extensions list contains the following fields:
+
+- `name`: A unique identifier for the Helm chart. It will also be used as the name of the Kubernetes deployment related resources.
+- `repository`: The URL of the repository where the Helm chart is stored. Must be a valid Helm repository URL.
+- `chart`: The name or path of the chart within the repository. must be compliant with the Helm chart naming conventions.
+- `version`: The specific version of the chart to be used.
+- `overrides`: Specific configurations to override default chart values.
+
+:::note Note
+The `overrides` field is optional. If not specified, the default values for the chart will be used.
+:::
+
+### Example
+
+Below we give an example showcasing how to install Redis using helm_extensions:
+
+```yaml
+helm_extensions:
+  - name: redis-deployment
+    repository: https://charts.bitnami.com/bitnami
+    chart: redis
+    version: 17.7.5
+    overrides:
+      architecture: standalone
+      master:
+        containerSecurityContext:
+          runAsUser: 0
+        persistence:
+          enabled: true
+          path: /redis/data
+          subPath: redis/data
+          existingClaim: <existing-claim-name-is-required>
+      replica:
+        persistence:
+          enabled: false
+        replicaCount: 0
+```
+
+:::warning Warning
+In the above example, we are assuming the current nebari kubernetes cluster already has an appropriate storage class and persistent volume claim (PVC) created. If not, you will need to create a storage class and PVC before deploying the helm chart.
+:::

docs/docs/faq.md

@@ -31,8 +31,9 @@ There are drop-in replacements for `distributed`, `dask`, and `dask-gateway` wit
 ## What packages are needed in your environment to create a dashboard?
 
 When deploying an app with JHub App Launcher, you need to have the following in your environment:
+
 - `jhub-apps` package
-- packages corresponding  to the dashboard framework (for example, `panel`, `gradio`, etc.)
+- packages corresponding to the dashboard framework (for example, `panel`, `gradio`, etc.)
 - any other libraries required for the analysis in the dashboard creation script/notebook
 
 ## How can I install a package locally? Will this package be available to Dask workers?
@@ -163,6 +164,7 @@ Nebari automatically shuts down servers when users are idle, as described in Neb
 :::note
 Until this issue is addressed, we recommend manually shutting down your VS Code server when it is not in use.
 :::
+
 <!-- Internal links -->
 
 [dask-tutorial]: tutorials/using_dask.md

docs/docs/get-started/installing-nebari.md

@@ -22,7 +22,8 @@ Nebari heavily depends on [Terraform](https://www.terraform.io/) and Python. The
 - To install conda, visit the [official conda documentation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html), or if you prefer, visit the [mamba installation documentation](https://github.com/mamba-org/mamba#installation)
 
 ## How to install the Nebari Package
-*See [Environment Management][environment-management] for best practices for using `conda` and `pip` to control your deployment environment.*
+
+_See [Environment Management][environment-management] for best practices for using `conda` and `pip` to control your deployment environment._
 
 There are currently two ways to install the Nebari CLI:
 
@@ -45,12 +46,13 @@ There are currently two ways to install the Nebari CLI:
    ```
 
 :::note
-The version of Nebari in your `nebari-config.yaml` must match your currently installed Nebari package version; otherwise, a warning will be raised when attempting to deploy.  See [Upgrading Nebari][nebari-upgrade] for techniques for upgrading your Nebari CLI or safely updating your older config file to match your Nebari package version.
+The version of Nebari in your `nebari-config.yaml` must match your currently installed Nebari package version; otherwise, a warning will be raised when attempting to deploy. See [Upgrading Nebari][nebari-upgrade] for techniques for upgrading your Nebari CLI or safely updating your older config file to match your Nebari package version.
 :::
 
 :::note
-The Nebari CLI will auto-detect and then deploy any [Nebari Extensions][nebari-extension-system] that are installed in your Python environment, and extensions once deployed cannot be uninstalled.  For this reason, we recommend creating a [unique environment][environment-management] for each Nebari deployment, especially when managing multiple deployments with extensions.
+The Nebari CLI will auto-detect and then deploy any [Nebari Extensions][nebari-extension-system] that are installed in your Python environment, and extensions once deployed cannot be uninstalled. For this reason, we recommend creating a [unique environment][environment-management] for each Nebari deployment, especially when managing multiple deployments with extensions.
 :::
+
 ## Verify installation
 
 You can verify that the Nebari package is properly installed and you can execute the client commands by running:

docs/docs/get-started/quickstart.md

@@ -121,14 +121,14 @@ nebari init azure --project projectname \
 
 ## Validate (optional)
 
-After creating the `nebari-config.yaml` file, you can customize it. The Nebari package uses Pydantic for schema validation.  To ensure your customizations are valid, run:
+After creating the `nebari-config.yaml` file, you can customize it. The Nebari package uses Pydantic for schema validation. To ensure your customizations are valid, run:
 
 ```bash
 nebari validate -c nebari-config.yaml
 ```
 
 :::note
-Extensions built using the [Nebari Extension System][nebari-extension-system] may extend the Nebari schema.  If you are intending to use an extension and `nebari validate` returns an error `Extra inputs are not permitted`, ensure that the the correct versions of the extensions you intend to use are installed in your active Python environment.
+Extensions built using the [Nebari Extension System][nebari-extension-system] may extend the Nebari schema. If you are intending to use an extension and `nebari validate` returns an error `Extra inputs are not permitted`, ensure that the the correct versions of the extensions you intend to use are installed in your active Python environment.
 :::
 
 :::note
@@ -143,13 +143,12 @@ You can generate the (Terraform) deployment workflow scripts with:
 nebari render -c nebari-config.yaml
 ```
 
-This is the actual step that loads the stage classes/models and generates physical IaC files based on your Nebari config file and installed package versions.  It is not necessary to manually run a render to deploy.  However, it can be useful (especially if you use a GitOps workflow with GitHub Actions or GitLab CI/CD) to review the effects of config files changes on the resulting IaC before deploying.
+This is the actual step that loads the stage classes/models and generates physical IaC files based on your Nebari config file and installed package versions. It is not necessary to manually run a render to deploy. However, it can be useful (especially if you use a GitOps workflow with GitHub Actions or GitLab CI/CD) to review the effects of config files changes on the resulting IaC before deploying.
 
 :::note
 This command is automatically run when you `deploy`.
 :::
 
-
 ## Deploy
 
 <Tabs>

docs/docs/how-tos/access-logs-loki.md

@@ -0,0 +1,76 @@
+---
+title: Access system logs (Loki) via Grafana
+description: Access common system logs on Nebari
+---
+
+# How to access system logs (Loki) via Grafana
+
+Below is a step-by-step walkthrough of how to view JupyterHub pod logs in Grafana [Loki](https://grafana.com/docs/loki/latest/). As similar approach can be used to find other commonly accessed logs (summarized at the end of this document).
+
+To view the Loki logs via Grafana in Nebari, you'll need to have [set up monitoring](/docs/how-tos/setup-monitoring) on your deployment.
+
+## Getting Started
+
+Access the Monitoring UI for your Nebari installation at https://{your-nebari-domain}/monitoring/.
+
+:::note
+The **Explore** functionality shown below is only available to users who have `grafana_admin` permissions. In Nebari's default configuration, only the users in the `admin` Keycloak user group will have this role. `grafana_admin` is a client role in Keycloak which can be assigned to other groups or users. See [Configure Keycloak](/docs/how-tos/configuring-keycloak#in-depth-look-at-roles-and-groups) for more information.
+:::
+
+First, click "Explore".
+
+![Grafana Explore Page](/img/how-tos/1_grafana-explore.png)
+
+Select Loki as the data source at the top:
+
+![Grafana Select Loki](/img/how-tos/2_grafana-select-loki.png)
+
+Select a Label to search. For this example, we will select `pod`:
+
+![Grafana Select Loki](/img/how-tos/3_grafana-log-browser-pod.png)
+
+Begin typing the name of the desired pod. In this case we are looking
+for a JupyterHub pod. The exact name will vary on each deployment but it will begin with `hub-` followed by a unique identifier.
+
+![Grafana Select Loki](/img/how-tos/4_grafana-log-search-pod.png)
+
+Select the pod from the list of pods and then click on "Run Query":
+
+![Grafana Select Loki](/img/how-tos/5_grafana-log-select-pod.png)
+
+After clicking on "Run Query", you should be able to see logs for JupyterHub pod as shown below:
+
+![Grafana Select Loki](/img/how-tos/6_grafana-view-pod-logs.png)
+
+You can also filter by time by clicking on the time filter on top right (next to "Run query").
+
+## Common Queries
+
+The approach above can be used to access a wide variety of logs on Nebari. Below are some common queries.
+
+### JupyterLab Server Logs
+
+Each user will have their own JupyterLab instance (which may or may not be running at any given time) which will contain its own set of logs. To view the logs for a JupyterLab server, use `pod` label and begin typing the username of interest. The pod name will be `jupyter-{username}`.
+
+### Conda-Store Logs
+
+Conda-store runs multiple pods on the backend. The conda-store server runs continuously while conda-store-workers are only started when an environment is being built. Therefore, worker pods may not always exist. Generally, user requests and access logs will be in the server pod, while logs related to environment builds will be in a worker pod. In the list of conda-store pods shown below, you'll also see pods for `minio`, `postgresql`, and `redis`. These are used internally by conda-store are not likely to have any logs of interest.
+
+- `nebari-conda-store-server-[id]`
+- `nebari-conda-store-worker-[id]`
+- `nebari-conda-store-minio-[id]`
+- `nebari-conda-store-postgresql-postgresql-0`
+- `nebari-conda-store-redis-master-0`
+
+### Deployed app logs (via jhub-apps)
+
+If you have `jhub-apps` enabled on your deployment, you can view the logs to debug deployed apps.
+
+To see the logs from **all deployed apps**, use the label filter `container` = `notebook`. The main container in each deployed app pod is named `notebook`.
+
+To see **logs from a specific app**, use the `pod` label and begin typing either the name of the user running the app or the app name to find the correct pod. App pods are named with the convention `jupyter-[username]--[app_name]-[pod_id]`.
+
+## Additional Information
+
+- [Understand Log Query Structure](https://grafana.com/docs/loki/latest/query/log_queries/)
+- [Use the Query Editor](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/#choose-a-query-editing-mode)