Merge branch 'main' into patch-1

Author: Adam-D-Lewis

.pre-commit-config.yaml

@@ -17,7 +17,7 @@ ci:
 repos:
   # Codespell: Spell checks the code and documentation
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.2.6
+    rev: v2.3.0
     hooks:
       - id: codespell
         entry: codespell

docs/docs/community/decision-making.md

@@ -0,0 +1,50 @@
+---
+id: decision-making
+title: Decision making process
+description: Decision making workflows for the Nebari OSS project
+---
+
+Nebari's [core team][core-team] is the primary decision making authority for the project.
+
+This group makes decisions about the following non-exhaustive list of items:
+
+- Major updates to the Nebari project and community, through the Request-for-discussion (RFD) process (see explanation below)
+- Adding and removing members from the [Nebari team following the guidelines][nebari-team]
+
+## Consent-based decision making
+
+Nebari follows the [consent-based approach][consent-decision-making] for making all project-related decisions.
+
+In this approach, everyone gets a chance to:
+
+- understand the proposals by asking questions,
+- share their thoughts and reactions, and
+- present any specific objections
+
+Each objection is discussed and the proposal is updated accordingly. The proposal is accepted when the team reaches a point where there are no objections.
+
+## Major updates - Request-for-discussion (RFD) process
+
+New features, sub-projects, and workflow changes that affect a majority of end-users of the core project or the Nebari community, need to follow the Request-for-discussion workflow (sometimes called Enhancement Proposals):
+
+1. Open an [RFD-issue in the governance repository][rfd-issue] with your proposal describing the details, benefits, impact, and more as mentioned in the issue template. Add the following labels to the issue: `needs: discussion 💬`, `type: RFD 🗳`
+2. Tag the `@nebari-dev/core-team` and any specific people for questions, comments, and objections on your proposal.
+3. Answer questions and update the proposal addressing the objections. If there are major objections, the best course of action is declining the RFD, closing the issue, and opening a new RFD issue with the updated design proposal.
+4. Once all comments are addressed, request the core-team to cast votes. Team members are expected to vote "yes" with a 👍 reaction on the proposal if they have no objection. Votes can also be taken over synchronous community meetings. In this case, the decision is documented with a comment on the RFD issue. Remove the `needs: discussion 💬` label and add `status: being voted 🗳` at this stage.
+5. If over 50% of team members vote "yes", the proposal is accepted. Remove the `status: being voted 🗳` label and add `status: approved 💪🏾`. Do not close the issue yet.
+6. Once the proposal is implemented, close the RFD issue.
+
+Learn more in [Gitpod's documentation on decision making][gitpod-rfd].
+
+## Minor updates - Discussion in issues and pull requests
+
+Minor updates to the codebase and documentation can be discussed in GitHub issues or in pull requests during code review. Contributors are expected to (informally) follow the consent-based decision making philosophy in these discussions.
+
+
+<!-- Reusable links -->
+
+[nebari-team]: /docs/community/team-structure
+[core-team]: https://github.com/orgs/nebari-dev/teams/core-team
+[consent-decision-making]: https://www.sociocracyforall.org/consent-decision-making/
+[rfd-issue]: https://github.com/nebari-dev/governance/issues/new?assignees=&labels=type%3A+RFD&projects=&template=RFD.md&title=RFD+-+Title
+[gitpod-rfd]: https://gitpod.notion.site/Decision-Making-RFCs-eb4a57f3a34f40f1afbd95e05322af70

docs/docs/explanations/advanced-configuration.md

@@ -283,19 +283,20 @@ domain: demo.nebari.dev
 
 ### Continuous integration and continuous deployment
 
-Nebari uses [infrastructure-as-code](https://en.wikipedia.org/wiki/Infrastructure_as_code) to allow developers and users to request changes to the environment via pull requests (PRs) which then get approved by administrators.
-You may configure a CI/CD process to watch for pull-requests or commits on specific branches.
-Currently, CI/CD can be setup for either [GitHub Actions](https://docs.github.com/en/actions) or [GitLab CI](https://docs.gitlab.com/ee/ci/).
+Nebari uses [infrastructure-as-code](https://en.wikipedia.org/wiki/Infrastructure_as_code) to maintain a description of the deployed infrastructure in source control.  By using a git repository with CI/CD configured, teams can more quickly modify their deployment, empowering developers and data scientists to request the changes and have them approved by an administrator.
 
+When a `ci_cd` section is configured within your `nebari-config.yaml`, the first `nebari deploy` command will create all related files that describe a [CI/CD](https://about.gitlab.com/topics/ci-cd/) process.  These pipelines will then be responsible for redeploying Nebari as changes are made to a specified branch.  (Alternatively, an administrator can use `nebari render` to generate the necessary files as if running a dry-run.)  Currently, Nebari can generate CI/CD for [GitHub Actions](https://docs.github.com/en/actions) and [GitLab CI](https://docs.gitlab.com/ee/ci/).
+
+Below is an example `ci_cd` section in a `nebari-config.yaml` file.
 ```yaml
 ### Continuous integration ###
 ci_cd:
-  type: gitlab-ci
-  branch: main
-  commit_render: true
-  before_script:
+  type: gitlab-ci # 'gitlab-ci' or 'github-actions'
+  branch: main # Branch that triggers deployment
+  commit_render: true # During deployment, commit the rendered IaC back into the repository
+  before_script: # GitLab only
     - echo "running commands before ci completes"
-  after_script:
+  after_script: # GitLab only
     - echo "running commands after ci completes"
     - echo "additional commands to run"
 ```
@@ -309,8 +310,18 @@ ci_cd:
   resources. Currently only supported on `gitlab-ci`.
 - `after_script` (optional): Script to run after CI ends infrastructure deployment. This is useful in cases to notify resources of successful Nebari deployment. Currently supported on `gitlab-ci`.
 
-If `ci_cd` is not supplied, no CI/CD will be auto-generated, however, we advise employing an infrastructure-as-code approach.
-This allows teams to more quickly modify their deployment, empowering developers and data scientists to request the changes and have them approved by an administrator.
+The CI/CD workflow that is best for you will depend on your organization, but the following tenets will be appropriate for most situations.
+
+* You will want to have an upstream Git repository configured - we recommend either GitHub or GitLab since we support generating CI/CD jobs for these products.
+* The branch that triggers deployment (typically `main`, but you can set other ones in Nebari config's `ci_cd.branch`) should be protected so that only sys admins can commit or approve pull (or merge) requests into it.
+* CI/CD variables must be set in your repository so the pipeline can access your cloud (see Note below)
+* Non-admin users who have write access to the repository's non-protected branches may create their own branch off of `main`, locally make changes to the `nebari-config.yaml` and other files, and then push that branch to the origin and propose they be deployed via a Pull Request.
+* Advanced Nebari users may also want to add a step in their deployment flow that includes a `nebari render` so that the administrator may preview the resulting diffs to IaC and/or CI/CD files before `nebari deploy` is executed.
+
+
+:::note
+In order for your CI/CD pipeline to be able to deploy changes into your Nebari cloud hosting provider, you must set the appropriate authentication environment variables for your GitLab or GitHub CI/CD execution environment.  See the Authentication section for deploing to [AWS](https://www.nebari.dev/docs/how-tos/nebari-aws/#authentication), [Azure](https://www.nebari.dev/docs/how-tos/nebari-azure#authentication), [GCP](https://www.nebari.dev/docs/how-tos/nebari-gcp/#authentication), or [Digital Ocean](https://www.nebari.dev/docs/how-tos/nebari-do/#authentication) for Nebari's required variables.  Guidance on how to set these for your repository/project can be found in the documentation for [GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/variables) and [GitLab CI/CD](https://docs.gitlab.com/ee/ci/variables/).
+:::
 
 ### Certificates
 
@@ -414,4 +425,4 @@ Learn to configure more aspects of your Nebari deployment with the following top
 - [JupyterLab and Dask profile configuration](./advanced-profiles-settings.md)
 - [Customize JuputerHub theme](./advanced-custom-settings.md)
 - [Environment configuration](./advanced-env-configuration.md)
-- [Custom settings and overrides](./advanced-custom-settings.md)
+- [Custom settings and overrides](./advanced-custom-settings.md)

docs/docs/explanations/advanced-custom-settings.md

@@ -276,6 +276,7 @@ One way to achieve this is by creating a Virtual Machine (VM) inside the virtual
 Select the virtual network and subnet name under the networking settings of your cloud provider while creating the VM
 and then follow the usual deployment instructions as you would deploy from your local machine.
 
+=======
 #### Conda store worker
 
 You can use the following settings to change the defaults settings (shown) used for Conda store workers.
@@ -292,3 +293,55 @@ conda_store:
 :::note Note
 Current `conda_store.worker_resources` defaults are set at the minimum recommended resources for conda-store-workers - (conda-store [docs](https://conda.store/conda-store/references/faq#what-are-the-resource-requirements-for-conda-store-server))
 :::
+
+## Helm Extensions
+
+Nebari provides a way for any user to expand the infrastructure available by default by using the `helm_extensions` attribute. This attribute allows for the management and customization of Kubernetes applications through Helm charts. The helm_extensions is a configuration construct that specifies a list of Helm charts and their respective settings.
+
+### Overview
+
+Each entry in the `helm_extensions` list represents a single Helm chart configuration, allowing you to define the chart source, version, and specific overrides or settings for that chart. When Nebari is deployed, it will install the specified Helm charts using the provided settings.
+
+### Structure
+
+Each entry in the helm_extensions list contains the following fields:
+
+- `name`: A unique identifier for the Helm chart. It will also be used as the name of the Kubernetes deployment related resources.
+- `repository`: The URL of the repository where the Helm chart is stored. Must be a valid Helm repository URL.
+- `chart`: The name or path of the chart within the repository. must be compliant with the Helm chart naming conventions.
+- `version`: The specific version of the chart to be used.
+- `overrides`: Specific configurations to override default chart values.
+
+:::note Note
+The `overrides` field is optional. If not specified, the default values for the chart will be used.
+:::
+
+### Example
+
+Below we give an example showcasing how to install Redis using helm_extensions:
+
+```yaml
+helm_extensions:
+  - name: redis-deployment
+    repository: https://charts.bitnami.com/bitnami
+    chart: redis
+    version: 17.7.5
+    overrides:
+      architecture: standalone
+      master:
+        containerSecurityContext:
+          runAsUser: 0
+        persistence:
+          enabled: true
+          path: /redis/data
+          subPath: redis/data
+          existingClaim: <existing-claim-name-is-required>
+      replica:
+        persistence:
+          enabled: false
+        replicaCount: 0
+```
+
+:::warning Warning
+In the above example, we are assuming the current nebari kubernetes cluster already has an appropriate storage class and persistent volume claim (PVC) created. If not, you will need to create a storage class and PVC before deploying the helm chart.
+:::

docs/docs/get-started/installing-nebari.md

@@ -5,9 +5,7 @@ description: A guide to help you install Nebari for your team.
 
 # Installing Nebari
 
-<!-- TODO: Add link to advanced settings section -->
-
-This installation guide provides the basic instructions to install and deploy Nebari for the first time, and assumes you are already familiar with the [Conda](https://docs.conda.io/projects/conda/en/latest/) and [Python packaging](https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-packages) ecosystems. If you are already familiar with Nebari and would like information on advanced configuration options, skip to the advanced-settings section in this documentation.
+This installation guide provides the basic instructions to install the Nebari deployer package CLI for the first time, and assumes you are already familiar with the [Conda](https://docs.conda.io/projects/conda/en/latest/) and [Python packaging](https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-packages) ecosystems. If you are already familiar with Nebari and would like information on advanced configuration options, skip to the [Advanced Configuration][advanced-configuration] section in this documentation.
 
 :::note
 This guide focuses on installing Nebari for **cloud usage**.
@@ -23,9 +21,10 @@ Nebari heavily depends on [Terraform](https://www.terraform.io/) and Python. The
 - For more details on Terraform and its dependencies, visit the [official Terraform documentation](https://learn.hashicorp.com/tutorials/terraform/install-cli)
 - To install conda, visit the [official conda documentation](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html), or if you prefer, visit the [mamba installation documentation](https://github.com/mamba-org/mamba#installation)
 
-## How to install Nebari
+## How to install the Nebari Package
+*See [Environment Management][environment-management] for best practices for using `conda` and `pip` to control your deployment environment.*
 
-There are currently two ways to install Nebari:
+There are currently two ways to install the Nebari CLI:
 
 1. You can install Nebari directly from the Python Package Index (PyPI) using `pip`. For most common architectures and platforms (`Linux x86-64` and `macOS x86-64`), `pip` will download and install the most recent version available.
 
@@ -45,6 +44,13 @@ There are currently two ways to install Nebari:
    mamba install nebari
    ```
 
+:::note
+The version of Nebari in your `nebari-config.yaml` must match your currently installed Nebari package version; otherwise, a warning will be raised when attempting to deploy.  See [Upgrading Nebari][nebari-upgrade] for techniques for upgrading your Nebari CLI or safely updating your older config file to match your Nebari package version.
+:::
+
+:::note
+The Nebari CLI will auto-detect and then deploy any [Nebari Extensions][nebari-extension-system] that are installed in your Python environment, and extensions once deployed cannot be uninstalled.  For this reason, we recommend creating a [unique environment][environment-management] for each Nebari deployment, especially when managing multiple deployments with extensions.
+:::
 ## Verify installation
 
 You can verify that the Nebari package is properly installed and you can execute the client commands by running:
@@ -96,11 +102,15 @@ Already made your mind about deployment? Check our handy how-to-guides:
 
 <!-- Internal links -->
 
+[advanced-configuration]: /explanations/advanced-configuration.md
 [nebari-aws]: /how-tos/nebari-aws.md
 [nebari-azure]: /how-tos/nebari-azure.md
 [nebari-do]: /how-tos/nebari-do.md
+[environment-management]: /how-tos/nebari-environment-management.md
+[nebari-extension-system]: /how-tos/nebari-extension-system.md
 [nebari-gcp]: /how-tos/nebari-gcp.md
 [nebari-local]: /how-tos/nebari-local.md
 [nebari-deploy]: /get-started/deploy.mdx
 [nebari-troubleshooting]: /troubleshooting.mdx
+[nebari-upgrade]: /how-tos/nebari-upgrade.md
 [supported-cloud-providers]: /get-started/cloud-providers.mdx