Merge branch 'main' into develop-to-main

Author: Adam-D-Lewis

.pre-commit-config.yaml

@@ -36,7 +36,7 @@ repos:
 
   # Misc...
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.6.0
+    rev: v5.0.0
     # ref: https://github.com/pre-commit/pre-commit-hooks#hooks-available
     hooks:
       # Autoformat: Makes sure files end in a newline and only a newline

docs/docs/explanations/advanced-provider-configuration.md

@@ -98,6 +98,103 @@ amazon_web_services:
   permissions_boundary: arn:aws:iam::01234567890:policy/<permissions-boundary-policy-name>
 ```
 
+### EKS KMS ARN (Optional)
+
+You can use AWS Key Management Service (KMS) to enhance security by encrypting Kubernetes secrets in
+Amazon Elastic Kubernetes Service (EKS). This approach adds an extra layer of protection for sensitive
+information, like passwords, credentials, and TLS keys, by applying user-managed encryption keys to Kubernetes
+secrets, supporting a [defense-in-depth strategy](https://aws.amazon.com/blogs/containers/using-eks-encryption-provider-support-for-defense-in-depth/).
+
+Nebari supports setting an existing KMS key while deploying Nebari to implement encryption of secrets
+created in Nebari's EKS cluster. The KMS key must be a **Symmetric** key set to **encrypt and decrypt** data.
+
+:::warning
+Enabling EKS cluster secrets encryption, by setting `amazon_web_services.eks_kms_arn`, is an
+_irreversible_ action and re-deploying Nebari to try to remove a previously set `eks_kms_arn` will fail.
+On the other hand, if you try to change the KMS key in use for cluster encryption, by re-deploying Nebari
+after setting a _different_ key ARN, the re-deploy should succeed but the KMS key used for encryption will
+not actually change in the cluster config and the original key will remain set. The integrity of a faulty
+deployment can be restored, following a failed re-deploy attempt to remove a previously set KMS key, by
+simply re-deploying Nebari while ensuring `eks_kms_arn` is set to the original KMS key ARN.
+:::
+
+:::danger
+If the KMS key used for envelope encryption of secrets is ever deleted, then there is no way to recover
+the EKS cluster.
+:::
+
+:::note
+After enabling cluster encryption on your cluster, you must encrypt all existing secrets with the
+new key by running the following command:
+`kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - kms-encryption-timestamp="time value"`
+Consult [Encrypt K8s secrets with AWS KMS on existing clusters](https://docs.aws.amazon.com/eks/latest/userguide/enable-kms.html) for more information.
+:::
+
+Here is an example of how you would set KMS key ARN in `nebari-config.yaml`.
+
+```yaml
+amazon_web_services:
+  # the arn for the AWS Key Management Service key
+  eks_kms_arn: "arn:aws:kms:us-west-2:01234567890:key/<aws-kms-key-id>"
+```
+
+### Launch Templates (Optional)
+
+Nebari supports configuring launch templates for your node groups, enabling you to customize settings like the AMI ID and pre-bootstrap commands. This is particularly useful if you need to use a custom AMI or perform specific actions before the node joins the cluster.
+
+:::warning
+If you add a `launch_template` to an existing node group that was previously created without one, AWS will treat this as a change requiring the replacement of the entire node group. This action will trigger a reallocation of resources, effectively destroying the current node group and recreating it. This behavior is due to how AWS handles self-managed node groups versus those using launch templates with custom settings.
+:::
+
+:::tip
+To avoid unexpected downtime or data loss, consider creating a new node group with the launch template settings and migrating your workloads accordingly. This approach allows you to implement the new configuration without disrupting your existing resources.
+:::
+
+#### Configuring a Launch Template
+
+To configure a launch template for a node group in your `nebari-config.yaml`, add the `launch_template` section under the desired node group:
+
+```yaml
+amazon_web_services:
+  region: us-west-2
+  kubernetes_version: "1.18"
+  node_groups:
+    custom-node-group:
+      instance: "m5.large"
+      min_nodes: 1
+      max_nodes: 5
+      gpu: false  # Set to true if using GPU instances
+      launch_template:
+        # Replace with your custom AMI ID
+        ami_id: ami-0abcdef1234567890
+        # Command to run before the node joins the cluster
+        pre_bootstrap_command: |
+          #!/bin/bash
+          # This script is executed before the node is bootstrapped
+          # You can use this script to install additional packages or configure the node
+          # For example, to install the `htop` package, you can run:
+          # sudo apt-get update
+          # sudo apt-get install -y htop"
+```
+
+**Parameters:**
+
+- `ami_id` (Optional): The ID of the custom AMI to use for the nodes in this group; this assumes the AMI provided is an EKS-optimized AMI derivative. If specified, the `ami_type` is automatically set to `CUSTOM`.
+- `pre_bootstrap_command` (Optional): A command or script to execute on the node before
+  it joins the Kubernetes cluster. This can be used for custom setup or configuration
+  tasks. The format should be a single string in conformation with the shell syntax.
+  This command is injected in the `user_data` field of the launch template. For more
+  information, see [User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html).
+
+> If you're using a `launch_template` with a custom `ami_id`, there's an issue with updating the `scaling.desired_size` via Nebari configuration (terraform). To scale up, you must recreate the node group or adjust the scaling settings directly in the AWS Console UI (recommended). We are aware of this inconsistency and plan to address it in a future update.
+
+:::note
+If an `ami_id` is not provided, AWS will use the default Amazon Linux 2 AMI for the
+specified instance type. You can find the latest optimized AMI IDs for Amazon EKS in your
+cluster region by inspecting its respective SSM parameters. For more information, see
+[Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html).
+:::
+
 </TabItem>
 
 <TabItem value="azure" label="Azure">

docs/docs/how-tos/jhub-app-launcher.md

@@ -40,6 +40,24 @@ JHub App Launcher is was integrated into Nebari in version
 and is not enabled by default.
 :::
 
+## Overrides
+
+This integration also supports overrides, as in configuring jhub-apps via `nebari-config.yml`.
+The syntax for the same is given below:
+
+```yaml
+jhub_apps:
+  enabled: true
+  overrides:
+    # Anything that can be customized via
+    # c.JAppsConfig.<ATTRIBUTE>
+    # See https://github.com/nebari-dev/jhub-apps/blob/5ed5c9d3d1eeb08a5710001fef1e63295d7cb48d/jhub_apps/config_utils.py#L5
+    service_workers: 4
+    blocked_frameworks:
+      - jupyterlab
+      - custom
+```
+
 ## Usage
 
 Documentation on how to create apps is included in the

docs/docs/how-tos/nebari-gcp.md

@@ -66,7 +66,11 @@ management.
 
 If it's your first time creating a service account, please follow
 [these detailed instructions](https://cloud.google.com/iam/docs/creating-managing-service-accounts) to create a Google Service Account with the following roles attached:
-"roles/editor", "roles/resourcemanager.projectIamAdmin" and "roles/container.admin".
+
+- [`roles/editor`](https://cloud.google.com/iam/docs/understanding-roles#editor)
+- [`roles/resourcemanager.projectIamAdmin`](https://cloud.google.com/iam/docs/understanding-roles#resourcemanager.projectIamAdmin)
+- [`roles/container.admin`](https://cloud.google.com/iam/docs/understanding-roles#container.admin)
+- [`roles/storage.admin`](https://cloud.google.com/iam/docs/understanding-roles#storage.admin)
 
 For more information about roles and permissions, see the
 [Google Cloud Platform IAM documentation](https://cloud.google.com/iam/docs/choose-predefined-roles). Remember to check the active project before creating resources, especially if

docs/docs/how-tos/nebari-local.md

@@ -159,6 +159,20 @@ security:
         tag: sha-b4a2d1e
 ```
 
+### Increase fs watches
+
+Depending on your host system, you may need to increase the `fs.inotify.max_user_watches` and
+`fs.inotify.max_user_instances kernel parameters` if you see the error "too many open files" in the logs of
+a failing pod.
+
+```bash
+sudo sysctl fs.inotify.max_user_watches=524288
+sudo sysctl fs.inotify.max_user_instances=512
+```
+
+See the [kind troubleshooting
+docs](https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files) for more information.
+
 ## Deploying Nebari
 
 With the `nebari-config.yaml` configuration file now created, Nebari can be deployed for the first time with:

docs/docs/references/RELEASE.md

@@ -9,11 +9,64 @@ This file is copied to nebari-dev/nebari-docs using a GitHub Action. -->
 
 ---
 
-### Release 2024.7.1 - August 8, 2024
+## Release 2024.9.1 - September 27, 2024
+
+> WARNING: This release changes how group directories are mounted in JupyterLab pods: only groups with specific permissions will have their directories mounted. If you rely on custom group mounts, we strongly recommend running `nebari upgrade` before updating. This will prompt you to confirm how Nebari should handle your groups—either keep them mounted or allow unmounting. **No data will be lost**, and you can reverse this anytime.
+
+### What's Changed
+
+- Fix: KeyValueDict error when deploying to existing infrastructure by @oftheaxe in https://github.com/nebari-dev/nebari/pull/2560
+- Remove unused AWS terraform modules by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2623
+- Upgrade Hashicorp Vault action by @aktech in https://github.com/nebari-dev/nebari/pull/2616
+- Pass `oauth_no_confirm=True` to jhub-apps by @krassowski in https://github.com/nebari-dev/nebari/pull/2631
+- Use Rook Ceph for Jupyterhub and Conda Store drives by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2541
+- Fix typo in guided init by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2635
+- Action var tests off by @BrianCashProf in https://github.com/nebari-dev/nebari/pull/2632
+- add a "moved" block to account for refactored terraform code without deleting/recreating NFS disks by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2639
+- Use Helm Chart for JupyterHub 5.1.0 by @krassowski in https://github.com/nebari-dev/nebari/pull/2661
+- Add a how to test section to PR template by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2659
+- Support disallowed nebari config changes by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2660
+- Fix converted init command in guided init by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2666
+- Add initial uptime metrics by @dcmcand in https://github.com/nebari-dev/nebari/pull/2609
+- Refactor and extend Playwright tests by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2644
+- Remove Cypress remaining tests/files by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2672
+- refactor jupyterhub user token retrieval within pytest by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2645
+- add moved block to account for terraform changes on AWS only by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2673
+- Refactor shared group mounting using RBAC by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2593
+- Dashboard fix usage report by @kenafoster in https://github.com/nebari-dev/nebari/pull/2671
+- only capture stdout not stdout+stderr when capture_output=True by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2704
+- revert breaking change to azure deployment test by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2706
+- Refactor GitOps approach prompt flow in guided init by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2269
+- template the kustomization.yaml file by @dcmcand in https://github.com/nebari-dev/nebari/pull/2667
+- Fix auto-provisioned GitHub repo description after guided init by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2708
+- Add amazon_web_services configuration option to specify EKS cluster api server endpoint access setting by @joneszc in https://github.com/nebari-dev/nebari/pull/2618
+- Use Google Auth and Cloud Python APIs instead of `gcloud` CLI by @swastik959 in https://github.com/nebari-dev/nebari/pull/2083
+- fix broken links in README.md, SECURITY.md, and CONTRIBUTING.md by @blakerosenthal in https://github.com/nebari-dev/nebari/pull/2720
+- add test for changing dicts and lists by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2724
+- 2024.9.1 upgrade notes by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2726
+- Add Support for AWS Launch Template Configuration by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2668
+- Run terraform init before running terraform show by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2734
+- Release Process Checklist Updates by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2727
+- Test implicit aiohttp's TCP to HTTP connector change by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2741
+- remove comments by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2743
+- Deploy Rook Ceph Helm only when Ceph FS Needed by @kenafoster in https://github.com/nebari-dev/nebari/pull/2742
+- fix group mounting paths by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2738
+- Add compatibility prompt and notes for shared group mounting by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2739
+
+### New Contributors
+
+- @oftheaxe made their first contribution in https://github.com/nebari-dev/nebari/pull/2560
+- @joneszc made their first contribution in https://github.com/nebari-dev/nebari/pull/2618
+- @swastik959 made their first contribution in https://github.com/nebari-dev/nebari/pull/2083
+- @blakerosenthal made their first contribution in https://github.com/nebari-dev/nebari/pull/2720
+
+**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.7.1...2024.9.1
+
+## Release 2024.7.1 - August 8, 2024
 
 > NOTE: Support for Digital Ocean deployments using CLI commands and related Terraform modules is being deprecated. Although Digital Ocean will no longer be directly supported in future releases, you can still deploy to Digital Ocean infrastructure using the current `existing` deployment option.
 
-## What's Changed
+### What's Changed
 
 - Enable authentication by default in jupyter-server by @krassowski in https://github.com/nebari-dev/nebari/pull/2288
 - remove dns sleep by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2550
@@ -35,14 +88,14 @@ This file is copied to nebari-dev/nebari-docs using a GitHub Action. -->
 - Move codespell config to pyproject.toml only by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2611
 - Add `depends_on` for bucket encryption by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2615
 
-## New Contributors
+### New Contributors
 
 - @BrianCashProf made their first contribution in https://github.com/nebari-dev/nebari/pull/2569
 - @yarikoptic made their first contribution in https://github.com/nebari-dev/nebari/pull/2583
 
 **Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.6.1...2024.7.1
 
-### Release 2024.6.1 - June 26, 2024
+## Release 2024.6.1 - June 26, 2024
 
 > NOTE: This release includes an upgrade to the `kube-prometheus-stack` Helm chart, resulting in a newer version of Grafana. When upgrading your Nebari cluster, you will be prompted to have Nebari update some CRDs and delete a DaemonSet on your behalf. If you prefer, you can also run the commands yourself, which will be shown to you. If you have any custom dashboards, you'll also need to back them up by [exporting them as JSON](https://grafana.com/docs/grafana/latest/dashboards/share-dashboards-panels/#export-a-dashboard-as-json), so you can [import them](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/#import-a-dashboard) after upgrading.
 
@@ -83,9 +136,9 @@ This file is copied to nebari-dev/nebari-docs using a GitHub Action. -->
 
 **Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.5.1...2024.6.1
 
-### Release 2024.5.1 - May 13, 2024
+## Release 2024.5.1 - May 13, 2024
 
-## What's Changed
+### What's Changed
 
 - make userscheduler run on general node group by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2415
 - Upgrade to Pydantic V2 by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2348
@@ -323,7 +376,7 @@ command and follow the instructions
 - paginator for boto3 ec2 instance types by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1923
 - Update README.md -- fix typo. by @teoliphant in https://github.com/nebari-dev/nebari/pull/1925
 - Add more unit tests, add cleanup step for Digital Ocean integration test by @iameskild in https://github.com/nebari-dev/nebari/pull/1910
-- Add cleanup step for AWS integration test, ensure diable_prompt is passed through by @iameskild in https://github.com/nebari-dev/nebari/pull/1921
+- Add cleanup step for AWS integration test, ensure disable_prompt is passed through by @iameskild in https://github.com/nebari-dev/nebari/pull/1921
 - K8s 1.25 + More Improvements by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1856
 - adding lifecycle ignore to eks node group by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1905
 - nebari init unit tests by @sblair-metrostar in https://github.com/nebari-dev/nebari/pull/1931
@@ -471,7 +524,7 @@ This is a hot-fix release that resolves an issue whereby users in the `analyst`
 - improve CLI tests by @pmeier in https://github.com/nebari-dev/nebari/pull/1710
 - Fix Existing dashboards by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1723
 - Fix dashboards by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/1727
-- Typo in the conda_store key by @costrouc in https://github.com/nebari-dev/nebari/pull/1740
+- Typo in the conda-store - conda_store key by @costrouc in https://github.com/nebari-dev/nebari/pull/1740
 - use -V (upper case) for --version short form by @pmeier in https://github.com/nebari-dev/nebari/pull/1720
 - [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/1692
 - improve pytest configuration by @pmeier in https://github.com/nebari-dev/nebari/pull/1700
@@ -1312,7 +1365,7 @@ Explicit user facing changes:
 
 - `qhub deploy -c qhub-config.yaml` no longer prompts unsupported argument for `load_config_file`.
 - Minor changes on the Step-by-Step walkthrough on the docs.
-- Revamp of README.md to make it concise and highlight QHub HPC.
+- Revamp of README.md to make it concise and highlight Nebari Slurm.
 
 ### Breaking changes