Merge branch 'main' into fix_full_buffer

Author: dcmcand

.github/CODEOWNERS

@@ -0,0 +1,3 @@
+# The Nebari Maintenance Team will be the default owners for everything in the repo.
+# The Maintenance team will be assigned to review by default when someone opens a pull request.
+*       @nebari-dev/maintenance-team

RELEASE.md

@@ -9,7 +9,7 @@ This file is copied to nebari-dev/nebari-docs using a GitHub Action. -->
 
 ---
 
-## Release 2025.1.1 - January 27, 2025
+## Release 2025.2.1 - February 7, 2025
 
 > NOTE: In this release, we have updated our maximum supported Kubernetes version from
 > 1.29 to 1.31. we strongly recommend updating the Kubernetes version
@@ -18,30 +18,34 @@ This file is copied to nebari-dev/nebari-docs using a GitHub Action. -->
 > Remember that Kubernetes minor versions must be upgraded incrementally (1.29 → 1.30 →
 > 1.31).
 
-## What's Changed
-* fix bug to allow --import-plugin to work by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2864
-* Add azure kubernetes policy add-on by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2888
-* Yaml config sets by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2876
-* Add ability to list user installed plugins from the CLI by @soapy1 in https://github.com/nebari-dev/nebari/pull/2891
-* [AUTO] Update CLI doc by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/2841
-* [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/2896
-* [ENH] - Include "--attempt-fixes" flag from Nebari upgrade CLI in upgrade steps logic by @smokestacklightnin in https://github.com/nebari-dev/nebari/pull/2839
-* add authorized ip range variable for azure by @dcmcand in https://github.com/nebari-dev/nebari/pull/2880
-* Upgrade conda-store to 2024.11.2 by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2815
-* Handle default value for azure addon policy by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2905
-* Update conda-store-ui tests for updated page verbiage by @soapy1 in https://github.com/nebari-dev/nebari/pull/2911
-* Remove unintended character at the end of the TF_LOG variable by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2912
-* Update k8s max version by @dcmcand in https://github.com/nebari-dev/nebari/pull/2903
-* [ENH] - Use GitHub secrets instead of Vault by @smokestacklightnin in https://github.com/nebari-dev/nebari/pull/2889
-* adds info command text display & change the order of command display by @kernel-loophole in https://github.com/nebari-dev/nebari/pull/2916
-* [AUTO] Update CLI doc by @nebari-sensei in https://github.com/nebari-dev/nebari/pull/2921
-
-## New Contributors
-* @soapy1 made their first contribution in https://github.com/nebari-dev/nebari/pull/2891
-* @smokestacklightnin made their first contribution in https://github.com/nebari-dev/nebari/pull/2839
-* @kernel-loophole made their first contribution in https://github.com/nebari-dev/nebari/pull/2916
-
-**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.12.1...2025.1.1
+### What's Changed
+- fix bug to allow --import-plugin to work by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2864
+- Add azure kubernetes policy add-on by @viniciusdc in https://github.com/nebari-dev-nebari/pull/2888
+- Yaml config sets by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/287-
+- Add ability to list user installed plugins from the CLI by @soapy1 in https://githu-.com/nebari-dev/nebari/pull/2891
+- [ENH] - Include "--attempt-fixes" flag from Nebari upgrade CLI in upgrade steps log-c by @smokestacklightnin in https://github.com/nebari-dev/nebari/pull/2839
+- add authorized ip range variable for azure by @dcmcand in https://github.com/nebari-dev/nebari/pull/2880
+- Upgrade conda-store to 2024.11.2 by @marcelovilla in https://github.com/nebari-dev/-ebari/pull/2815
+- Handle default value for azure addon policy by @viniciusdc in https://github.com/ne-ari-dev/nebari/pull/2905
+- Update conda-store-ui tests for updated page by @soapy1 in https://github.com/nebar--dev/nebari/pull/2911
+- Remove unintended character at the end of the TF_LOG variable by @marcelovilla in h-tps://github.com/nebari-dev/nebari/pull/2912
+- Update k8s max version by @dcmcand in https://github.com/nebari-dev/nebari/pull/290-
+- [ENH] - Use GitHub secrets instead of Vault by @smokestacklightnin in https://github.com/nebari-d-v/nebari/pull/2889
+- adds info command text display & change the order of command display by @kernel-loophole in https-//github.com/nebari-dev/nebari/pull/2916
+- `2025.1.1` Upgrade step and version bump by @viniciusdc in https://github.com/nebari-dev/nebari/p-ll/2924
+- Retrieve all conda-store environments by @soapy1 in https://github.com/nebari-dev/nebari/pull/291-
+- [BUG] - Make sure to get envs when the number of envs is less than page limit by @soapy1 in https://github.com/nebar--dev/nebari/pull/2939
+- Fix Playwright CI errors & update local instructions by @viniciusdc in https://github.com/nebari-dev/nebari/pull/294-
+- Update conda-store-server image + use public auth_schema module for AuthenticationToken by @soapy1 in https://github.com/nebari-dev/nebari/pull/2931-
+
+### New Contributors
+
+- @soapy1 made their first contribution in https://github.com/nebari-dev/nebari/pull2891
+- @smokestacklightnin made their first contribution in https://github.com/nebari-dev
+/nebari/pull/2839
+- @kernel-loophole made their first contribution in https://github.com/nebari-dev/nebari/pull/2916
+
+**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2024.12.1...2025.2.1
 
 ## Release 2024.12.1 - December 13, 2024
 
@@ -96,7 +100,7 @@ This file is copied to nebari-dev/nebari-docs using a GitHub Action. -->
 
 > NOTE: This hotfix addresses several major bugs identified in the 2024.9.1 release. For a detailed overview, please refer to the related discussion at #2798. Users should upgrade directly from 2024.7.1 to 2024.11.1.
 
-## What's Changed
+### What's Changed
 
 - fix `CHECK_URL` in kuberhealthy checks to respect namespaces by @dcmcand in https://github.com/nebari-dev/nebari/pull/2779
 - fix bug where `check_immutable_fields` throws error with old version of Nebari by @Adam-D-Lewis in https://github.com/nebari-dev/nebari/pull/2796

src/_nebari/constants.py

@@ -1,4 +1,4 @@
-CURRENT_RELEASE = "2025.1.1"
+CURRENT_RELEASE = "2025.2.1"
 
 HELM_VERSION = "v3.15.3"
 KUSTOMIZE_VERSION = "5.4.3"
@@ -16,7 +16,7 @@
 DEFAULT_NEBARI_IMAGE_TAG = CURRENT_RELEASE
 DEFAULT_NEBARI_WORKFLOW_CONTROLLER_IMAGE_TAG = CURRENT_RELEASE
 
-DEFAULT_CONDA_STORE_IMAGE_TAG = "2024.11.2"
+DEFAULT_CONDA_STORE_IMAGE_TAG = "2025.2.1"
 
 LATEST_SUPPORTED_PYTHON_VERSION = "3.10"
 

src/_nebari/stages/infrastructure/template/gcp/main.tf

@@ -5,6 +5,9 @@ data "google_compute_zones" "gcpzones" {
 
 module "registry-jupyterhub" {
   source = "./modules/registry"
+
+  repository_id = "${var.name}-${var.environment}"
+  location      = var.region
 }
 
 

src/_nebari/stages/infrastructure/template/gcp/modules/registry/main.tf

@@ -1,3 +1,6 @@
-resource "google_container_registry" "registry" {
-  location = var.location
+resource "google_artifact_registry_repository" "registry" {
+  # https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/artifact_registry_repository#argument-reference
+  repository_id = var.repository_id
+  location      = var.location
+  format        = var.format
 }

src/_nebari/stages/infrastructure/template/gcp/modules/registry/variables.tf

@@ -1,6 +1,17 @@
 variable "location" {
-  # https://cloud.google.com/container-registry/docs/pushing-and-pulling#pushing_an_image_to_a_registry
+  # https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling
   description = "Location of registry"
   type        = string
-  default     = "US"
+}
+
+variable "format" {
+  # https://cloud.google.com/artifact-registry/docs/reference/rest/v1/projects.locations.repositories#Format
+  description = "The format of packages that are stored in the repository"
+  type        = string
+  default     = "DOCKER"
+}
+
+variable "repository_id" {
+  description = "Name of repository"
+  type        = string
 }

src/_nebari/stages/infrastructure/template/gcp/versions.tf

@@ -2,7 +2,7 @@ terraform {
   required_providers {
     google = {
       source  = "hashicorp/google"
-      version = "4.8.0"
+      version = "6.14.1"
     }
   }
   required_version = ">= 1.0"

src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/conda-store/config/conda_store_config.py

@@ -11,8 +11,8 @@
 
 import requests
 from conda_store_server import api
-from conda_store_server._internal import schema
 from conda_store_server._internal.server.dependencies import get_conda_store
+from conda_store_server.server import schema as auth_schema
 from conda_store_server.server.auth import GenericOAuthAuthentication
 from conda_store_server.storage import S3Storage
 
@@ -357,7 +357,7 @@ def _get_conda_store_client_roles_for_user(
         return client_roles_rich
 
     def _get_current_entity_bindings(self, username):
-        entity = schema.AuthenticationToken(
+        entity = auth_schema.AuthenticationToken(
             primary_namespace=username, role_bindings={}
         )
         self.log.info(f"entity: {entity}")
@@ -387,7 +387,7 @@ async def authenticate(self, request):
 
         # superadmin gets access to everything
         if "conda_store_superadmin" in user_data.get("roles", []):
-            return schema.AuthenticationToken(
+            return auth_schema.AuthenticationToken(
                 primary_namespace=username,
                 role_bindings={"*/*": {"admin"}},
             )
@@ -425,7 +425,7 @@ async def authenticate(self, request):
                 if _namespace is None:
                     api.ensure_namespace(db, name=namespace)
 
-        return schema.AuthenticationToken(
+        return auth_schema.AuthenticationToken(
             primary_namespace=username,
             role_bindings=role_bindings,
         )

src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/02-spawner.py

@@ -10,6 +10,9 @@
 
 from kubespawner import KubeSpawner  # noqa: E402
 
+# conda-store default page size
+DEFAULT_PAGE_SIZE_LIMIT = 100
+
 
 @gen.coroutine
 def get_username_hook(spawner):
@@ -23,25 +26,66 @@ def get_username_hook(spawner):
     )
 
 
+def get_total_records(url: str, token: str) -> int:
+    import urllib3
+
+    http = urllib3.PoolManager()
+    response = http.request("GET", url, headers={"Authorization": f"Bearer {token}"})
+    decoded_response = json.loads(response.data.decode("UTF-8"))
+    return decoded_response.get("count", 0)
+
+
+def generate_paged_urls(base_url: str, total_records: int, page_size: int) -> list[str]:
+    import math
+
+    urls = []
+    # pages starts at 1
+    for page in range(1, math.ceil(total_records / page_size) + 1):
+        urls.append(f"{base_url}?size={page_size}&page={page}")
+
+    return urls
+
+
+# TODO: this should get unit tests. Currently, since this is not a python module,
+# adding tests in a traditional sense is not possible. See https://github.com/soapy1/nebari/tree/try-unit-test-spawner
+# for a demo on one approach to adding test.
 def get_conda_store_environments(user_info: dict):
+    import os
+
     import urllib3
-    import yarl
+
+    # Check for the environment variable `CONDA_STORE_API_PAGE_SIZE_LIMIT`. Fall
+    # back to using the default page size limit if not set.
+    page_size = os.environ.get(
+        "CONDA_STORE_API_PAGE_SIZE_LIMIT", DEFAULT_PAGE_SIZE_LIMIT
+    )
 
     external_url = z2jh.get_config("custom.conda-store-service-name")
     token = z2jh.get_config("custom.conda-store-jhub-apps-token")
     endpoint = "conda-store/api/v1/environment"
 
-    url = yarl.URL(f"http://{external_url}/{endpoint}/")
-
+    base_url = f"http://{external_url}/{endpoint}/"
     http = urllib3.PoolManager()
-    response = http.request(
-        "GET", str(url), headers={"Authorization": f"Bearer {token}"}
-    )
 
-    # parse response
-    j = json.loads(response.data.decode("UTF-8"))
+    # get total number of records from the endpoint
+    total_records = get_total_records(base_url, token)
+
+    # will contain all the environment info returned from the api
+    env_data = []
+
+    # generate a list of urls to hit to build the response
+    urls = generate_paged_urls(base_url, total_records, page_size)
+
+    # get content from urls
+    for url in urls:
+        response = http.request(
+            "GET", url, headers={"Authorization": f"Bearer {token}"}
+        )
+        decoded_response = json.loads(response.data.decode("UTF-8"))
+        env_data += decoded_response.get("data", [])
+
     # Filter and return conda environments for the user
-    return [f"{env['namespace']['name']}-{env['name']}" for env in j.get("data", [])]
+    return [f"{env['namespace']['name']}-{env['name']}" for env in env_data]
 
 
 c.Spawner.pre_spawn_hook = get_username_hook

src/_nebari/stages/terraform_state/template/gcp/main.tf

@@ -24,7 +24,7 @@ terraform {
   required_providers {
     google = {
       source  = "hashicorp/google"
-      version = "4.83.0"
+      version = "6.14.1"
     }
   }
   required_version = ">= 1.0"