chore: update climate-ref components to v0.18.1

Author: lewisjared

changelog/60.improvement.md

@@ -0,0 +1,3 @@
+Updates the climate-ref components to v0.18.1.
+This ingests obs4REF data under its own `obs4ref` source type,
+and runs micromamba without file locks inside the workers, so their `/ref/cache` can stay read-only.

docker/docker-compose.yaml

@@ -42,7 +42,7 @@ services:
   # One-shot service to initialise the shared /ref volume.
   # Creates subdirectories and populates providers.
   ref-init:
-    image: ghcr.io/climate-ref/climate-ref:v0.17.2
+    image: ghcr.io/climate-ref/climate-ref:v0.18.1
     platform: linux/amd64
     depends_on:
       - postgres
@@ -99,7 +99,7 @@ services:
 
   # Base worker to track the executions of async tasks
   climate-ref:
-    image: ghcr.io/climate-ref/climate-ref:v0.17.2
+    image: ghcr.io/climate-ref/climate-ref:v0.18.1
     platform: linux/amd64
     restart: always
     depends_on:

docs/runbooks/bootstrap-a-deployment.md

@@ -193,7 +193,7 @@ To exercise the deployment before any archive is mounted, fetch the sample data:
 ```bash
 ref-orch ref datasets fetch-data --registry sample-data --output-directory /ref/data/sample
 ref-orch ref datasets ingest --source-type cmip6 /ref/data/sample/CMIP6
-ref-orch ref datasets ingest --source-type obs4mips /ref/data/sample/obs4REF
+ref-orch ref datasets ingest --source-type obs4ref /ref/data/sample/obs4REF
 ```
 
 The sample data does not contain the datasets required by ilamb.

helm/Chart.yaml

@@ -10,7 +10,7 @@ kubeVersion: ">=1.23.0-0"
 version: 0.6.4
 
 # Application version tracks the climate-ref core version
-appVersion: "0.17.2"
+appVersion: "0.18.1"
 
 dependencies:
 - name: dragonfly

helm/README.md

@@ -133,7 +133,6 @@ and is consitent across all pods.
 | `/ref/software`    | RO  | RO               | RW           | —           |
 | `/ref/scratch`     | —   | RW               | RW           | —           |
 | `/ref/cache`       | —   | RO               | RW           | —           |
-| `/ref/cache/mamba` | —   | RW               | RW           | —           |
 | `/ref/results`     | RO  | —                | RW           | —           |
 | `/ref/db`          | RW  | RW               | RW           | RW          |
 | `/ref/log`         | —   | RW               | RW           | RW          |
@@ -146,9 +145,10 @@ The split follows from how the work is dispatched:
   That queue carries `handle_result`, which copies each execution from scratch into results.
 - `/ref/scratch` must stay a **shared** volume, not a per-pod `emptyDir`.
   The worker writes the outputs and the orchestrator reads them back out from a different pod, so a per-pod scratch loses every result.
-- `/ref/cache` holds the fetched reference data, which the workers only read.
-  The image sets `XDG_CACHE_HOME` to it, and micromamba takes a lock under `mamba/` there on every `run`,
-  so that one directory must be writable or every execution fails before the diagnostic starts.
+- `/ref/cache` holds the fetched reference data and micromamba's `mamba/proc` directory.
+  `providers setup` on the orchestrator writes both, and the workers only read them.
+  A worker runs micromamba without file locks (climate-ref 0.18.1 and later),
+  but it still needs `mamba/proc` to exist, so run `providers setup` before the first solve.
 - `/ref/log` is written by every Celery worker, not just the orchestrator.
   A worker opens a log file there as it starts, so a read-only `/ref/log` stops the worker before it consumes anything.
 - `/ref/db` only matters with the default SQLite database.
@@ -310,7 +310,7 @@ and carries the same `PriorityClass` prerequisite as the API.
 | `defaults.replicaCount`                  | Number of worker replicas                                  | `1`                               |
 | `defaults.concurrency`                   | Celery child processes per pod                             | `1`                               |
 | `defaults.image.repository`              | Worker image repository                                    | `ghcr.io/climate-ref/climate-ref` |
-| `defaults.image.tag`                     | Worker image tag                                           | `v0.17.2`                         |
+| `defaults.image.tag`                     | Worker image tag                                           | `v0.18.1`                         |
 | `defaults.image.pullPolicy`              | Image pull policy                                          | `IfNotPresent`                    |
 | `defaults.resources`                     | Resource requests/limits                                   | 4 CPU / 16Gi, limits 6 CPU / 32Gi |
 | `defaults.strategy`                      | Deployment update strategy                                 | `type: Recreate`                  |

helm/ci/gh-actions-values.yaml

@@ -60,10 +60,6 @@ defaults:
     mountPath: /ref/scratch
     subPath: scratch
     readOnly: false
-  - name: gha-cached
-    mountPath: /ref/cache/mamba
-    subPath: cache/mamba
-    readOnly: false
   - name: gha-cached
     mountPath: /ref/log
     subPath: log

helm/examples/small-values.yaml

@@ -46,18 +46,14 @@ defaults:
       cpu: 2
       memory: 8Gi
   volumes: *refVolumes
-  # Workers read the config and conda environments, and write scratch, the log and the SQLite db.
-  # micromamba locks under $XDG_CACHE_HOME/mamba on every run, so that one directory is writable too.
+  # Workers read the config, the conda environments and the cache, and write scratch, the log and the SQLite db.
   volumeMounts:
   - name: ref
     mountPath: /ref
     readOnly: true
   - name: ref
     mountPath: /ref/scratch
     subPath: scratch
-  - name: ref
-    mountPath: /ref/cache/mamba
-    subPath: cache/mamba
   - name: ref
     mountPath: /ref/log
     subPath: log

helm/values.yaml

@@ -314,7 +314,7 @@ defaults:
   image:
     repository: ghcr.io/climate-ref/climate-ref
     pullPolicy: IfNotPresent
-    tag: v0.17.2
+    tag: v0.18.1
 
   annotations: {}
 

pyproject.toml

@@ -20,13 +20,13 @@ classifiers = [
 # We use exact versions here to ensure that the AFT tests run against the same combination of components that will be deployed in production.
 # Also update versions.toml when updating these versions to keep the changelog in sync
 dependencies = [
-    "climate-ref==0.17.2",
-    "climate-ref-core==0.17.2",
-    "climate-ref-celery==0.17.2",
+    "climate-ref==0.18.1",
+    "climate-ref-core==0.18.1",
+    "climate-ref-celery==0.18.1",
     # Diagnostic providers
-    "climate-ref-esmvaltool==0.17.2",
-    "climate-ref-pmp==0.17.2",
-    "climate-ref-ilamb==0.17.2",
+    "climate-ref-esmvaltool==0.18.1",
+    "climate-ref-pmp==0.18.1",
+    "climate-ref-ilamb==0.18.1",
 ]
 
 [project.optional-dependencies]

tests/test_helm_render.py

@@ -787,8 +787,8 @@ def test_orchestrator_and_migrate_job_inherit_the_default_ref_mount(values):
     assert "/ref" in _mount_paths(find(docs, "Job", "-migrate"))
 
 
-def test_small_example_gives_workers_only_the_mamba_lock_directory():
-    # micromamba locks under $XDG_CACHE_HOME/mamba on every run, and the rest of the cache stays read-only.
+def test_small_example_gives_workers_a_read_only_cache():
+    # The orchestrator writes the cache during providers setup, and the workers only read it.
     docs = render(values="helm/examples/small-values.yaml")
     for component in ("esmvaltool", "pmp", "ilamb"):
         mounts = {
@@ -798,8 +798,7 @@ def test_small_example_gives_workers_only_the_mamba_lock_directory():
             ]
         }
         assert mounts["/ref"].get("readOnly") is True
-        assert "/ref/cache" not in mounts
-        assert mounts["/ref/cache/mamba"].get("readOnly") is not True
+        assert not any(path.startswith("/ref/cache") for path in mounts)
 
 
 def _scaled_components(docs: list[dict]) -> set[str]: