R2D2 v3 Adaptation (#618)

* Script to setup new r2d2 credentials

* Create a new swell task to test new r2d2

* Adapt get_observations to new R2D2

* Remove exit()

* Add r2d2 configs (#318)

* Update swell tasks to new R2D2 (#318)

* Update r2d2 version of  save obs diagnostics #318

* Remove unused files #318

* Create r2d2 file register script #318

* Add scripts for manual setup for R2D2 #318

* Clean up files (#318)

* Clean up the files (#318)

* Update Python coding norms (#318)

* Fix pycode styles

* Remove redundant lines

* Load R2D2 credentials under TaskBase (#318)

* Load credentials under create R2D2 config (#318)

* make R2D2 host/compiler detection support dynamic (#318)

* Add docs for credential setup (#318)

* Update r2d2_config for cascade (#318)

* Move credentials under create_task (#318)

* Move scripts under utilities (#318)

* Fix pylint errors

* Fix AttributeError when fetching bias correction files (#318)

* Fix bias correction arguments (#318)

* Fix bias correction argument (#318)

* Add file type argument (#318)

* Fix bias correction ingest

* Improve file extension support

* Add logging

* Fix bias coefficient ingestion

* Go back to existing bias naming convention

* Use JCSDA enums for bias files

* Improve logging

* Fix code style

* extend exclude list to ignore venv and build directories

* Change the script name register_files with ingest_files

* Fix logger

* Fix string literal

* Moved readme to docs

* Fix pycode style

* Remove redundant lines

* Make the comment more generic

* Add readme for ingest_files.py (#318)

* Update src/swell/deployment/platforms/nccs_discover_sles15/r2d2_config.yaml

Co-authored-by: Doruk Ardağ <38666458+Dooruk@users.noreply.github.com>

* Remove redundant loging

Co-authored-by: Doruk Ardağ <38666458+Dooruk@users.noreply.github.com>

* Remove redundant logging

* Remove unnecessary statements

* Bump version to 1.11.0

* Bump version to 1.20.0

* Add how to register experiments to readme

* Upgrade r2d2-client (#618)

* Add register experiment to readme

* r2d2 v3 turnoffrestart (#653)

---------

Co-authored-by: Doruk Ardağ <38666458+Dooruk@users.noreply.github.com>

Author: ftgoktas

.gitignore

@@ -136,3 +136,8 @@ dmypy.json
 # Pyre type checker
 .pyre/
 *.egg-info
+
+# R2D2 credentials
+r2d2_credentials.yaml
+**/r2d2_credentials.yaml
+.swell/r2d2_credentials.yaml

docs/configs/r2d2_v3_credentials.md

@@ -0,0 +1,100 @@
+# R2D2 v3 Credentials Configuration
+
+This document explains how to configure R2D2 v3 credentials for SWELL workflows.
+
+## Overview
+
+SWELL now uses R2D2 v3 for metadata-driven data storage and retrieval. R2D2 v3 requires authentication credentials to access the centralized API. SWELL automatically loads these credentials from a YAML configuration file.
+
+## Quick Setup
+
+1. **Create the credentials directory:**
+   ```bash
+   mkdir -p ~/.swell
+   ```
+
+2. **Create the credentials file:**
+   ```bash
+   cp /path/to/swell/r2d2_credentials.yaml ~/.swell/r2d2_credentials.yaml
+   ```
+
+3. **Edit with your credentials:**
+   ```bash
+   vim ~/.swell/r2d2_credentials.yaml
+   ```
+
+4. **Set secure permissions:**
+   ```bash
+   chmod 600 ~/.swell/r2d2_credentials.yaml
+   ```
+
+## Credentials File Format
+
+Create `~/.swell/r2d2_credentials.yaml` with the following structure:
+
+```yaml
+# R2D2 v3 credentials file
+# Save this as ~/.swell/r2d2_credentials.yaml
+# Set permissions: chmod 600 ~/.swell/r2d2_credentials.yaml
+
+# Required credentials
+user: your_username              # Your R2D2 username
+api_key: your_api_key            # Your R2D2 API key
+
+# Platform-specific values (automatically determined by SWELL with an option to use YAML-first)
+# host: discover-gmao            # Automatically set based on platform
+# compiler: intel                # Automatically set based on platform
+
+```
+
+## Required Fields
+
+| Field | Description | Example |
+|-------|-------------|---------|
+| `user` | Your R2D2 username | `jdoe` |
+| `api_key` | Your R2D2 API authentication key | `abcd1234-ef56-7890-abcd-1234567890ab` |
+
+## Platform-Specific Fields (Automatically Set)
+
+| Field | Description | NCCS Discover Value |
+|-------|-------------|---------------------|
+| `host` | Compute host identifier | `discover-gmao` |
+| `compiler` | Compiler type used | `intel` |
+
+**Important**: `host` and `compiler` are automatically determined by SWELL based on your platform configuration. You can also set these manually in your credentials file.
+
+### Loading Precedence
+
+The credential loading follows this priority order:
+
+1. **Environment Variables** (highest priority)
+2. **YAML Configuration File** 
+3. **Platform Detection** (for host/compiler only)
+
+**For host and compiler specifically:**
+- YAML `host`/`compiler` values override platform detection
+- Platform detection is used as fallback when not specified in YAML
+
+### Platform-Specific Configuration
+
+SWELL automatically determines `host` and `compiler` based on your platform:
+
+| Platform | R2D2 Host | R2D2 Compiler | Notes |
+|----------|-----------|---------------|-------|
+| `nccs_discover_sles15` | `discover-gmao` | `intel` | NCCS Discover SLES15 |
+| `nccs_discover_cascade` | `discover-gmao` | `intel` | NCCS Discover Cascade |
+<!-- | `aws` | `aws-gmao` | `intel` | AWS cloud platform |
+| `generic` | `None` | `None` | Fallback to YAML/env vars | -->
+
+
+## Environment Variables Set
+
+When loaded, the following environment variables are set:
+
+- `R2D2_USER`: Your R2D2 username
+- `R2D2_API_KEY`: Your R2D2 API key  
+- `R2D2_HOST`: Compute host name
+- `R2D2_COMPILER`: Compiler type
+<!-- - `R2D2_SERVER_HOST`: (Optional) API server override
+- `R2D2_SERVER_PORT`: (Optional) API server port override -->
+

pycodestyle.cfg

@@ -8,4 +8,5 @@
 max-line-length = 100
 indent-size = 4
 statistics = True
-exclude = ._*, build, .venv
+exclude = ._*, build, .venv, venv, .git, __pycache__, .tox, .eggs, *.egg-info, dist
+

r2d2_credentials.yaml

@@ -0,0 +1,13 @@
+# R2D2 v3 credentials file
+# Save this as ~/.swell/r2d2_credentials.yaml
+# Set permissions: chmod 600 ~/.swell/r2d2_credentials.yaml
+
+# R2D2 v3 API credentials
+user: your_username
+api_key: your_api_key_here
+
+# Platform configuration  
+host: discover-gmao
+compiler: intel
+
+

src/swell/__init__.py

@@ -9,4 +9,4 @@
 repo_directory = os.path.dirname(__file__)
 
 # Set the version for swell
-__version__ = '1.10.0'
+__version__ = '1.20.0'

src/swell/deployment/platforms/nccs_discover_cascade/modules

@@ -37,9 +37,7 @@ module load other/mepo
 # Load r2d2 modules
 # -----------------
 module use -a /discover/nobackup/projects/gmao/advda/JediOpt/modulefiles/core
-module load solo/sles15_skylab9
-module load py-boto3
-module load r2d2/sles15_spack19
+module load r2d2-client/112025
 
 # Load eva and jedi_bundle
 # ------------------------

src/swell/deployment/platforms/nccs_discover_cascade/r2d2_config.yaml

@@ -1,22 +1,21 @@
 databases:
-
   ${USER}:
     class: LocalDB
     root: {{r2d2_local_path}}
     cache_fetch: false
 
-  gmao-shared:
-    class: LocalDB
-    root: /discover/nobackup/projects/gmao/advda/R2D2DataStore/Shared
-    cache_fetch: false
+# v3 API configuration (for centralized nccs-gmao data)
+data_hub: nccs-gmao
+data_store: r2d2-experiments-nccs-gmao
+compute_host: discover-gmao-intel
 
-# when fetching data, in which order should the databases accessed?
+# Fetch order: try local first, then API
 fetch_order:
   - ${USER}
-  - gmao-shared
 
-# when storing data, in which order should the databases accessed?
-store_order:
-  - ${USER}
+# Optional: credentials file for v3 API authentication
+# credentials_file: ~/.swell/swell-r2d2-credentials.yaml
 
+# Cache settings
 cache_name: ${USER}
+cache_fetch: false

src/swell/deployment/platforms/nccs_discover_sles15/modules

@@ -37,15 +37,16 @@ module load other/mepo
 # Load r2d2 modules
 # -----------------
 module use -a /discover/nobackup/projects/gmao/advda/JediOpt/modulefiles/core
-module load solo/sles15_skylab9
-module load py-boto3
-module load r2d2/sles15_spack19
+module load r2d2-client/112025
 
 # Load eva and jedi_bundle
 # ------------------------
 module load eva/sles15_milan_1.6.5
 module load jedi_bundle/sles15_skylab9
 
+# Remove any r2d2 v2 paths from PYTHONPATH to prevent conflicts
+export PYTHONPATH=$(echo $PYTHONPATH | tr ':' '\n' | grep -v "/r2d2/sles15_spack19/" | tr '\n' ':' | sed 's/:$//')
+
 # Set the swell paths
 # -------------------
 PATH={{swell_bin_path}}:$PATH
@@ -57,3 +58,4 @@ ulimit -S -s unlimited
 ulimit -S -v unlimited
 umask 022
 
+

src/swell/deployment/platforms/nccs_discover_sles15/r2d2_config.yaml

@@ -1,22 +1,21 @@
 databases:
-
   ${USER}:
     class: LocalDB
     root: {{r2d2_local_path}}
     cache_fetch: false
 
-  gmao-shared:
-    class: LocalDB
-    root: /discover/nobackup/projects/gmao/advda/R2D2DataStore/Shared
-    cache_fetch: false
+# v3 API configuration (for centralized nccs-gmao data)
+data_hub: nccs-gmao
+data_store: r2d2-experiments-nccs-gmao
+compute_host: discover-gmao-intel
 
-# when fetching data, in which order should the databases accessed?
+# Fetch order: try local first, then API
 fetch_order:
   - ${USER}
-  - gmao-shared
 
-# when storing data, in which order should the databases accessed?
-store_order:
-  - ${USER}
+# Optional: credentials file for v3 API authentication
+# credentials_file: ~/.swell/swell-r2d2-credentials.yaml
 
+# Cache settings
 cache_name: ${USER}
+cache_fetch: false

src/swell/suites/3dfgat_cycle/flow.cylc

@@ -77,12 +77,8 @@
             LinkGeosOutput-{{model_component}} => GenerateBClimatology-{{model_component}}
 
             # Data assimilation preperation
-            GetObservations-{{model_component}}
-            GenerateBClimatologyByLinking-{{model_component}} :fail? => GenerateBClimatology-{{model_component}}
-
-            LinkGeosOutput-{{model_component}} => RunJediFgatExecutable-{{model_component}}
             StageJediCycle-{{model_component}} => RunJediFgatExecutable-{{model_component}}
-            GenerateBClimatologyByLinking-{{model_component}}? | GenerateBClimatology-{{model_component}} => RunJediFgatExecutable-{{model_component}}
+            GenerateBClimatology-{{model_component}} => RunJediFgatExecutable-{{model_component}}
             GetObservations-{{model_component}} => RunJediFgatExecutable-{{model_component}}
 
             # Run analysis diagnostics
@@ -94,14 +90,17 @@
             RunJediFgatExecutable-{{model_component}} => EvaIncrement-{{model_component}}
             {% if 'cice6' in models[model_component]["marine_models"] %}
             PrepareAnalysis-{{model_component}} => RunJediConvertStateSoca2ciceExecutable-{{model_component}}
-            RunJediConvertStateSoca2ciceExecutable-{{model_component}} => SaveRestart-{{model_component}}
+            # RunJediConvertStateSoca2ciceExecutable-{{model_component}} => SaveRestart-{{model_component}}
+            RunJediConvertStateSoca2ciceExecutable-{{model_component}} => MoveDaRestart-{{model_component}}
             RunJediConvertStateSoca2ciceExecutable-{{model_component}} => CleanCycle-{{model_component}}
             {% else %}
-            PrepareAnalysis-{{model_component}} => SaveRestart-{{model_component}}
+            # PrepareAnalysis-{{model_component}} => SaveRestart-{{model_component}}
+            PrepareAnalysis-{{model_component}} => MoveDaRestart-{{model_component}}
             {% endif %}
 
+            # Temporarily disable saving restarts
             # Move restart to next cycle
-            SaveRestart-{{model_component}} => MoveDaRestart-{{model_component}}
+            # SaveRestart-{{model_component}} => MoveDaRestart-{{model_component}}
 
             # Save analysis output
             # RunJediFgatExecutable-{{model_component}} => SaveAnalysis-{{model_component}}
@@ -192,9 +191,6 @@
     [[MoveDaRestart-{{model_component}}]]
         script = "swell task MoveDaRestart $config -d $datetime -m {{model_component}}"
 
-    [[SaveRestart-{{model_component}}]]
-        script = "swell task SaveRestart $config -d $datetime -m {{model_component}}"
-
     [[StageJedi-{{model_component}}]]
         script = "swell task StageJedi $config -m {{model_component}}"
 
@@ -213,9 +209,6 @@
             --{{key}} = {{value}}
         {%- endfor %}
 
-    [[GenerateBClimatologyByLinking-{{model_component}}]]
-        script = "swell task GenerateBClimatologyByLinking $config -d $datetime -m {{model_component}}"
-
     {% if 'cice6' in models["geos_marine"]["marine_models"] %}
 
     [[RunJediConvertStateSoca2ciceExecutable-{{model_component}}]]
@@ -253,8 +246,8 @@
             --{{key}} = {{value}}
         {%- endfor %}
 
-    [[SaveRestart-{{model_component}}]]
-        script = "swell task SaveRestart $config -d $datetime -m {{model_component}}"
+    # [[SaveRestart-{{model_component}}]]
+    #     script = "swell task SaveRestart $config -d $datetime -m {{model_component}}"
 
     [[SaveObsDiags-{{model_component}}]]
         script = "swell task SaveObsDiags $config -d $datetime -m {{model_component}}"