Multiple fixes, save_restart, abandon sles12, another example in the documentation (#516)

* dataatm fix

* update EVA plots for sea-ice increments

* background filters for insitu obs

* fix for missing output obs and EVA to be executed on compute nodes

* fix for history output shenanigans

* messy task

* save restart and increments to R2D2 folder

* don't use _rst for forecast begin time

* take out sles12 instruction

* add new example for 3dfgat_cycle

* special handling for mom6_increment

* eva fixes for sea ice domain

* bye bye sles12

* code_tests related fixes

* fix unused variables

* use spack v1.9 for py_lmod

* fix uv instructions

* save diag outputs while cycling

* fix sea-ice increments

* update docu file

* None perhost fix

Author: Dooruk

docs/_media/20210702T060000Z_inc_Temp.png

@@ -7,12 +7,10 @@
   - [Monitoring an experiment](monitoring_an_experiment.md)
   - [Manually running tasks](manually_running_tasks.md)
 
-
 - Platform specific instructions
 
   - [Platforms](platforms/platforms.md)
   - **Discover**
-    - [Installing Swell on SLES12](platforms/discover/installing_swell_discover_sles12.md)
     - [Installing Swell on SLES15](platforms/discover/installing_swell_discover_sles15.md)
     - [Installing Swell using `uv` and `venv`](platforms/discover/installing_swell_uv_venv.md)
     - [Configuring `cylc`](platforms/discover/configuring_cylc_discover.md)
@@ -22,6 +20,7 @@
   - [Description](examples/description.md)
   - **SOCA**
     -  [3DVAR](examples/soca/3dvar.md)
+    -  [3DFGAT_cycle](examples/soca/3dfgat_cycle.md)
 
 - Configuration files in swell
 
@@ -30,19 +29,13 @@
   - Model configuration:
     - [CICE6](configs/model_configurations/cice6.md)
     - [MOM6](configs/model_configurations/mom6.md)
-    - [MOM6](configs/model_configurations/history_outputs.md)
-
-- Tasks
-
-  - [Task infrastructure](task_infrastructure.md)
-  - [Tasks](tasks.md)
+    - [History Outputs](configs/model_configurations/history_outputs.md)
 
 - Code test before pull request
 
   - [Code Tests](code_tests/code_tests.md)
   - [Suite Tests](code_tests/suite_tests.md)
 
-
 - Governance
 
   - [Requirements](requirements.md)

docs/_media/20210702T060000Z_inc_aice_h.png

@@ -2,10 +2,11 @@
 
 Here are a few steps the CI test will run online after your PR, but it can be easier if you run on your terminal before hand.
 
-- Python coding norms: run `python pycodestyle_run.py` at your swell root directory and resolve potential code style issues
+- Python coding norms: run `python pycodestyle_run.py` at your swell root directory and resolve potential code style issues.
 
-- Code test:  run `swell test code_tests`.
-  - The `swell/test/code_tests/code_tests.py` will test unused variables.
-  - If you get `assert tq_dicts_rc == 0; AssertionError`, that means your `tasks/task_questions.yaml` source code needs to be updated with the regenerated yaml file, e.g., named `/tmp/task_questions_RKznhVXN.yaml` (see NOTE).  If you do not see the print-out information following the error code, check
-Line 28 of `swell/test/code_tests/code_tests.py`, which may read `os.environ["LOG_INFO"] = "0"  # Set this to 1 when errors are being debugged `.  Set its value to `1`, rebuild swell, and run again `swell test code_tests`.
+- Code test: after loading SWELL as a module, run `swell test code_tests` command. The `swell/test/code_tests/code_tests.py` will test multiple code tests, which are subject to change over time.
+  Say you received `assert tq_dicts_rc == 0; AssertionError`, that means your `tasks/task_questions.yaml` source code needs to be updated with the regenerated yaml file, e.g., named `/tmp/task_questions_RKznhVXN.yaml` (see NOTE).
+
+  If you do not see the print-out information following the error code this could be due to `LOGLEVEL` is set to `WARNING` by default. See
+line 32 of `swell/test/code_tests/code_tests.py`, which reads `os.environ.setdefault("LOGLEVEL", "WARNING")`. To debug, set the `LOGLEVEL` environment value to `DEBUG` and run `swell test code_tests` again, this time more details will be provided regarding the failed tests.
     - NOTE: `/tmp` here will be different depending on your environment, following the semantics of Python's [`tempfile.gettempdir()`](https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir)

docs/_media/20210702T060000Z_increment_n20.png

@@ -16,15 +16,15 @@ background:
   ocn_filename: MOM6.res.20210702T030000Z.nc
   read_from_file: 1
   state variables:
-  - cicen
-  - hicen
-  - hsnon
-  - socn
-  - tocn
-  - ssh
-  - hocn
-  - mld
-  - layer_depth
+  - sea_ice_area_fraction
+  - sea_ice_thickness
+  - sea_ice_snow_thickness
+  - sea_water_salinity
+  - sea_water_potential_temperature
+  - sea_surface_height_above_geoid
+  - sea_water_cell_thickness
+  - ocean_mixed_layer_thickness
+  - sea_water_depth
 ```
 
 and ocean and sea-ice states:

docs/_sidebar.md

@@ -1,8 +1,67 @@
-# MOM6 Settings for GEOS/SOCA Setup (GEOSv11.6)
+# MOM6 Settings for GEOS/SOCA Setup (GEOSgcm v11.6.0)
 
 
-## History Outputs
+## History Outputs (`diag_table`)
 
 >In SWELL, history outputs are utilized to obtain the aggregated variables in desired output frequencies. See more details about [history outputs](history_outputs.md).
 
-TODO: `diag_table`
+`diag_table` for SOCA is defined as below, which are the prognostic and diagnostic ocean fields required for SOCA applications to work. This particular setup spits out 3-hourly history outputs but users can experiment with different values.
+
+```fortran
+ocean_da
+1970 1 1 0 0 0
+
+##############################################
+ "his%4yr%2mo%2dy%2hr",      3,  "hours", 1, "hours", "time", 3, "hours", "1901 1 1 0 0 0"
+##############################################
+
+# "module_name","field_name","output_name","file_name","time_sampling","reduction_method","regional_section",  packing
+#===============================================
+ "ocean_model", "geolon",      "geolon",      "his%4yr%2mo%2dy%2hr", "all", "none", "none", 2
+ "ocean_model", "geolat",      "geolat",      "his%4yr%2mo%2dy%2hr", "all", "none", "none", 2
+#===============================================
+ "ocean_model", "SSH",       "ave_ssh",      "his%4yr%2mo%2dy%2hr","all", "none","none",2
+ "ocean_model", "MLD_0125",  "MLD",      "his%4yr%2mo%2dy%2hr","all",none,"none",2
+#===============================================
+ "ocean_model","u","u"            ,"his%4yr%2mo%2dy%2hr","all",.false.,"none",2
+ "ocean_model","v","v"            ,"his%4yr%2mo%2dy%2hr","all",.false.,"none",2
+ "ocean_model","salt","Salt"      ,"his%4yr%2mo%2dy%2hr","all",.false.,"none",2
+ "ocean_model","temp","Temp"      ,"his%4yr%2mo%2dy%2hr","all",.false.,"none",2
+ "ocean_model","h","h"            ,"his%4yr%2mo%2dy%2hr","all",.false.,"none",2
+#===============================================
+```
+
+## Incremental Analysis Update (IAU) with `MOM_oda_incupd`
+
+`MOM_oda_incupd` is a module that stands for Ocean Data Assimilation (DA) incremental update. This is similar to other
+modules that are located in `MOM_input` and currently handled as an extra file in the experiment directory. It divides increment files into smaller, linear segments. For example, if the model step is 900 seconds and `ODA_INCUPD_NHOURS` is set to 3 hours, the increments will be divided by 12 and applied to the state over 12 time steps.
+
+In GEOS and SWELL context it needs to be included in the GEOS experiment folder. For it to be active,  `ODA_INCUPD` should ne set as `True`. `ODA_INCUPD_FILE` name shouldn't be changed as that is also used in SWELL. `ODA_INCUPD_UV` is set to false for
+now until horizontal surface current fields are assimilated. In terms of `ODA_INCUPD_NHOURS`, there is no set rule but we aim
+for half DA window length.
+
+```fortran
+! === module MOM_oda_incupd ===
+ODA_INCUPD = True               ! [Boolean] default = False
+                                ! If true, oda incremental updates will be applied
+                                ! everywhere in the domain.
+ODA_INCUPD_FILE = "mom6_increment.nc"   ! The name of the file with the T,S,h increments.
+
+ODA_TEMPINC_VAR = "Temp"        ! default = "ptemp_inc"
+                                ! The name of the potential temperature inc. variable in
+				      	        ! ODA_INCUPD_FILE.
+ODA_SALTINC_VAR = "Salt"        ! default = "sal_inc"
+                                ! The name of the salinity inc. variable in
+                                ! ODA_INCUPD_FILE.
+ODA_THK_VAR = "h"               ! default = "h"
+                                ! The name of the int. depth inc. variable in
+                                ! ODA_INCUPD_FILE.
+ODA_INCUPD_UV = false           !
+ODA_UINC_VAR = "u"              ! default = "u_inc"
+                                ! The name of the zonal vel. inc. variable in
+                                ! ODA_INCUPD_UV_FILE.
+ODA_VINC_VAR = "v"              ! default = "v_inc"
+                                ! The name of the meridional vel. inc. variable in
+                                ! ODA_INCUPD_UV_FILE.
+ODA_INCUPD_NHOURS = 3.0         ! default=3.0
+```

docs/code_tests/code_tests.md

@@ -23,7 +23,7 @@ sbatch ... --account=x1234 --nodes=1 --qos=allqueues --no-requeue
 
 Note that the slurm directives such as `--no-requeue` take no arguments, so it was set to `''`. This is a Cylc reqirement for their SLURM directives handling. More details on this can be seen in their documentation [here](https://cylc.github.io/cylc-doc/stable/html/user-guide/task-implementation/job-submission.html#directives-section-quirks-pbs-sge).
 
-Since there are a few SLURM directives that are platform specific, they are stored under `deployment/platforms/{platform_name}/slurm.yaml`. For instance, requested nodes are `constraint: cas` for `nccs_discover` and `constraint: mil` for `nccs_discover_sles15`.
+Since there are a few SLURM directives that are platform specific, they are stored under `deployment/platforms/{platform_name}/slurm.yaml`. For instance, requested nodes are `constraint: cas` for `nccs_discover_cascade` and `constraint: mil` for `nccs_discover_sles15`.
 
 All `sbatch` directives are supported (see [`man sbatch`](https://slurm.schedmd.com/sbatch.html)).
 However, note that SWELL will only validate that a given directive exists; we do not validate data types, or do anything fancy with type conversion (e.g., concatenation of arrays).