Merge PR #389 (Use separate species database files for Ref & Dev)

This merge brings PR #389 (Modify benchmark scripts to look for
species_database.yml files in Ref and Dev rundirs, by @yantosca) into
the GCPy 1.7.0 development stream

In PR #389, we have done the following:

1. Removed the paths:spcdb_dir YAML tag in *benchmark.yml config files.

2. Added function get_species_database_files in
   gcpy/benchmark/modules/benchmark_utils.py, which returns
   the absolute paths to species_database.yml files in Ref and Dev.

3. Added function read_species_metadata to gcpy/util.py,
   which accepts either a single file path to species_database.yml
   or a list of file paths to the species_database.yml files in
   the Ref & Dev rundirs) and returns the Ref and Dev species
   database dicts. If only one file path is passed, the same species
   database will be returned for both Ref and Dev.

4. Replaced the spcdb_dir keyword argument in several functions with
   spcdb_files, which can be of type str or list.

5. Modified gcpy/plot/compare_single_level.py and
   gcpy/plot/compare_zonal_mean.py to accept spcdb_files instead of
   spcdb_dir. Added corresponding logic so that the Ref species
   database is used with Ref data and the Dev species database is used
   with Ref data.

6. Fixed several issues:

   a. Renamed spcdb to metadata make_benchmark_aerosol_tables
   b. Added parentheses in routine face_area (in cstools.py) to force c
      orrect operator order
   c. Updated make_benchmark_aerosol_tables to include all dust
      species in the aerosol burdens table
   d. Updated make_benchmark_aerosol_tables to remove hardwiring in
      the computation of global AOD
   e. Restored missing YAML tags in 1yr_fullchem_benchmark.yml
      configuration file
   f. Added structural updates suggested by Pylint
   g. Fixed error in the determination of which variables are "Ref only"
      and "Dev only" in routine create_benchmark_emission_tables

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>

Author: yantosca

CHANGELOG.md

@@ -16,6 +16,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Added new constants for default lon/lat and stretched-grid settings in `gcpy/constants.py`
 - Added PyDoc headers in routines where they were missing
 - Added `examples/grids/display_gcclassic_grid_info.py`  to display info about a GEOS-Chem Classic horizontal grid
+- Added functions `get_molwt_from_metadata` and `read_species_metadata` to `gcpy/util.py`
+- Added function `get_species_database_files` to `gcpy/benchmark/modules/benchmark_utils.py`
+- Added constant `SPECIES_DATABASE` to `gcpy/benchmark/modules/benchmark_utils.py`
 
 ### Changed
 - Modified criteria for terminating read of log files in `benchmark_scrape_gcclassic_timers.py` to avoid being spoofed by  output that is attached by Intel VTune
@@ -39,17 +42,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Updated default GCPy Python environment to use Python 3.13 (instead of 3.12)
 - Benchmark routines now look for `species_database.yml` in the `Ref` and `Dev` run directories
 - Replaced `get_species_database_dir` with `get_species_database_files` in `gcpy/benchmark/modules/benchmark_funcs.py`
-- Replaced `spcdb_dir` YAML tag with directory-specific `species_metadata` tags to specify paths to `species_database.yml` files  
+- Updated `gcpy/benchmark/modules/benchmark_scrape_gchp_timers.py` to look for GCHP timers in `allPEs.log` if not found in the log file
+- Updated routine `make_benchmark_aerosol_tables` to include all dust species in the aerosol burdens table
 
 ### Fixed
 - Fixed grid area calculation scripts of `grid_area` in `gcpy/gcpy/cstools.py`
 - Fixed various security issues in GitHub Actions workflows
 - Fixed colorbar bounds for case of comparing cubed-sphere grids
 - Fixed the restart regridding for stretched GCHP when target lat/lon is exactly 0.0 in `gcpy/regrid_restart_file.py`
+- Fixed computation of the global AOD benchmark table caused by hardwired species names
+- Fixed error in `create_benchmark_emissions_table` where all species were assumed to be in Ref and Dev even if they were not
 
 ### Removed
 - Removed `PdfMerger()` from `compare_single_level` and `compare_zonal_mean`, it has been removed in pypdf >= 5.0.0
 - Removed `.load()` statements from xarray Datasets to improve performance
+- Removed `paths:spcdb_dir` YAML tag in benchmark configuration files
+- Removed `st_Ox` from `benchmark_categories.yml`; this species is no longer used in TransportTracers simulations
 
 ## [1.6.2] - 2025-06-12
 ### Added

gcpy/benchmark/cloud/template.1hr_benchmark.yml

@@ -29,16 +29,17 @@ paths:
   main_dir: ${GEOSCHEM_BENCHMARK_WORKING_DIR}
   results_dir: BenchmarkResults
   weights_dir: ${GEOSCHEM_BENCHMARK_WORKING_DIR}/weights
-  spcdb_dir: default
 #
 # data: Contains configurations for ref and dev runs
-#   version:         Version string (must not contain spaces)
-#   dir:             Path to run directory
-#   outputs_subdir:  Subdirectory w/ GEOS-Chem diagnostic files
-#   restarts_subdir: Subdirectory w/ GEOS-Chem restarts
-#   bmk_start:       Simulation start date (YYYY-MM-DDThh:mm:ss)
-#   bmk_end:         Simulation end date (YYYY-MM-DDThh:mm:ss)
-#   resolution:      GCHP resolution string
+#   version:          Version string (must not contain spaces)
+#   dir:              Path to run directory
+#   outputs_subdir:   Subdirectory w/ GEOS-Chem diagnostic files
+#   restarts_subdir:  Subdirectory w/ GEOS-Chem restarts
+#   logs_subdir:      Subdirectory w/ GEOS-Chem log files
+#   logs_template:    Template for log file names (may include tokens)
+#   bmk_start:        Simulation start date (YYYY-MM-DDThh:mm:ss)
+#   bmk_end:          Simulation end date (YYYY-MM-DDThh:mm:ss)
+#   resolution:       GCHP resolution string
 #
 data:
   ref:

gcpy/benchmark/cloud/template.1mo_benchmark.yml

@@ -32,13 +32,15 @@ paths:
   spcdb_dir: default
 #
 # data: Contains configurations for ref and dev runs
-#   version:         Version string (must not contain spaces)
-#   dir:             Path to run directory
-#   outputs_subdir:  Subdirectory w/ GEOS-Chem diagnostic files
-#   restarts_subdir: Subdirectory w/ GEOS-Chem restarts
-#   bmk_start:       Simulation start date (YYYY-MM-DDThh:mm:ss)
-#   bmk_end:         Simulation end date (YYYY-MM-DDThh:mm:ss)
-#   resolution:      GCHP resolution string
+#   version:          Version string (must not contain spaces)
+#   dir:              Path to run directory
+#   outputs_subdir:   Subdirectory w/ GEOS-Chem diagnostic files
+#   restarts_subdir:  Subdirectory w/ GEOS-Chem restarts
+#   logs_subdir:      Subdirectory w/ GEOS-Chem log files
+#   logs_template:    Template for log file names (may include tokens)
+#   bmk_start:        Simulation start date (YYYY-MM-DDThh:mm:ss)
+#   bmk_end:          Simulation end date (YYYY-MM-DDThh:mm:ss)
+#   resolution:       GCHP resolution string
 #
 data:
   ref:

gcpy/benchmark/config/1mo_benchmark.yml

@@ -21,24 +21,22 @@
 #   main_dir:    High-level directory containing ref & dev rundirs
 #   results_dir: Directory where plots/tables will be created
 #   weights_dir: Path to regridding weights
-#   spcdb_dir:   Folder in which the species_database.yml file is
-#                located.  If set to "default", then will look for
-#                species_database.yml in one of the Dev rundirs.
 #
 paths:
   main_dir: /path/to/benchmark/main/dir
   results_dir: /path/to/BenchmarkResults
   weights_dir: /n/holylfs06/LABS/jacob_lab/Shared/GEOS-CHEM/gcgrid/gcdata/ExtData/GCHP/RegriddingWeights
-  spcdb_dir: default
 #
 # data: Contains configurations for ref and dev runs
-#   version:         Version string (must not contain spaces)
-#   dir:             Path to run directory
-#   outputs_subdir:  Subdirectory w/ GEOS-Chem diagnostic files
-#   restarts_subdir: Subdirectory w/ GEOS-Chem restarts
-#   bmk_start:       Simulation start date (YYYY-MM-DDThh:mm:ss)
-#   bmk_end:         Simulation end date (YYYY-MM-DDThh:mm:ss)
-#   resolution:      GCHP resolution string
+#   version:          Version string (must not contain spaces)
+#   dir:              Path to run directory
+#   outputs_subdir:   Subdirectory w/ GEOS-Chem diagnostic files
+#   restarts_subdir:  Subdirectory w/ GEOS-Chem restarts
+#   logs_subdir:      Subdirectory w/ GEOS-Chem log files
+#   logs_template:    Template for log file names (may include tokens)
+#   bmk_start:        Simulation start date (YYYY-MM-DDThh:mm:ss)
+#   bmk_end:          Simulation end date (YYYY-MM-DDThh:mm:ss)
+#   resolution:       GCHP resolution string
 #
 data:
   ref:

gcpy/benchmark/config/1yr_ch4_benchmark.yml

@@ -21,15 +21,11 @@
 #   main_dir:    High-level directory containing ref & dev rundirs
 #   results_dir: Directory where plots/tables will be created
 #   weights_dir: Path to regridding weights
-#   spcdb_dir:   Folder in which the species_database.yml file is
-#                located.  If set to "default", then will look for
-#                species_database.yml in one of the Dev rundirs.
 #
 paths:
   main_dir: /path/to/benchmark/main/dir    # EDIT AS NEEDED
   results_dir: /path/to/BenchmarkResults   # EDIT AS NEEDED
   weights_dir: /n/holylfs06/LABS/jacob_lab/Shared/GEOS-CHEM/gcgrid/gcdata/ExtData/GCHP/RegriddingWeights
-  spcdb_dir: default
   #
   # Observational data dirs are on Harvard Cannon, edit if necessary
   #
@@ -43,13 +39,15 @@ paths:
       site_file: allozonesondes_site_elev.csv
 #
 # data: Contains configurations for ref and dev runs
-#   version:         Version string (must not contain spaces)
-#   dir:             Path to run directory
-#   outputs_subdir:  Subdirectory w/ GEOS-Chem diagnostic files
-#   restarts_subdir: Subdirectory w/ GEOS-Chem restarts
-#   bmk_start:       Simulation start date (YYYY-MM-DDThh:mm:ss)
-#   bmk_end:         Simulation end date (YYYY-MM-DDThh:mm:ss)
-#   resolution:      GCHP resolution string
+#   version:          Version string (must not contain spaces)
+#   dir:              Path to run directory
+#   outputs_subdir:   Subdirectory w/ GEOS-Chem diagnostic files
+#   restarts_subdir:  Subdirectory w/ GEOS-Chem restarts
+#   logs_subdir:      Subdirectory w/ GEOS-Chem log files
+#   logs_template:    Template for log file names (may include tokens)
+#   bmk_start:        Simulation start date (YYYY-MM-DDThh:mm:ss)
+#   bmk_end:          Simulation end date (YYYY-MM-DDThh:mm:ss)
+#   resolution:       GCHP resolution string
 #
 data:
   ref:

gcpy/benchmark/config/1yr_fullchem_benchmark.yml

@@ -21,16 +21,12 @@
 #   main_dir:     High-level directory containing ref & dev rundirs
 #   results_dir:  Directory where plots/tables will be created
 #   weights_dir:  Path to regridding weights
-#   spcdb_dir:    Folder in which the species_database.yml file is
-#                  located.  If set to "default", then will look for
-#                  species_database.yml in one of the Dev rundirs.
 #   obs_data:     Paths to observations (for models vs. obs plots)
 #
 paths:
   main_dir: /path/to/benchmark/main/dir
   results_dir: /path/to/BenchmarkResults
   weights_dir: /n/holylfs06/LABS/jacob_lab/Shared/GEOS-CHEM/gcgrid/gcdata/ExtData/GCHP/RegriddingWeights
-  spcdb_dir: default
   #
   # Observational data dirs are on Harvard Cannon, edit if necessary
   #
@@ -44,13 +40,16 @@ paths:
       site_file: allozonesondes_site_elev.csv
 #
 # data: Contains configurations for ref and dev runs
-#   version:         Version string (must not contain spaces)
-#   dir:             Path to run directory
-#   outputs_subdir:  Subdirectory w/ GEOS-Chem diagnostic files
-#   restarts_subdir: Subdirectory w/ GEOS-Chem restarts
-#   bmk_start:       Simulation start date (YYYY-MM-DDThh:mm:ss)
-#   bmk_end:         Simulation end date (YYYY-MM-DDThh:mm:ss)
-#   resolution:      GCHP resolution string
+#   version:          Version string (must not contain spaces)
+#   dir:              Path to run directory
+#   outputs_subdir:   Subdirectory w/ GEOS-Chem diagnostic files
+#   restarts_subdir:  Subdirectory w/ GEOS-Chem restarts
+#   logs_subdir:      Subdirectory w/ GEOS-Chem log files
+#   logs_template:    Template for log file names (may include tokens)
+#   bmk_start:        Simulation start date (YYYY-MM-DDThh:mm:ss)
+#   bmk_end:          Simulation end date (YYYY-MM-DDThh:mm:ss)
+#   resolution:       GCHP resolution string
+#
 #
 data:
   ref:
@@ -134,15 +133,18 @@ options:
     plot_drydep: True
     plot_emis: True
     plot_jvalues: True
+    plot_models_vs_obs: True
     plot_options:
       by_spc_cat: True
       by_hco_cat: True
     #
     # Benchmark tables
     #
+    aer_budget_table: True
     emis_table: True
     mass_accum_table: False
     mass_table: True
+    Ox_budget_table: True
     OH_metrics: True
     ops_budget_table: False
     sanity_check_table: True

gcpy/benchmark/config/1yr_tt_benchmark.yml

@@ -21,24 +21,23 @@
 #   main_dir:    High-level directory containing ref & dev rundirs
 #   results_dir: Directory where plots/tables will be created
 #   weights_dir: Path to regridding weights
-#   spcdb_dir:   Folder in which the species_database.yml file is
-#                located.  If set to "default", then will look for
-#                species_database.yml in one of the Dev rundirs.
 #
 paths:
   main_dir: /path/to/benchmark/main/dir
   results_dir: /path/to/BenchmarkResults
   weights_dir: /n/holylfs06/LABS/jacob_lab/Shared/GEOS-CHEM/gcgrid/gcdata/ExtData/GCHP/RegriddingWeights
-  spcdb_dir: default
 #
 # data: Contains configurations for ref and dev runs
-#   version:         Version string (must not contain spaces)
-#   dir:             Path to run directory
-#   outputs_subdir:  Subdirectory w/ GEOS-Chem diagnostic files
-#   restarts_subdir: Subdirectory w/ GEOS-Chem restarts
-#   bmk_start:       Simulation start date (YYYY-MM-DDThh:mm:ss)
-#   bmk_end:         Simulation end date (YYYY-MM-DDThh:mm:ss)
-#   resolution:      GCHP resolution string
+#   version:          Version string (must not contain spaces)
+#   dir:              Path to run directory
+#   outputs_subdir:   Subdirectory w/ GEOS-Chem diagnostic files
+#   restarts_subdir:  Subdirectory w/ GEOS-Chem restarts
+#   logs_subdir:      Subdirectory w/ GEOS-Chem log files
+#   logs_template:    Template for log file names (may include tokens)
+#   bmk_start:        Simulation start date (YYYY-MM-DDThh:mm:ss)
+#   bmk_end:          Simulation end date (YYYY-MM-DDThh:mm:ss)
+#   resolution:       GCHP resolution string
+#
 #
 data:
   ref:

gcpy/benchmark/modules/benchmark_categories.yml

@@ -256,7 +256,6 @@ TransportTracersBenchmark:
       - e90_n
       - e90_s
       - st80_25
-      - stOX
   WetLossConv:
     WetLossConv:
       - Pb210

gcpy/benchmark/modules/benchmark_drydep.py

@@ -18,6 +18,7 @@ def make_benchmark_drydep_plots(
         refstr,
         dev,
         devstr,
+        spcdb_files,
         collection="DryDep",
         dst="./benchmark",
         subdst=None,
@@ -30,7 +31,6 @@ def make_benchmark_drydep_plots(
         n_job=-1,
         time_mean=False,
         varlist=None,
-        spcdb_dir=None,
 ):
     """
     Creates six-panel comparison plots (PDF format) from GEOS-Chem
@@ -48,6 +48,8 @@ def make_benchmark_drydep_plots(
             data set.
         devstr: str
             A string to describe dev (e.g. version number)
+        spcdb_files : list 
+            Paths to species_database.yml files in Ref & Dev rundirs
 
     Keyword Args (optional):
         collection : str
@@ -79,20 +81,13 @@ def make_benchmark_drydep_plots(
             Set to 1 to disable parallel plotting. Value of -1 allows the
             application to decide.
             Default value: -1
-        spcdb_dir: str
-            Directory of species_datbase.yml file
-            Default value: None
         time_mean : bool
             Determines if we should average the datasets over time
             Default value: False
         varlist: list of str
             List of variables to plot.  If varlist is None, then
             all common variables in Ref & Dev will be plotted.
     """
-    # Make sure the species database folder is passed
-    if spcdb_dir is None:
-        msg = "The spcdb_dir argument has not been specified!"
-        raise ValueError(msg)
 
     # Replace whitespace in the ref and dev labels
     refstr = util.replace_whitespace(refstr)
@@ -144,7 +139,7 @@ def make_benchmark_drydep_plots(
         sigdiff_list=sigdiff_list,
         weightsdir=weightsdir,
         n_job=n_job,
-        spcdb_dir=spcdb_dir
+        spcdb_files=spcdb_files,
     )
     util.add_bookmarks_to_pdf(
         pdfname,