catalogs for SPEAR cmorized data

[ariaradick]

Describe the bug
I am attempting to make a catalog on some data that is publicly available, located in
/data/2/GFDL-LARGE-ENSEMBLES/CMIP/NOAA-GFDL/GFDL-SPEAR-MED/historical/
on the GFDL systems. I have tried using the fre version of catalog builder via fre/2025.03 and fre/2025.04, running fre catalog builder $INPUT $OUTPUT, and also trying with a config file. When I run with the verbose flag, I see "skipping" for each dataset of interest.

To Reproduce
Run fre catalog builder --verbose /data/2/GFDL-LARGE-ENSEMBLES/CMIP/NOAA-GFDL/GFDL-SPEAR-MED/historical/ $OUTPUT after loading fre with module load

Expected behavior
I expected an intake-esm catalog that is not empty as output.

[ariaradick]

Good news: the updates in #134 solve this!

Bad news: because this file structure lacks 'realm', the new version fails to run in slow mode --slow with error:

Traceback (most recent call last):
  File "/nbhome/Aria.Radick/conda/envs/catalogbuilder/bin/gen_intake_gfdl", line 7, in <module>
    sys.exit(create_catalog_cli())
             ~~~~~~~~~~~~~~~~~~^^
  File "/nbhome/Aria.Radick/conda/envs/catalogbuilder/lib/python3.13/site-packages/click/core.py", line 1462, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nbhome/Aria.Radick/conda/envs/catalogbuilder/lib/python3.13/site-packages/click/core.py", line 1383, in main
    rv = self.invoke(ctx)
  File "/nbhome/Aria.Radick/conda/envs/catalogbuilder/lib/python3.13/site-packages/click/core.py", line 1246, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nbhome/Aria.Radick/conda/envs/catalogbuilder/lib/python3.13/site-packages/click/core.py", line 814, in invoke
    return callback(*args, **kwargs)
  File "/home/Aria.Radick/Documents/CatalogBuilder/catalogbuilder/scripts/gen_intake_gfdl.py", line 181, in create_catalog_cli
    return create_catalog(**kwargs)
  File "/home/Aria.Radick/Documents/CatalogBuilder/catalogbuilder/scripts/gen_intake_gfdl.py", line 103, in create_catalog
    list_files = gfdlcrawler.crawlLocal(project_dir, dictFilter, dictFilterIgnore, configyaml,slow)
  File "/home/Aria.Radick/Documents/CatalogBuilder/catalogbuilder/intakebuilder/gfdlcrawler.py", line 137, in crawlLocal
    qualities=(dictInfo["variable_id"],dictInfo["realm"])
                                       ~~~~~~~~^^^^^^^^^
KeyError: 'realm'

are there any cases where a variable has a different long name depending on the realm?

[ariaradick]

The config file that I'm attempting to use is available here

[aradhakrishnanGFDL]

@Ciheim this looks like an interesting and relevant use-case for catalog builder. Are you able to take a look? I'd start with the cmip template json in our repo to cross-check configs. I am available offline to help as needed.