intake-esgf configuration for NERSC/m4931

[pochedls]

Describe the solution you'd like

@sashakames has been moving data to and updating CMIP5/6 holdings at NERSC (project space m4931). It would be helpful if intake-esfg could "see" that data (so that it isn't re-downloaded).

Describe alternatives you've considered

Alternatives could include creating some kind of query-able index that could be used separate from intake-esgf.

[nocollier]

Thanks for this report, I wasn't sure when data was available nor where the new location is. I will happily add it to the defaults if you can let me know what the new path is? Should the old /global/cfs/projectdirs/m3522/cmip6/ be removed yet?

To test it and in the near term, you can configure intake-esgf with any number of "roots":

import intake_esgf
intake_esgf.conf.set(esg_dataroot=intake_esgf.conf["esg_dataroot"]+["/new/path/to/add"])
intake_esgf.conf

[sashakames]

/global/cfs/projectdirs/m4931/ is the new home for ESGF data. There are several subdirectories there. We are trying to mirror the structure that began at LLNL and has been mirrored to ORNL and ANL, namely css03_data and user_pub_work, effectively using the Globus subfolder naming convention.
For instance most of the CMIP6 data can be found at /global/cfs/projectdirs/m4931/css03_data/CMIP6
I may have merged the CMIP6 under user_pub_work to the css03 path. We can relocate those if need be.

[sashakames]

Follow-up question that may be of interest: is CMIP5 supported? My understanding is that it is still in use at PCMDI.

[nocollier]

From the point of view of intake-esgf, we will form the local path from the facet information and prepend each esg_dataroot and check for existence. So I don't care about which projects are present as long as I can form the correct path in that way.

CMIP5 should be functional, note that I even have a software fix for the republishing oddities. To the user it will appear to work at CMIP6 does--a dataset_id per variable, not table.

[jsta]

As kind of an aside, I'm using ecgtools to create a query-able index separate from intake-esgf

[pochedls]

As kind of an aside, I'm using ecgtools to create a query-able index separate from intake-esgf

@jsta – Thanks for chiming in! Do you mean that you're creating this index on NERSC? You probably saw this, but the "data lake" on NERSC is currently being augmented by @sashakames. I'd be interested in using your index (is it a .csv file like I've seen in other intake-[] projects?).

[jsta]

No, I'm creating an index for a relatively small (but growing) mirrored subset of files pulled via intake-esgf at my organization. It is json.

[mgrover1]

@jsta - this is related some of the requests here
#44 (comment)

As one of the maintainers of ecgtools/intake-esm, we would like to add support for those catalogs within the intake-esgf interface, and this is a great use-case to use for motivation.