[Proposal] Add a catalog field

[pzwsk]

What is the context or reason for the change?

When searching for a risk dataset, I would like to be able to filter by catalog such as DDH World Bank or JRC Risk Data Hub when relevant. This is not possible in the current version of RDLS to my knowledge.

Why is this not covered by the existing model?

Dataset has a related-resource element that might be used but has been created to describe publications rather than catalogs.

See https://docs.riskdatalibrary.org/en/latest/reference/schema/#related-resource

What is your proposed change?

I propose to add a field catalog and used DCAT standard to reference catalogs where the dataset is referenced.

Can you provide an example?

To be done ...

[stufraser1]

What happens when the data is in multiple catalogs?

[pzwsk]

It should allow multiple catalog entries

[stufraser1]

ODS to address this in their contract

[matamadio]

Please specify:

• at which level to include/show the catalogue attribute array?
  • general
  • general/resource
  • links
• what type of codelist? If it reflects into a filter in Jkan, it should be a closed codelist (I guess)

[stufraser1]

Propose this is added at the general level based on the expectation that the data resources held in one dataset are derived from the same catalogue.

Questions to resolve:

• 1a. Is the intention to list the catalog in which the data can be accessed/downloaded from? In which case it would be one catalog and resource includes access/download url.
• 1b. OR is the intention to list all catalogs in which the data can be found - and then do we need the URL of each catalog listed here, requiring a new field for catalog URLs.

I would say 1a would be more manageable; listing all catalogs the data can be found in, is beyond our scope really.

• 2. Do we always list JKAN / RDL Collection as a catalog, since by creating the metadata and adding it to the JKAN we are putting it into this catalog

My view: I believe we use this field to cite which catalog the DATA is hosted in. So JKAN would not feature (it is metadata only) but if the data comes from WB and its primary location is WB Data Catalog RDL Collection then we list that catalog. If it's primary homw is e.g. JRC DRMKC, then we list that only.

• 3. Codelist: Intention is to use it as a filter, so I agree it is better to use closed codelist, but it would be very challenging to list all catalogs that the datasets could come from. We may never achieve comprehensive codelist, but could make a good stab at listing most common and including as an open codelist. Without this though, it would need to be a string field.

As string field:

"catalogs": {
      "title": "Catalogs",
      "type": "string",
      "description": "Catalog or catalogs that host this dataset. Multiple catalogs can be included in a comma-separated list. See the dataset `resource` metadata for access or download URLs.",
      "minLength": 1
    },

or as array field:

"catalogs": {
      "title": "Catalogs",
      "type": "array",
      "description": "An array of catalog names that host this dataset. See the dataset `resource` metadata for the access or download URLs.",
      "items": {
        "type": "string",
        "enum": [
          "catalog_name_1",
          "catalog_name_2",
          "catalog_name_3",
          "catalog_name_4"
        ]
      },

[matamadio]

To me "Access URL" already does the job without being constrained to catalogs only, so it would be like duplicating the field while making it more specific.
Plus there will be cases when metadata are produced before knowing which catalog will hold them.
But I agree it's useful to have those sorted by catalog.

What if this is a function of JKAN instead of a RDLS attribute?
I.e. when adding to JKAN, it will index all dataset access urls and compare against a list of known catalogues, if they fit with one of those they got grouped in the corresponding filter.

Example: dataset access url includes "https://datacatalog.worldbank.org/" > it gets listed in the "WB DDH" filter.

We can as well have similar function with the implementation of AI search bar, which could output results ordered by hosting catalogue.

[matamadio]

We can also use the "links" group of attributes to add reference to the original catalog.
And have the JKAN fallback to parsing the access url if no external catalog link.
If the url doesn't map to any catalog from our list, it goes grouped under "other".

---

Draft list of listed catalogues:

Disaster & Humanitarian Data

• OASIS HUB - oasishub.co
• JRC DRMKC risk hub - drmkc.jrc.ec.europa.eu/risk-data-hub#
• Humanitarian Data Exchange - data.humdata.org
• EM-DAT (Emergency Events Database) - emdat.be
• GDACS (Global Disaster Alert and Coordination System) - gdacs.org
• GAR PreventionWeb - risk.preventionweb.net

Climate Data

• NOAA - ncei.noaa.gov
• Copernicus Climate Data Store - cds.climate.copernicus.eu
• NASA Earth Data - earthdata.nasa.gov

Development Data

• World Bank Data Catalog - datacatalog.worldbank.org
• UNDP GeoHub - geohub.data.undp.org
• UNOSAT - unosat.org
• FAO Data - fao.org
• Asian Development Bank Data Library - data.adb.org
• African Development Bank Data Portal - dataportal.opendataforafrica.org

Geospatial Data

• OpenStreetMap - openstreetmap.org
• Copernicus Open Access Hub - scihub.copernicus.eu
• NASA SEDAC - sedac.ciesin.columbia.edu

Multi-thematic Research Repositories

• Zenodo - zenodo.org
• figshare - figshare.com
• Dryad - datadryad.org
• European Data Portal - data.europa.eu

[matamadio]

@lydiascarf after fixing priority issues, please have a look at this.