Support dataset "collections"

[lkluft]

While we advocate for single large dataset rather than many smaller ones, there are situations where this is not feasible (or not wanted by the data providers). Some examples in the ORCESTRA dataset are the individual HALO position/attitude datasets (could be solved by a single dataset) or the lower level dropsondes data (not easy to merge into one dataset).

It would be nice to be able to define "collections" or "sets" or datasets. A simple solution would be to also check for metadata in specified files, e.g. metadata.json. This file could contain all the relevant attributes to render a summary page, and also have a special datasets keyword to link to other datasets.

[d70-t]

This raises the question of how to format the metadata file. Let's try to gather some options:

• Currently, we are creating a STACItem for each "dataset" we are showing in the data browser. So naturally, a JSON file containing all the information we need to display a landing page would be a STACItem representing this dataset. A STAC item must represent something bounded in space and time, which likely is possible for any kind of data collection within ORCESTRA. And an item can have many assets (e.g. files / datasets etc...). Currently we only have single-asset items, but I could imagine having items with multiple assets as well.
• There's a specification for a STACCollection, which is a way to group several STACItems into a group sharing some commonality. A colleciton also has to have some bounds in space and time, but it also has to refer to individual stac items, thus it may be even more complicated to build.
• There's ro-crate which is a generic way of describing "datasets". Creating RO-Crate Metadata is probably even more complicated than STACItem metadata, and it may raise the issue of how to translate RO-Crate Metadata to STAC metadata.
• We could build something on our own. This might be the simplest format to use at first, but probably we'll have to come up with a way to encode most of what's necessary for a STACItem anyways, so probably it's not worth creating something new in this place.

Note that "dataset", is an overloaded term in this context. When doing zarr->STACItem conversions, we use the dataset in the sense of a zarr / netCDF-like dataset. But RO-Crate has a more general notion of a dataset, basically just any collection of files can do. And STACItems are somewhat similar to datasets, but as they may consist of an arbitrary number of assets, they are also somewhat more close to the RO-Crate definition than to the zarr definition.

[lkluft]

I would argue that the multi-asset STACItems are quite close to our definition of a dataset collection. It even seems to cover cases like HEALPix hierarchies, which is nice.

A practical question is whether the JSON file is still a more human-writable abstraction of the dataset, i.e. containing only the attributes we currently define in the global dataset attributes. Alternatively, one could add a STACItem directly, but this would probably mean more work for the data provider, and doesn't seem necessary since the conversion from metadata to STACItem is already implemented.

In fact, we could support both. If there is a valid JSON representation of a STACItem, it could be used, otherwise we try to parse and convert the metadata.

[d70-t]

I like the idea to support both. Probably we also have to divert a little from the true STAC spec, as we probably want to generate the id of the STAC item based of the CID of the containing folder, so we couldn't have the ID set in a stac item contained in that folder.

I'm wondering a bit however, how we would want to fill in e.g. the spatial and temporal extent of the data, and probably also the assets. Those are required for STAC items and are the most annoing things to collect manually. But if we don't have a well-defined dataset (or collection of those), then it's not straight-forward how to collect this information automatically.

[lkluft]

I like the idea to support both. Probably we also have to divert a little from the true STAC spec, as we probably want to generate the id of the STAC item based of the CID of the containing folder, so we couldn't have the ID set in a stac item contained in that folder.

I would argue that this is more of an argument for providing metadata that is converted into a STAC item.

I'm wondering a bit however, how we would want to fill in e.g. the spatial and temporal extent of the data, and probably also the assets. Those are required for STAC items and are the most annoing things to collect manually. But if we don't have a well-defined dataset (or collection of those), then it's not straight-forward how to collect this information automatically.

I think there are several points to this:

1. I think there is no way around collecting the assets. However, I would argue that most data providers will fall into either the "handful and manual" or "dozens and scripted" categories.
2. In general, this seems even more of a case for automated creation of the STAC item itself. That way we could iterate over the assets and collect the spatiotemporal extent. Unless of course...
3. The collection/metadata includes datasets from which we cannot retrieve such information directly (e.g. ASCII files). I'm not sure there is a proper solution for this (other than an additional level of metadata JSON). But maybe this is an acceptable level to gently remind data providers to change their data formats.