List non-Zarr datasets in browser

[lkluft]

Currently, only Zarr datasets can be shown in the ORCESTRA browser, because the necessary (meta-)data is fetched from the dataset directly. However, it would be nice to be able to show other sources (PDFs, images, ...).

This likely has large overlaps with #42 but maybe one or the other might be easier to implement first.

[lkluft]

Let's try to push this forward!

For non-Zarr datasets, I think the general idea is to add the metadata explicitly by adding a text file to the top-level directory.

The straight-forward solution would be to add a STAC-like (but reduced) item:

{
  "properties": {
    "title": "BEACH dropsonde dataset (Level 3)",
    "description": "This dataset is the Level 3 BEACH dataset.",
    "keywords": ["ORCESTRA", "BEACH"],
    "license": "CC-BY-4.0",
    "references": ["https://github.com/atmdrops/pydropsonde"],
    "platform": "HALO",
    "mission": "ORCESTRA, PERCUSION, MAESTRO",
    "processing:lineage": "Level 1 ASPEN processing with Aspen V4.0.4",
    "cube:dimensions": {
      "altitude": {
        "type": "spatial",
        "axis": "z",
        "unit": "m",
      }
    },
    "cube:variables": {
      "iwv": {
        "dimensions": [
          "sonde"
        ],
        "type": "data",
        "unit": "kg m-2",
      }
    },
    "contacts": [
      {
        "name": "Helene Gloeckner",
        "emails": [
          {
            "value": "helene.gloeckner@mpimet.mpg.de",
            "roles": []
          }
        ]
      },
      {
        "name": "Theresa Mieslinger",
        "emails": [
          {
            "value": "theresa.mieslinger@mpimet.mpg.de",
            "roles": []
          }
        ]
      }
      ],
    "datetime": "2024-09-03T16:58:42.633Z",
    "start_datetime": "2024-08-09T14:26:37.333Z",
    "end_datetime": "2024-09-28T19:30:47.934Z"
  },
}

The advantage of this would be that users could basically get as close to the full STAC item as they want to (e.g. listing variables). One obvious disadvantage is that these items are almost impossible to create by hand. Since manual metadata addition is likely to be necessary for external datasets and/or people who do not want to adjust their workflows, it is doubtful that tools will be created for this purpose.

Alternatively, we could define our own intermediate format, which is more human writable. One could even consider to make the field names resemble the actual CF Convention attributes.

title: "BEACH dropsonde dataset (Level 3)"
description: "This dataset is the Level 3 BEACH dataset."
keywords: "ORCESTRA", "BEACH"
license: "CC-BY-4.0"
references: "https://github.com/atmdrops/pydropsonde"
platform": "HALO"
mission": "ORCESTRA, PERCUSION, MAESTRO"
processing:lineage": "Level 1 ASPEN processing with Aspen V4.0.4"
contacts:
  - name: "Helene Gloeckner"
    email: "helene.gloeckner@mpimet.mpg.de"
  - name: "Theresa Mieslinger"
    email: "theresa.mieslinger@mpimet.mpg.de"
start_datetime: "2024-08-09T14:26:37.333Z"
end_datetime: "2024-09-28T19:30:47.934Z"

Although this approach is more likely to be adopted by a broader user base, it is also limited in its functions because the converter needs to be adapted.

/cc @tmieslinger @hgloeckner @d70-t

[lkluft]

After letting this sink over the weekend, I tend to prefer the user-friendly approach.

I would suggest to define a metadata.yaml, which allows data creators to add the same attributes as recommended in the attribute convention. My reasoning is that this:

1. Reduces confusion as we are using the same set of attributes/keywords when annotating datasets in different ways (attributes vs. text file)
2. The metadata scanner works in the same way, just with a different source
3. It should be enough to serve as a proof-of-concept, no matter what we decide for in other contexts (e.g. HALO-DB 2.0)

One drawback of this approach is that information that is only available in the datasets, such as start/end dates or the variable list, would not be included. However, at this stage, this might be acceptable, or it might actually nudge users to create proper datasets. Also, a Markdown-formatted list of variables could be added to the summary as a fallback.

[d70-t]

My feeling would be, that we should go for something human-writable, (as you say) this is meant as an option for somwhat irregular datasets, which don't carry standardized metadata themselves. If users would be to automate this, they should rather change to a different format (e.g. zarr) instead of using this kind of metadata.

However, I do think, that we should have some possibility to add information like temporal or spatial extent to the datasets for two reasons:

• I guess people like to search for these things eventually
• We still (?) of plan to use STAC items internally. I'm trying to understand the item spec (again), and my current understanding would be, that at least one datetime is necessary, but the geometry may be empty (I guess it's not recommended though).

My current ideas are:

• define metadata.yaml, but place the global attributes in one specific key (e.g. global or attrs or attributes, and have an option for further top-level keys (e.g. time, geometry, variables etc...)
• implement two formats, the simple metadata.yaml as @lkluft suggested and additional stac.json which could be used for complicated cases

[lkluft]

I agree that it should be possible to add information beyond the global attributes of the dataset. However, we should keep the list of extensions as short as possible.

Using the item spec directly seems to cause a lot of path dependencies, because we'd have to make our mapping from CF to STAC public (thus somehow establishing a second convention) and limit ourselves to STAC in the future.

I do like your idea to use a YAML file that allows extra fields.

Proposal

A directory that contains a file dataset_meta.yaml is considered a dataset. The YAML file must provide an attributes field. The attributes must adhere to the ORCESTRA Attribute Convention.

The YAML file may contain additional fields to fill-in missing information from dataset coordinates:

Key	Value
start_date	Start date of the dataset in ISO 8601 format
end_date	End date of the dataset in ISO 8601 format
extent	Geospatial extent of the dataset defined by a GeoJSON Bounding Box

Example:

attributes:
  title: BEACH dropsonde dataset (Level 3)
  summary: This dataset is the Level 3 BEACH dataset. It contains quality controlled
    dropsonde data from the ORCESTRA field campaign with a common altitude dimension.
  creator_name: Helene Gloeckner, Theresa Mieslinger, Nina Robbins
  creator_email: helene.gloeckner@mpimet.mpg.de, theresa.mieslinger@mpimet.mpg.de, nina.robbins@mpimet.mpg.de
  license: CC-BY-4.0
  featureType: trajectoryProfile
  history: "Level 1 ASPEN processing with Aspen V4.0.4"
  keywords: ORCESTRA, BEACH, Sounding, Dropsondes, Atmospheric Profiles
  platform: HALO
  project: ORCESTRA, PERCUSION, MAESTRO
  references: https://github.com/atmdrops/pydropsonde
  source: dropsondes
start_date: "2024-08-09T14:26:37"
end_date: "2024-09-28T19:30:47"
extent: [-59.45647812, 1.29273319, -19.62099838, 22.03603554]

[lkluft]

I have refined the above proposal based on offline discussions. It now seems ready for a first prototype, but comments are still welcome.

I will keep this issue open until a first non-Zarr dataset can be presented properly in the data browser.

[d70-t]

I mostly agree with the current version, but somehow I still feel, that "extent" shouldn't only be used for spatial extent, so my idea would be to go for something like:

extent:
  temporal: ["2024-08-09T14:26:37", "2024-09-28T19:30:47"]
  spatial: [-59.45647812, 1.29273319, -19.62099838, 22.03603554]

[lkluft]

Fine with me :)

I will open a PR that adds a section to the ORCESTRA Attribute Conventions describing this method of annotating datasets.

Helene has agreed to test this by adding metadata to the Level 0 dropsonde data, which will provide us with a test dataset.

[tmieslinger]

Thanks @lkluft for reviving this issue!
I agree with the discussion above and understand it such that the extra dataset_meta.yaml file would be rather easy to generate out of the dataset collection that it shall describe. I'd assume that the info from the YAML file is enough to build the usual landing page in the O data browser (with parameters/variables being optional?).

One point that is not yet clear to me is how it's communicated to the user that this is a collection of individual datasets and maybe a second related point, how to work with the datasets within this collection (code example?). Maybe the latter goes beyond this issue as it rather relates to the question "how to best communicate dataset collections in the data browser / on a landing page". But do you see a need to add any information already to the metadata YAML that somehow points to the fact that this is meant to describe many datasets?

[lkluft]

[...] how it's communicated to the user that this is a collection of individual datasets

and [...], how to work with the datasets within this collection (code example?).

In short, the answer to both points is: "It isn't". Currently, the dataset_meta.yaml is intended as a generic fallback solution for any data that does not provide accessible attributes. Therefore, a dataset collection is seen as any other directory. Also, we don't limit the data format that can be annotated in this way (it might be CSV, PDF or NetCDF).

It's a good idea to consider some optional metadata that provide more information about the "dataset", but I think this could quickly become unwieldy.