GeoZarr SWG Strategic Reset

[christophenoel]

Hello,

The following text was shared on the mailing list to summarise the outcome of our recent meeting:

During yesterday’s meeting, members agreed that the approach followed over the past two years has shown limited effectiveness. Efforts to design a unified abstract data model and to harmonise all data frameworks proved too complex, difficult to maintain, and challenging for contributors to adopt. Reaching consensus under this model has also been difficult.

The group therefore agreed to refocus its work on developing concrete, thematic, and practical Zarr conventions that address needs already expressed by the community. A Zarr convention defines a clear and self-contained extension for a specific topic. Examples under consideration include:

• Zarr Multiscales Convention – support for pyramidal map overviews and multidimensional multiscale structures (already under development)
• Affine Coordinates Convention – definition of coordinates based on affine transforms rather than explicit coordinate arrays.
• NetCDF Convention – mapping of NetCDF and CF constructs to Zarr.
• GeoTIFF Convention – mapping of GeoTIFF structures to Zarr.

These conventions may be developed directly within the GeoZarr repository, in external repositories, or through a hybrid model. The GeoZarr specification will be produced once a set of concrete conventions is mature enough to form the basis of a recommended and standardised suite of Zarr conventions.

To reflect this shift in strategy within our GitHub space, suggestions for next steps include:

• Updating the README and wiki to present the new direction clearly.
• Moving the current material into an archive/ directory to provide a clean starting point (??)
• Defining a clear structure for hosting conventions developed within this repository.
• Listing existing external repositories already working on conventions relevant to GeoZarr.
• Planning for a lightweight GeoZarr specification that will primarily reference the set of adopted conventions.
• Preparing a simple roadmap to identify candidate convention topics and potential contributors.

Feedback on these proposed actions, or additional suggestions, would be welcome.

[christophenoel]

Note: I have reset the repository.

[pvanlaake]

I would suggest to rename the Affine Coordinates Convention to just the Coordinates Convention. The reason for this is that there are multiple ways to represent coordinates and an affine transform is just one particular way of doing that (and also a specific form of a more general system of coordinates). The Coordinates Convention could thus support affine transforms (such as GDAL's GeoTransform tag) but also other approaches/conventions (such as those used in CF).

I would also suggest to align the GeoZarr constructs with OGC standards, such as "Referencing by Coordinates" (18-005r5).

[chris-little]

OGC standards, such as "Referencing by Coordinates" [18-005r5](https://docs.ogc.org/as/18-005r5/18-005r5.html) is also an ISO standard ISO 19111:2019.

[mdsumner]

I would suggest to rename the Affine Coordinates Convention to just the Coordinates Convention. The reason for this is that there are multiple ways to represent coordinates and an affine transform is just one particular way of doing that (and also a specific form of a more general system of coordinates). The Coordinates Convention could thus support affine transforms (such as GDAL's GeoTransform tag) but also other approaches/conventions (such as those used in CF).

I would also suggest to align the GeoZarr constructs with OGC standards, such as "Referencing by Coordinates" (18-005r5).

Agreed. It's often enough just to have "bbox" which makes it much clearer (when) this is a regular grid, this is used to great effect in the R package terra and in many gdal utilities and api functions and in the GDAL multidimensional framework. Another aspect that is conflated by a focus on "coordinates" is when they are redundant (compactable to a bounding box or transform), or cryptic (explicitly stored coordinates transformed from a regular grid - perhaps necessarily for the original model work, but unnecessary and unhelpful for downstream usage). We tend to say "georeferencing", but that's too domain specific (I'm not arguing against calling this "Coordinates", but very shortly into any content it should refer to compact forms of that and their value). The new xarray indexing is a great example of how this can be folded in to otherwise "coordinate" centric language and tools, and also how compact forms are specifiable in n-dimensional forms and in mixed forms (we can have a mix of rectlinear/curvilinear/regular "coordinates" on any dataset, regular steps in Time are an obvious example but can be any individual axis or set of axes).

[christophenoel]

I would suggest to rename the Affine Coordinates Convention to just the Coordinates Convention.

I assume that "NetCDF Convention" or "Common Data Model Convention" would describe the explicit coordinates array approach, so an "affine coordinate convention" would add an extra approach for encoding coordinates.

[pvanlaake]

Since this is a strategic reset, I would think that this would also be the time to reflect on approaches that have been trialled over the past two years. Use of arrays as coordinate variables (which is neither netCDF nor CDM, it is a CF Metadata Convention that was absorbed from the COARDS convention) would be one of those things.

What has always puzzled me is that GeoZarr does not explicitly define its coordinate space. Use of coordinate variables at best provides an implicit coordinate space, but it is clunky, inefficient, incomplete and non-standard. In CF, it is paired with other constructs to cover edge cases, with the "scalar axis" probably being the most prominent example. That said, CF is a treasure trove of ideas that address many of the problems that GeoZarr also needs to solve so let's build on it.

Seeing that GeoZarr is a prospective OGC standard, I would argue that it adopts a definition of array coordinate space that aligns with other OGC standards, such as "Referencing by Coordinates". That can be restricted to the main aspects or fully developed. As a very initial idea, how about this:

"coordinates": {
  "name": <character_value>,
  "crs": [<crs>, ...],                 -- may be compound, restrictions on combinations of CRS apply
  "axes": [<axis>, ...],               -- must agree with the cs_axes of the crs
}

"axis": {
  "name": <character_value>,
  "abbreviation": <character_value>,                --| These 3 are repeated from OGC cs_axis
  "direction": <direction>,                         --| so that they are easily accessible
  "unit": <character_value>,                        --| if CRS is specified by identifier
  "dimension": <number>,                            -- dimension, 0..n-1, of the shape, omitted for length-1 axis
  "values": [<first_value>, <increment>] | <key>,   -- <key> for irregular coordinate arrays
  "bounds": [<left>, <right>] | <key>,              -- [<left>, <right>] for regular offsets, <key> for irregular offsets
}

Note that this supports CF coordinate variables, CF scalar axes, CF boundary variables, but also GDAL GeoTransform parameters for the basic case of no-rotation, no shear (using two axes X and Y with "values": [<first_value>, <increment>]).

Happy to spin this off into another issue for discussion, if there is interest to do so.

[dblodgett-usgs]

How does this strategic reset relate to the SWG charter? Does this warrant a rewrite or any modification?

If I'm following the suggestion, you are suggesting definition of several purpose-specific "convention" documents rather than a unified data model that encompasses the broader set of needs laid out in the charter?

In my experience, a clear SWG charter should provide the "north start" for the group such that these kinds of strategic issues can be overcome or avoided. My read of the current charter leaves me wondering if encoding the decision recorded in this issue in the charter would be a wise move?