NetCDF - Zarr Users Guide

[dblodgett-usgs]

Hi Zarr Developers -- I hope this message finds you safe and thriving!

As many of you are aware, the topic of NetCDF (specifically the widely adopted CF convention for NetCDF) has been a hot one with relation to Zarr since its inception. Yet in all the time that Zarr has been able to store data from NetCDF-CF sources, there has always been a degree of ambiguity in the finer points of encoding / round tripping data.

I've personally dipped in and out of this topic over the years and have recently come back to it in earnest. After a fair bit of rumination, I think I've realized something that could unstick the topic.

In short -- we need a NetCDF User's Guide for Zarr that defines the structure upon which CF is built to be defined by the community that stewards the Zarr format.

Background: Referred to simply as the "NUG" in the cf-convention -- it is the formative document providing an underlying structural reference upon which CF is built. The NUG is also simply a user's manual that Unidata put out when it first published NetCDF (the API, the format, etc.). As a result, it has content that pertains to much much more than the NetCDF file format itself. The details of the file format in the NUG have been stable / advancing over time as NetCDF3 transitioned to 4 and features were added. The core of the NetCDF data model is defined in the NUG and has been the same since its inception.

So what would a "NZUG" look like and who would own/steward it? I'd like to imagine a world where it would look like a zarr-developers repository modeled on the zarr-spec and the zarr-convention template. It would break some of the rules of zarr-conventions because it's structural, but it would be wholly defined on top of the Zarr V3 spec. It could, like the NUG, include both normative and informative content where the normative stuff would be strictly versioned and controlled and the informative stuff could evolve according to practice and ongoing need.

Scope, you ask? Read the front matter of a mockup spec I pasted below the fold.

I have mocked up what the core of such a thing would look like (with extensive back and forth with Claude). Some of the front matter to give folks a little more fidelity on what I'm suggesting is pasted below the fold on this discussion opener.

I'm very much in spitball / exploratory mode here. A discussion lit off on the cf-conventions discussion board over the last several days and this is one of the things I've been toying with related to that.

Good idea? Bad idea? What on earth are you talking about Dave? How do folks feel about the concept?

Cheers!!

Dave

p.s. my host organization (DOI-USGS) has very strict limits on creating new projects in my personal user or new projects in our organization's collection of open source projects. As a result, sharing this whole project will require a little hacking or someone to make a repo for me to drop the content into. I've formatted it as a zarr-convention and could contribute to the zarr-experiments or some other throw away place if there's a good place to do it.

---

Description

NZUG-1.0 (NetCDF Zarr Users Guide) is a Zarr convention that defines a structural interoperability layer for array-oriented scientific data on top of the Zarr v3 specification. It serves the same structural role for the Zarr ecosystem that the NetCDF Users Guide (NUG) serves for netCDF: it defines a shared vocabulary of structural concepts, a typed data model, naming rules, and a small set of reserved attributes that domain conventions can be written against without modification.

The primary design goal is to allow existing domain conventions — in particular the CF Metadata Conventions — to be applied to Zarr v3 datasets by replacing the NUG as the underlying structural reference, with no changes required to the domain convention itself.

NZUG-1.0 is not a geospatial convention, not a CF extension, and not a replacement for the Zarr v3 specification. It is the structural layer between the format specification and domain conventions.

Motivation

• Interoperability layer gap: Domain conventions for scientific array data (like CF) are written against a format specification that provides structural primitives — named dimensions, typed variables, typed attributes, named coordinate systems. The Zarr v3 specification defines a capable storage model but deliberately leaves these structural refinements to conventions. NZUG fills that gap.
• CF adoption without modification: By providing NUG-equivalent structural concepts, NZUG allows CF to be applied to Zarr using a simple reference substitution (NUG → NZUG), without waiting for geospatial convention debates to resolve.
• Stable foundation for domain conventions: GeoZarr, OME-NGFF, and related conventions can build on NZUG rather than independently re-solving the same structural problems.
• Implementation convergence: Implementations can converge on shared structural behavior before domain-level semantics are fully standardized.

Separation of Concerns

NZUG-1.0 is explicitly scoped to the structural interoperability layer. The following are out of scope and left to conventions that build on NZUG:

• Coordinate reference systems and geospatial metadata
• Standard names, units vocabularies, and physical quantity identification
• Coordinate types (latitude, longitude, time, vertical) and their encoding
• Cell methods, bounds variables, and climatological statistics
• Discrete sampling geometries and feature types
• Multi-scale, tiled, or pyramidal data structures
• Any other domain-specific semantic content

This scoping is intentional and load-bearing. The geospatial and Earth science communities are actively debating CRS encoding, axis order, coordinate role identification, and related questions. NZUG does not enter those debates. It provides the structural floor on which they can be settled independently.

Relationship to the Zarr v3 Specification

NZUG-1.0 adds normative constraints and definitions above the Zarr v3 specification. It does not modify or contradict the Zarr v3 spec. Every valid NZUG-1.0 dataset is a valid Zarr v3 dataset. Implementations that support Zarr v3 can read NZUG-1.0 datasets; they will simply not enforce the additional constraints defined here.

Relationship to Domain Conventions

A domain convention is a document that specifies scientific semantics for datasets built on a structural interoperability layer. The CF Metadata Conventions are the primary domain convention this document is designed to support. NZUG-1.0 provides to a CF-Zarr profile exactly what the NUG provides to CF-netCDF.

[TomNicholas]

This is intriguing - but I'm not quite seeing how this fits in yet - can you give a concrete example of a problem which is solved with the NZUG that is not solved without it? I'm asking because the content in the NetCDF Users Guide (NUG) seems mostly like the sort of content that already exists for zarr, or just isn't relevant to it.

[dblodgett-usgs]

In this context, I think I prefer fairly thin but meaningful layers of specification / convention. Zarr V3 as a base is great. In the past, the "NUG" was a thin layer that NetCDF-C and friends (including xarray) adopted around really basic structural assumptions. People did a lot of different stuff based on that NUG baseline and the amount of tooling compatible with it in one way or another is HUGE. There's also a LOT of data out there that more or less follows the NUG assumptions. As you layer thicker conventions on, you add specificity, but you lose breadth of applicability.

Adding up that historical adoption in software and data, the thin layer defined in the NUG is a real enabler of general tooling that can be used in a very wide array of applications that add their own semantics. But without this thin layer, you really hinder tool interoperability. You can think of it kinda like medicine where you want to "minimum effective dose" to avoid side affects. That's VERY true in conventions. The arrangement of what's in the NUG and widely adopted in implementation is metaphorically the "minimum effective spec".

That said, I am VERY willing to be wrong. I just want to see us able to progress as a community and am putting this out there for others to try on and see if it fits. If not -- no sweat -- but if it does, cool!

[TomNicholas]

Agree with that - the Q I'm asking is whether the proposed NUG is actually the minimum, or the real minimum is something even smaller than the NUG. As 3 different things were mentioned, it sounds like the NUG might be decomposable into at least 3 different conventions? Perhaps the details of this are best discussed on your new repo though.

[dblodgett-usgs]

Oh -- Yeah ... I kinda feel like taking anything out would just make more boiler plate than would be justified, but if there are reasons to split more stuff out, it would be fine to do it IMHO.

[ChrisBarker-NOAA]

NUG is actually the minimum, or the real minimum is something even smaller than the NUG

I think it's smaller than the NUG :-) -- but it's also tricky to know. For my part, I'm starting to look at it from the point of view of CF -- what aspects of what's in the NUG are needed by (or assumed by) CF [*]?

Other standards built on top of netCDF may have other needs, so it's a bit tricky to know what is needed, in general.

[dblodgett-usgs]

I'm trying to look at this from the POV of a data user who has to deal with Zarr datasets from GDAL/NetCDF-C and xarray and also understanding the realities of CF evolution and compatibility from the perspective of software development and version release cycles.

People want consistency and predictability so the data they use "just works" and the software to enable that has to be as simple as possible but no simpler. To enable it, the shared specifications need to follow suit. To my eye, the "no simpler" bit is driven by what already exists. NetCDF-C established the pattern / scope decades ago and we'd need a pretty strong reason to change something that's not broken given that context.

[rabernat]

I think this is a great idea and should be formalized as a Zarr Convention. Thanks @dblodgett-usgs for writing it up!

[dblodgett-usgs]

Thanks @maxrjones. I just initialized the repo and will sync my work up there once I've finished my final review here.

I'm not sure I'd want to "take over" the CF zarr-convention ... more like "over take" ? What I'm working on is trying to be defined as a layer under CF. I'm happy to work with you on how to evolve the CF zarr-convention as we get into things though!

[dblodgett-usgs]

Alright -- I've got it all set for folks to look at over here: https://github.com/zarr-conventions/nzug

Is there a preferred place to discuss newly proposed conventions or is it just organic?

[d-v-b]

Is there a preferred place to discuss newly proposed conventions or is it just organic?

I'd suggest the Zarr Zulip chat#Zarr.

[d-v-b]

people can also open issues on the convention repo itself

[rabernat]

What I like about this proposal is that it is very simple. The NUG is a great core NUGGET of critical data modeling that is easy to understand and explain. The NUG is basically what Xarray implements.

CF conventions are a 1000 page document which immediately generates TLDR for 99% of people who encounter it.

Decoupling these two things seems prudent.

[d-v-b]

Seems like https://github.com/zarr-conventions/CF has a lot of overlap with this proposal -- are they different in some way?

[dblodgett-usgs]

Yes, it is very much different, although there may be some overlap in that to get to CF on Zarr you need what I'm trying to advocate for here. What I'm proposing is not attempting to replicate CF in Zarr. Rather, it's attempting to define the key NetCDF data model structure and syntax that CF adopts from its current format baseline (NetCDF).

I'd be curious how @maxrjones feels about this concept. If we were to adopt what I'm suggesting, the Zarr CF convention would build on the NZUG convention in the same way the the CF-NetCDF builds on the NUG.

[ChrisBarker-NOAA]

Zarr CF convention would build on the NZUG convention in the same way the the CF-NetCDF builds on the NUG.

And ideally in the future, the CF-Convention would build on this mid-layer -- so there would be NO difference between CF-NetCDF and CF-Zarr -- it would be one thing.