Support new datatypes from HDF5

[WardF]

Through a conversation with @ethanrd and in light of Unidata/netcdf-c#3217 (and other similar discussions) we are migrating the discussion around new datatypes in netcdf to this, the general netcdf repository. The impacts of new data types go beyond the C library, with potential impacts on netCDF-Java (and the TDS) and other interface libraries.

Adding new datatypes as supported by libhdf5 sounds good, we just want to make sure we take a measured approach, and don't stab ourselves in the foot.

[DennisHeimbigner]

Can you list a few other netcdf repo issues aside from the new types?
I need some idea of what you all are thinking about.

[captainkirk99]

I believe Unidata will deliberate carefully, and that process will naturally take some time. This topic has been under discussion for several years (see Unidata/netcdf-c#2359), and there is also a substantial backlog of unmerged PRs, some of them quite old. It’s important to give Unidata the space it needs to catch up.

Some years ago, while working on HPC code, I created a fork of netCDF-C, called HPC-netCDF. This allowed me to release versions of netCDF-C with important HPC improvements, while still submitting the changes back to the main Unidata repository. Cutting-edge HPC projects at NOAA and NCAR were able to use the new features immediately, while Unidata deliberated. Eventually, the Unidata repository caught up, and all the improvements from HPC-netCDF were incorporated and are now widely used.

A similar example is the Community Codec Repository, which Charlie @zender and I used to release new compression filters and glue code for netCDF (leaning heavily on the work of @DennisHeimbigner supporting filters). The CCR code was later integrated into the Unidata netdef-c distribution. The new compression types will be used by NOAA in the national weather forecast system, starting with the next upgrade, saving approximately 20% of compute time—a substantial efficiency gain, allowing NOAA atmospheric scientists to perform more calculations and produce richer output. (There are not many such simple improvements which yield a 20% increase! Much more work is spent chasing much smaller improvements!)

I believe we can expect a similar impact from 16-bit floats and native complex data types. They will be a big deal Waiting for Unidata’s deliberation could delay these benefits unnecessarily.

I suggest creating a new fork of netCDF-C that can move a bit faster, giving power users access to new features while Unidata continues its careful review. When we discussed compression, the programming itself was straightforward, but the deliberation took over a year. A small amount of code required extensive discussion before it was accepted. A high-performance fork allows important features to be used in practice, providing real-world validation while Unidata completes its review.

There is little reason to support a feature that isn’t yet in use by data producers, but 16-bit floats and native complex types would be highly valuable to the modeling and climate community, who have requested them for years. These types reduce errors compared to the many “home-rolled” solutions currently in use, and, crucially, they save energy. The main motivation for 16-bit floats is efficiency: less data to calculate, compress, transfer, and store, which directly reduces electricity use.

In short, a high-performance fork allows Unidata the time it needs for discussion and management, while still delivering new capabilities to the weather and climate modeling community in a timely manner. Software moves fast, and a year can feel like a long time in this field. Providing these features now ensures that the community can benefit immediately, without slowing Unidata’s careful review process.

We can use: https://github.com/Intelligent-Data-Design-Inc/netcdf-c/tree/main

I will take a stab at adding the FLOAT16 type so we can see how it will look.