Skip to content

Fix zero-dim variables in netCDF3#554

Merged
martindurant merged 2 commits intofsspec:mainfrom
martindurant:cdf
Jul 10, 2025
Merged

Fix zero-dim variables in netCDF3#554
martindurant merged 2 commits intofsspec:mainfrom
martindurant:cdf

Conversation

@martindurant
Copy link
Member

Fixes #551

@martindurant
Copy link
Member Author

martindurant commented Apr 16, 2025

@rsignell-usgs allows the following workflow on zarr3:

from kerchunk.netCDF3 import NetCDF3ToZarr
d0 = NetCDF3ToZarr("s3://umassd-fvcom/gom3/hindcast/gom3_197801.nc",
                  storage_options={"anon": True},
                  inline_threshold=400).translate()
import xarray as xr
import kerchunk
store = kerchunk.utils.refs_as_store(d0, remote_options={"anon": True}, remote_protocol="s3")
ds = xr.open_dataset(store, engine="zarr", decode_times=False)

giving

<xarray.Dataset> Size: 97GB
Dimensions:        (nele: 90415, node: 48451, siglay: 45, siglev: 46, three: 3,
                    maxnode: 11, maxelem: 9, four: 4, time: 745)
Coordinates:
    x              (node) float32 194kB ...
    y              (node) float32 194kB ...
    siglay         (siglay, node) float32 9MB ...
    siglev         (siglev, node) float32 9MB ...
  * time           (time) float32 3kB 4.351e+04 4.351e+04 ... 4.354e+04
Dimensions without coordinates: nele, node, three, maxnode, maxelem, four
Data variables: (12/46)
    partition      (nele) int32 362kB ...
    lon            (node) float32 194kB ...
    lat            (node) float32 194kB ...
    xc             (nele) float32 362kB ...
    yc             (nele) float32 362kB ...
    lonc           (nele) float32 362kB ...
    ...             ...
    l              (time, siglev, node) float32 7GB ...
    short_wave     (time, node) float32 144MB ...
    net_heat_flux  (time, node) float32 144MB ...
    uwind_stress   (time, nele) float32 269MB ...
    vwind_stress   (time, nele) float32 269MB ...
    nprocs         int32 4B ...
Attributes: (12/14)
    title:                       GOM3 Nesting  by Yf.Sun@umassd.edu
    institution:                 School for Marine Science and Technology
    source:                      FVCOM_3.0
    history:                     model started at: 22/12/2011   16:41
    references:                  http://fvcom.smast.umassd.edu, http://codfis...
    Conventions:                 CF-1.0
    ...                          ...
    Tidal_Forcing:               TIDAL ELEVATION FORCING IS OFF!
    River_Forcing:               THERE ARE 50 RIVERS IN THIS MODEL.\nRIVER IN...
    GroundWater_Forcing:         GROUND WATER FORCING IS OFF!
    Surface_Heat_Forcing:        FVCOM variable surface heat forcing file:\nF...
    Surface_Wind_Forcing:        FVCOM variable surface Wind forcing:\nFILE N...
    Surface_PrecipEvap_Forcing:  FVCOM periodic surface precip forcing:\nFILE...

The decode_times is needed because otherwise: unable to decode time units 'msec since 00:00:00' with 'the default calendar' (maybe what you mentioned)

@rsignell
Copy link

rsignell commented Jul 10, 2025

This works -- I thought I commented here but apparently not! From my POV this can be merged!

@martindurant martindurant merged commit ebff31c into fsspec:main Jul 10, 2025
4 checks passed
@martindurant martindurant deleted the cdf branch July 10, 2025 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Index error when creating refs for this AWS NetCDF3 dataset

2 participants