Data storage

[wpbonelli]

Flopy3 simulations have a dictionary in which all model/package data is stored and from which it is looked up. This is somewhat analogous (in concept, not implementation) to the memory manager in mf6. We have discussed whether to keep this approach, or to store data in-place in the component hierarchy in flopy4.

Related issues

• Dictionary-like behavior for components #117

Related PRs

• update xattree, misc fixes #95
• xattree now discovers children automatically #91
• minor cleanup for xattree compat, update uv.lock #90
• no beartype, closer with xattree #89
• use xattree #87
• Cleanup and coords prep #85
• seq diagram for attrs/xarray proxy chicanery #84

[wpbonelli]

Using xarray as data store means we should be able to easily import/export an entire simulation (or a piece thereof) to a netcdf file. But we need to consider how flopy's internal xarray representation may differ from the netcdf format expected by mf6's new netcdf reader capability, and from the format expected by 3rd party postprocessing tools. It seems ideal if all these representations were to converge, but 3rd party tools will require e.g. extra metadata that we likely don't want to carry around inside flopy. So the ability to map between the representations may be needed.

[wpbonelli]

Copying in some notes from other threads

• per the xarray coordinate docs, x, y, z, time etc are dimensional coords, and nlay, nrow, ncol, kstp, kper etc are non-dimensional coordinates
• we could use custom accessors for e.g. plotting, layers, any information or functionality we want to attach to a dataset
• we've discussed/prototyped a modflow-specific duck array type, supporting e.g. an optional multiplication factor, convenient layer access, and optimized storage for constant arrays — I wonder now if xarray can do at least the first two: layer as a labeled dimension and/or coordinate, factor stored as metadata and accessed/applied at write time
• xarray works with duck arrays so any extra duck array types we provide would work with it

[wpbonelli]

https://docs.xarray.dev/en/stable/generated/xarray.coding.variables.CFScaleOffsetCoder.html#xarray.coding.variables.CFScaleOffsetCoder might be able to handle the scaling factor

[wpbonelli]

Tentatively we have decided to store data in xarray datasets/trees (depending whether the component is a leaf or node in the component tree). Each component stores its own data in-house, but the datatree provides a unified, hierarchically-addressable view, allowing any component's variables to be accessed from any other component.

Attrs attributes can proxy data stored in xarray data structures, giving us easy introspection and automatic dunder methods without data duplication.

[wpbonelli]

really, this is like an attrs integration for xarray. or vice versa.

which says: give me a bunch of dead simple attrs classes with extra metadata (dims, coords, etc), and I'll tie them into a coherent hierarchical API, with variables aligned along any shared dims, and a datatree backing it.

[wpbonelli]

An initial version of this is more or less in place.

[wpbonelli]

xref #167. also I stumbled across more prior art in https://github.com/astropenguin/xarray-dataclasses/. and james has provided some inspiration in https://github.com/jmccreight/pws_phoenix.

[wpbonelli]

Maybe this is worth a separate discussion, but I'll put it here for now. Re: data schemas and runtime validations. @mjreno has a draft specification for providing input to mf6 via netcdf. We could combine this with an approach like https://github.com/casangi/xradio to specify how a simulation should look in xarray. This would make both sides (mf6 and flopy) of the runtime spec rigorous.