Arrays vs lists

[wpbonelli]

MF6 distinguishes array and list data.

Arrays are always homogeneous and often (but not always) represent some variable living on features of the grid.

Lists can be homogeneous (elements all having the same record type) or heterogeneous (elements can be any of several record types). The former are amenable to numpy, pandas, providing columns separately, etc. The latter not so. Period data for example.

Lists often represent variables living on grid features (just providing a sparse way to specify them), but not always.

To summarize the approach taken so far: to conform to the xarray data model, we are converting sparse (i.e. maxbound shaped) list data into grid/tdis-shaped arrays where applicable (i.e. where the variable is grid/tdis-shaped), and anticipate retaining sparsity where desired through sparse duck arrays. This works smoothly for scalar variables.

Things get a bit more complicated for composites, like heterogeneous (keystring) period data in OC or the possibility of multiple values per variable per grid feature in EVT/RCH. You can stuff any kind of object you want into an xarray array, so there's nothing preventing us from defining classes to represent the more complicated patterns supported by the MF6 list input style and using np.object_ in the type hint.

List data that is not aligned to the grid or time dimensions could remain a list or become a table (recarray or dataframe), depending if it's heterogeneous. Per our current backend, these would have to go in dataset attrs, but that's fine, scalars are already there. Or we could just make all collections data_vars in array form with custom objects as described above. Some of them use the inherited grid/tdis dims, some won't.

We could also consider libraries like ragged or awkward, though I'm not sure how compatible they are with xarray. Maybe also worth considering where we can rework the spec (cf #47 (comment)) or even flopy4's interpretation of it.

[wpbonelli]

Relevant reading re: avoiding awkwardly shaped data

• https://www.jstatsoft.org/article/view/v059i10
• https://docs.xarray.dev/en/stable/user-guide/pandas.html

[wpbonelli]

Enumerating cases of non-scalar grid/tdis-aligned variables:

• Multi-value scalar variables #209
• Timeseries support #210